Stat 414 – Review II

 

Due (preferably before Tuesday, 9am): In Canvas,

(1) post at least one question you have on the material described below.

(2) post at least one example question that you think I could ask on the material described below

 

Optional Review Session: Interest?

 

Optional Review Problems: review2probs.html

 

Format of Exam: The exam will cover Days 11-28, Quizzes 11-27, Homeworks 3-6, Ch. 4-8, and Ch. 10).  The exam will mostly be interpretation questions. The exam will be worth approximately 50 points.  Be ready to explain your reasoning, including in “laymen’s” language. You can use two pages 8.5x11 pages of notes (hard copy), but I encourage you to prepare as if a closed book exam. Advice: Review Quizzes/commentary, HW solutions/ grader comments, Lecture notes. 

 

Overall

“One can think of mixed-effects models as ordinary regression models that have additional variance terms for handling non-independence due to group membership. The key to mixed-effects models is to understand how nesting individuals within groups can produce additional sources of variance (non-independence) in data.”

 

From Day 11 (see also Sec 4.7), you should be able to

·       Explain the principle of Maximum Likelihood as another possible estimation method

o   One variable case

o   Two variable case

o   How relates to least squares estimation

o   How impacts estimation of residual standard error (now considered another parameter simultaneously estimated by the model)

·       Using AIC, BIC, and log likelihood values as Information criteria/Measures of Fit

·       Using achieved maximum likelihood values to test significance of individual coefficients, groups of coefficients, whole model

o   Likelihood ratio test (df)

·       Determine the df used by a model

o      Including if use additional variance terms in the model (e.g.,  per month)

 

From Day 12, you should be able to

·       Explain how restricted maximum likelihood (REML) is another possible estimation method that produces unbiased estimates of .

o     Accounts for the degrees of freedom used to estimate the s

o     Usually doesn’t change estimates of ’s but does change , likelihood, AIC, BIC values

o   When sample sizes are large there aren’t major differences among the three estimation methods

o   Be able to identify which method is being used by the software

·       Consider whether one model is “nested” in another

o   Can you get from one model to the other by setting some of the parameters to 0?

o   Can sometimes still get a “rough comparison” of not-nested models

·       Keep in mind that if use “anova” to compare models, the models need to be constructed using the same estimation method

o   Sometimes make distinction between whether are comparing models with different fixed effects or different random effects, but not critical at the moment

·       Keep in mind that weighted least squares is a special case of GLS = generalized least squares (allows for more interesting variance structures in the data)

·       Quiz: Philosophy of maximum likelihood, components of AIC, carry out LRT, chi-square values should be >> df to be significant, df for model/comparing models

 

From Day 13, you should be able to

·       Identify whether a study plan is likely to result in correlated observations

o   e.g., repeat observations, clustered data, cluster-randomized trials

·       Interpret ICC calculated from an ANOVA table as a measure of correlation between observations in the same group

·       Compute the effective sample size of a study based on the ICC, (common) group size, number of groups

o   Compromise between group size and number of groups

·       Discuss impact on standard errors

 

From Day 14 (and maybe Sec 3.3), you should be able to

·       Explain the distinction between treating a factor as random or fixed

·       Discuss when you might choose to treat a variable as having random effects rather than fixed effects

·       Observed “levels” are representative of a larger population (random sample?)

·       Want to make inferences to the larger population

·        Compute the ICC in terms of random effects (between group vs. total variation)

·       Standard errors reflect the randomness at the individual level and from the random effect (e.g., sampling error)

·       Discuss advantages such as

o   Generalizability

o   Degrees of freedom

o   Use of Level 2 units + Level 2 variables in same model

 

From Day 15 (and Sec. 4.8), you should be able to

·       Fit and interpret the “random intercepts” model

o   Like adding the “grouping variable” to the model but treating as random effects

o   may be all that is required to adequately account for the nested nature of the grouped data.”

o   there is virtually no downside to estimating mixed-effect models even when If  is small or non-significant because in these cases the mixed-effect models just return the OLS estimates)” (Bliese et al., 2018)     

o   Also known as “unconditional means” or “null” model or “one-way random effects ANOVA” (assumes equal group sizes)

·       Explain the principle of shrinkage/pooling as it pertains to multilevel models

o      where

o   No pooling « weights = 1 (treating as fixed effects)

o   Complete pooling « weights = 0 (no group to group variation)

·       Predict the impact of pooling on variance estimates

·       Identify and explain factors that impact the size of the weights/consequences/amount of shrinkage

o   Determine when (sample size) an estimated mean will be closer to the group mean vs. the overall mean

 

From Day 16+, you should be able to

·       Write out the random intercepts model:  where

·       Discuss consequences of treating a factor as random effects

o      Impact on model equations (now estimating one parameter = )

o     Total Var(Y) = +

o     ICC =

o   Variance-Covariance matrix for the observations

o   Impact on standard errors of coefficients (adjusting for dependence)

·       Compute and interpret intraclass correlation coefficient as expected correlation between two randomly selected individuals from the same group

o   Show how this correlation coefficient is “induced” by using the random intercepts model

·       Be able to explain/interpret the variance-covariance matrix for the responses

A close-up of a test

Description automatically generated

Covariance

 

Site 1 (beach 1)

Site 2 (beach 1)

Site 3 (beach 2)

Site 4 (beach 2)

Site 1 (beach 1)

0

0

Site 2 (beach 1)

0

Site 3 (beach 2)

Site 4 (beach 2)

0

0

 

Correlation

 

Site 1 (beach 1)

Site 2 (beach 1)

Site 3 (beach 2)

Site 4 (beach 2)

Site 1 (beach 1)

0

0

Site 2 (beach 1)

0

0

Site 3 (beach 2)

0

0

Site 4 (beach 2)

0

0

1

 

·       Assess the statistical significance of the variation among Level 2 units

o   Stating null and alternative hypotheses

o   Fixed effects ANOVA

o   Confidence intervals for variance components

o   Likelihood ratio test (MLE or REML approach)

§  Cut p-value in half?

·       Understand that we don’t normally make the fixed vs. random decision based on a likelihood ratio test, but instead consider the study design/research question of interest

·       Identify level 1 and level 2

·      Write out the statistical model for a “random intercepts” model (’s, uj, s, )

o   Interpret model components

o   Define indices

o   Interpretation of intercept

o   Standard deviations vs. variances vs. covariances

o   Composite equation vs. Two-level equations

·       Identify number of units at each level

·       Interpret R output (lme, lmer)

o   How to use nlme::lme and lme4::lmer in R

o     Is output reporting  or ?

 

From Day 17, you should be able to

·      Explain the distinction between  and

·       Add a Level 1 variable to the random intercepts model

o   Interpretation of “adjusted” association, parameter estimates

§  Also remember to zero out (intercept) or “fix” Level 2 units (e.g., for a particular school)

o   Visualize the model (e.g., parallel lines)

o   Level 1 and Level 2 equations, composite equation

o   R2 type calculations

o   Assessing significance (t-test vs. likelihood ratio test)

 

From Day 18, you should be able to

·       Explain the distinction between variation explained by fixed effects vs. by random effects

·       Explain the distinction between total variation and variation explained by random effects after adjusting for fixed effects

·       Calculate percentage change in Level 1, Level 2, and overall variation after including variable

·       If a variable coming into the model is related to the level 2 units, variation explained will depend on the nature of that relationship (similar to adjusted associations in regular regression models)

·       Include a Level 1 variable and reinterpret coefficients as adjusted for that variable

o   Also remember to zero out (intercept) or “fix” Level 2 units (e.g., for a particular school)

 

From Day 19 you should be able to:

·       Distinguish Level 1 vs. Level 2 predictors

·       Write out “level equations” and “composite” equation using appropriate symbols and indices

o   Use estimated equations (fixed and random effects) to predict a response value

·       Include a Level 2 variable to (ideally) explain variation in intercepts

o   Interpreting an aggregated group mean variable (“if the group mean IQ increases…”)

o     Slope coefficients with x and  vs. slope coefficients with  and

Model includes

Can explain variation

x

Level 1 and Level 2

Level 2

x –

Level 1 (“pure Level 1”)

x and

Level 1 (only) and “emergent” effect at Level 2

x- and

Level 1 (only) and Level 2 (separated)

 

From Day 20 (and Sec 5.1), you should be able to:

·       Interpret random slopes models (aka “random coefficients”)

o   Interpretation as interaction across higher level units

§  e.g., Level 1 variables can have random slopes at Level 2

·       Explain the distinction between a random slopes model and fitting a separate equation for each Level 2 group

o   Complete pooling vs. Partial pooling vs. No pooling

·       Interpret the standard deviation/variance of the slopes ()

·       Write out the Level 1 and Level 2 equations for the random slopes model

o   Including specifying error terms, and their distributions, and covariance terms

o   Do be careful with indices

o   Thinking of Level 2 equations as “intercepts as outcomes” and “slopes as outcomes” models

o   Interpretation of the model components

o   Generally include correlation between slopes and intercepts (and slopes and slopes) in the model but can force to be zero

·       Compare the random slopes to the fixed slopes models and decide significance of random slopes

·       Compare relative sizes of variance components in context

·       Interpretation/visualization of variance component for slopes

o   95% of slopes should fall within 2 SD of overall slope

·       Interpretation of covariance/correlation between random intercepts and random slopes

·      Distinguish between “random slopes” (and “slope effects” (

 

From Day 21 (and Sec. 5.2), you should be able to:

·       Add a Level 2 variable to explain variation in random slopes

o   Inclusion of cross-level interactions

§  Interpretation

o   Level equations (adding Level 2 variable to equation for intercept vs. slope vs. both)

o   Measuring change in Level 1 and Level 2 variances as percentage change

§  Could explain pretty much all of the Level 2 variation/a Level 2 variable can be sufficient adjusting for the clustering in the study design

 

No Day 22 Handout

 

From Day 23 (and Ch. 8) you should be able to:

·       Explain how random slopes models induce heteroscedasticity in the responses

o      Variance as quadratic function of

o      Minimized at

§      Does this occur within the range of  values in the dataset?

·       Explain how random slopes models different correlations among pairs of  depending on the  values involved

o     Translating between  and )

·       Interpret covariance/correlation between random intercepts and random slopes

o   Distinguish between cov(y’s) and cov(u’s)

o   Sign of covariance and implications for fanning in/fanning out of lines

§  Interpretation in context

o     Translating between  and )

§  Recognizing which is reported in the output

·       Interpret the variance-covariance matrix output (marginal vs. conditional)

o   Compare model predictions to observed results

·       Distinguish between Level 1 variance, Level 2 variance, and total variance

·       Explain limitation of ICC in random slopes model

 

From Day 24, you should be able to:

·       Consider random slopes for multiple variables

o   Do you want them to be correlated?

o   Can increase complexity of model pretty quickly

o   Interpretation of random effects correlations (and identifying pairs from output)

·       Determine the number of parameters being estimated in a model

o   SD and Var are just 1, Include covariances

 

 

From Day 25 (and Ch. 10), you should be able to:

·       Examine, identify, interpret residuals in a multilevel model

o   Marginal vs. Conditional residuals vs. Random effects

o   How they compare to each other based on relative position of observations/group

·       Explain how patterns in a residual plot could suggest changes to the model (e.g., interaction)

 

From Day 26, you should be able to:

·       Explore and interpret influential observations (what it means to be influential and why we care), possible remedies

o   DFBetas vs. Cook’s D vs. Change in significance

o   How to interpret graphs from influence package

o   Multicollinearity

 

 

From Case Study you should be able to:

·       Explore a data set, including graphs for Level 1 and Level 2 relationships and possible consequences for model building

·       Interpretation of different model components in context

o   Go beyond “variation in intercepts and slopes” but be able to explain in context what the intercepts and slopes represent

o   Including for categorical variables

o   Including variance explained

o   Including interactions

o   Including main effects when have interactions

§  When need to “fix” other variables and when need to set them to zero or mean

·       Suggest improvement/next steps to models

o   E.g., removing insignificant terms and what that means for the R code, the model equation, and the graphs of the fitted equations

 

In addition:

o   Using estimated effects to make predictions (e.g., “what is the expected response for a student with IQ=100” “What is the expected response for a student with IQ = 100 at school 2”)

o   Explain the principle of shrinkage/pooling as it pertains to multilevel models

o   Estimate the precision of the estimated random effects using estimated variance components and group size

§      

o   Understand that R stores the estimated effects in ranef(model)and the standard errors (variances) in ranef(model2, condVar=TRUE)

§  Can use this information to make confidence intervals around the random effects, but not usually of interest and certainly shouldn’t be used to make any “causal” claims.

·       Using estimated effects to make predictions (e.g., “what is the expected response for school 2” or “what is the expected response for a school at the median”)

o       (predicted means vs. estimated effects)

o   Use properties of the normal distribution to make predictions of (between or within) group effects (e.g., 16th percentile)

·       Comparing models

o   Using likelihood ratio test to assess whether a full model is significantly better than a reduced (nested) model

§  Be able to write out hypotheses

o   Using AIC, BIC to compare models

o   Can also calculate percentage change in variation explained at different levels (pseudo-R2)

·       Do be able to interpret model equations and model output

o   Parameters vs. statistics

·       Centering

o   Why good idea

o   Interpretation of coefficients

o   Group mean centering vs. Grand mean centering

o   Within group vs. Between group regressions

§  Including exploration/explanation for why they could differ

·       How fixed effects and random effects can mask each other when build a model sequentially

o   Can be difficult to know which modeling approach is better so be sure to explore some different possibilities

 

More reminders

·       Distinguish between aggregating and disaggregating data

·       Distinguish between correlation of observations, correlation of errors, and correlation of slopes

·       Be able to set up a model based on the research questions

o   Determine number of parameters

o   Translate R code into model description

·       Translate between equations and graphs and output

·       Be able to interpret interactions in context

 

Keeping track of variances

·       = variation in response variable

·        = standard deviation of explanatory (aka predictor) variable

·       = variation about regression line/unexplained variation in regression model, variation in response at a particular x value

·       = sample to sample variation in regression slope

·       = sample to sample variation in estimated group effect (estimated random effect)

·        = sample to sample variation in estimated predicted value.  There are actually two “se fit” values, one for a confidence interval to predict E(Y) and one for a prediction interval to predict . The latter can be approximated with , but actually depends on  etc.

·       variation in intercepts across Level 2 units

·       : variation in slopes across Level 2 units

·       : covariation between intercepts and slopes (vs. )

·       : correlation between intercepts and slopes

·       :  covariation between 1 set of random slopes and 2nd set of random slopes? (vs. ) – remember we are now ignoring the variance-covariance matrix for the fixed regression coefficient (previously the (square roots) of the diagonal elements had given us .  We can get the variance-covariance matrix for the random effects using getVarCov on a model fit using nlme::lme with variance components on diagonal (e.g., slopes and intercepts) and covariances on the off-diagonal.

o   getVarCov(model) vs. getVarCov(model, getVarCov(model1, type = "conditional")) vs. getVarCov(getVarCov(model1, type = "marginal")