Performance evaluation of recent information criteria for selecting multilevel models in Behavioral and Social Sciences

  1. Vallejo Seco, Guillermo 1
  2. Tuero Herrero, Ellián 1
  3. Núñez Pérez, José Carlos 1
  4. Rosário, Pedro 2
  1. 1 Universidad de Oviedo
    info

    Universidad de Oviedo

    Oviedo, España

    ROR https://ror.org/006gksa02

  2. 2 Universidade do Minho
    info

    Universidade do Minho

    Braga, Portugal

    ROR https://ror.org/037wpkx04

Revista:
International journal of clinical and health psychology

ISSN: 1697-2600

Año de publicación: 2014

Volumen: 14

Número: 1

Páginas: 48-57

Tipo: Artículo

DOI: 10.1016/S1697-2600(14)70036-5 DIALNET GOOGLE SCHOLAR

Otras publicaciones en: International journal of clinical and health psychology

Resumen

This study was designed to find the best strategy for selecting the correct multilevel model among several alternatives taking into account variables such as intraclass correlation, number of groups (m), group size (n), or others as parameter values and intercept-slope covariance. First, we examine this question in a simulation study and second, to illustrate the behavior of the criteria and to explore the generalizability of the findings, a previously published educational dataset is analyzed. The results showed that none of the selection criteria behaved correctly under all the conditions or was consistently better than the others. The intraclass correlation somewhat affects the performance of all selection criteria, but the extent of this influence is relatively minor compared to sample size, parameter values, and correlation between random effects. A large number of groups appears more important than a large number of individuals per group in selecting the best model (m = 50 and n = 20 is suggested). Finally, model selection tools such as Akaike's Information Criterion (AIC) or the conditional AIC are recommend when it is assumed that random effects are correlated, whereas use of the Schwarz's Bayesian Information Criterion or the consistent AIC are advantageous for uncorrelated random effects

€ Ver financiación

Información de financiación

We gratefully thank the Editor and the anonymous reviewers for the constructive comments that led to substantial improvements in the manuscript. This paper was prepared with support from the Spanish Ministry of Science and Innovation (Ref: PSI-2011-23395 & EDU-2010-16231).

Financiadores

Referencias bibliográficas

  • Best, N., Mason, A., Richardson, S.(2012).Two-pronged strategy for using DIC to compare selection models with non-ignorable missing responses.Bayesian Analysis, 7, 109-146.
  • Burnham, K.P., Anderson, D.R., Huyvaert, K.P.(2011). AIC model selection and multimodel inference in behavioral ecology: Some background, observations, and comparisons.Behavioral Ecology and Sociobiology, 65, 23-35.
  • Dettmers, S., Trautwein, U., Lüdtke, O., Kunter, M., & Baumert, J.(2010).Homework works if homework quality is high: Using multilevel modeling to predict the development of achievement in mathematics.Journal of Educational Psychology, 102, 467-482.
  • Gibbons, R.D., Hedeker, D., & DuToit, S.(2010).Advances in analysis of longitudinal data.Annual Review of Clinical Psychology, 6, 79-107.
  • Greven, S., & Kneib, T.(2010).On the behaviour of marginal and conditional AIC in linear mixed models.Biometrika, 97, 1-17.
  • Gurka, M.J.(2006).Selecting the best linear mixed model under REML.The American Statistician, 60, 19-26.
  • Hamaker, E.L., Van Hattum, P., Kuiper, R.M., & Hoijtink, H.(2011).Model selection based on information criteria in multilevel modeling.In J.J.Hox, & J.K.Roberts (Eds.),Handbook of advanced multilevel analysis (pp.231-255).New York: Taylor & Francis.
  • Hartley, J.(2012).New ways of making academic articles easier to read.International Journal of Clinical and Health Psychology,12, 143-160.
  • Hox, J.J.(2010).Multilevel analysis.Techniques and applications (2th.ed.).New York: Routledge.
  • Imel, Z.E., Hubbard, R.A., Rutter, C.M., & Simon, G.(2013).Patient-rated alliance as a measure of therapist performance in two clinical settings.Journal of Consulting and Clinical Psychology, 81, 154-165.
  • Johnson, J.B., & Omland, K.S.(2004).Model selection in ecology and evolution.Trends in Ecology and Evolution, 19, 101-108.
  • Lee, H., & Ghosh, S.K.(2009).Performance of information criteria for spatial models.Journal of Statistical Computation and Simulation, 79, 93-106.
  • Maas, C.M., & Hox, J.J.(2004).The influence of violations of assumptions on multilevel parameter estimates and their standard errors.Computational Statistics & Data Analysis, 46,427-440.
  • Núñez, J.C., Rosário, P., Vallejo, G., & González-Pienda, J.A.(2013).A longitudinal assessment of the effectiveness of a school-based mentoring program in middle school.Contemporary Educational Psychology, 38, 11-21.
  • Núñez, J.C., Vallejo, G., Rosário, P., Tuero-Herrero, E., & Valle,A.(in press).Variables from the students, the teachers and the school context predicting academic achievement: A multilevel perspective.Journal of Psychodidactics. DOI:http://dx.doi.org/10.1387/RevPsicodidact.7127.
  • Snijders, T., & Bosker, R.(2012).Multilevel analysis: An introduction to basic and advanced multilevel modeling (2th.ed.).Thousand Oaks, C.A.: Sage.
  • Sobral, J., Villar, P., Gómez-Fraguela, J.A., Romero, E., & Luengo,M.A.(2013).Interactive effects of personality and separationas acculturation style on adolescent antisocial behaviour.International Journal of Clinical and Health Psychology, 13,25-31
  • Spiegelhalter, D.J., Best, N.G., Carlin, B.P., & Van der Linde, A.(2002).Bayesian measures of model complexity and fit.Journal of the Royal Statistical Society Series B, 64, 583-640.
  • Srivastava M.S., & Kubokawa, T.(2010).Conditional information criteria for selecting variables in linear mixed models.Journal of Multivariate Analysis, 101, 1970-1980.
  • Sterba, S.K., & Pek, J.(2012).Individual influence on model selection.Psychological Methods, 17, 582-599.
  • Vaida, F., & Blanchard, S.(2005).Conditional Akaike information for mixed-effects models.Biometrika, 92, 351-370.
  • Verbeke, G., & Molenberghs, G.(2009).Linear mixed models for longitudinal data (3th.ed.).New York: Springer-Verlag.
  • Vallejo, G., Arnau, J., Bono, R., Fernández, M.P., & Tuero-Herrero,E.(2010).Nested model selection for longitudinal data using information criteria and the conditional adjustment strategy.Psicothema, 22, 323-333.
  • Vallejo, G., Ato, M., & Valdés, T.(2008).Consequences of misspecifying the error covariance structure in linear mixed models for longitudinal data.Methodology: Journal of Research Methods for the Behavioral and Social Sciences, 4, 10-21.
  • Vallejo, G., Fernández, M.P., Livacic-Rojas, P.E., & Tuero-Herrero, E.(2011).Selecting the best unbalanced repeated measures model.Behavior Research Methods, 43, 18-36.
  • Wang, J., & Schaalje, G. B. (2009). Model selection for linear mixed models using predictive criteria. Communications in Statistics - Simulation and Computation, 38, 788-801.
Fuente de los datos: Dialnet