Computerized Adaptive Assessment of Organizational Climate

  1. Fernando Menéndez
  2. Elsa Peña-Suárez 1
  3. Eduardo Fonseca-Pedrero 2
  4. José Muñiz 3
  1. 1 Educational Counselling of Avilés, The Regional Ministry of Education and Culture of Principality of Asturias, Spain
  2. 2 Universidad de La Rioja
    info

    Universidad de La Rioja

    Logroño, España

    ROR https://ror.org/0553yr311

  3. 3 Centro de Investigacion Biomedica en Red de Salud Mental
    info

    Centro de Investigacion Biomedica en Red de Salud Mental

    Madrid, España

    ROR https://ror.org/009byq155

Show affiliations +
Journal:
Anales de psicología

ISSN: 0212-9728,  1695-2294

Year of publication: 2017

Volume: 33

Issue: 1

Pages: 152-159

Type: Article

DOI: 10.6018/ANALESPS.33.1.225921 DIALNET GOOGLE SCHOLAR

More publications in: Anales de psicología

Sustainable development goals

Abstract

Computerized Adaptive Tests (CAT) constitutes a major advance in the field of the assessment of social sciences and health. It is interesting to incorporate this methodological development to organizational field. The objective of this research is to develop a CAT for evaluating the organizational climate. In order to do this, a bank of 160 items was constructed, which was administered to a sample of 3,163 workers (M age= 51.90 years; SD= 6.28 years). The results show that the CAT works efficiently for the three item banks used (150, 130 and 50 items). The information function shows that the latent trait in the whole range of values is measured with similar accuracy. The correlation between the scores obtained with the CAT, the three item banks and the CLIOR scale of organizational climate was higher than .90 in all cases (SE<0.32). With a SE< 0.32, the 150 item CAT uses a mean of 22.5 items (SD = 13.5 items). The CAT developed allows an accuracy and useful assessment of the organizational climate on organizational settings and research.

€ View funding

Funding information

This research was funded by the Department of Public Administration of the Government of the Principality of Asturias (FUO-EM-305-06) and Spanish Ministry of Science and Innovation (PSI 2011-28638, PSI 2014-56114-P). We thank the Government of the Principality of Asturias for their collaboration in the gathering of data.

Funders

Bibliographic References

  • Aguado, D., Rubio, V.J., & Haley, S.M. (2005). Propiedades psicométricas de un test adaptativo informatizado para la medición del ajuste emocional [Psychometric properties of an emotional adjustment computerized adaptive test]. Psicothema, 17(3), 484-491.
  • Allen, D.D., Ni, P., & Haley, S.M. (2008). Efficiency and sensitivity of multidimensional computerized adaptive testing of pediatric physical functioning. Disability and rehabilitation, 30, 479-484.
  • American Educational Research Association, American Psychological Association, and National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
  • Andrich, D. (1978). A rating scale formulation for ordered response categories. Psychometrika, 43, 561-573.
  • Asif, F. (2011). Estimating the impact of Denison’s (1996), "What is the difference between organizational culture and organizational climate? A native’s point of view on a decade of paradigm wars. Journal of Business Research, 64, 454-459.
  • Bartram, D., Robertson, I.T., & Callinan, M. (2002). A framework for examining organizational effectiveness. In I.T. Robertson, M. Callinan, & D. Bartram (Eds.), Organizational effectiveness: The role of psychology (pp. 1-10). Chichester, UK: Wiley and Sons.
  • Bjorner, J.B., Chang, C., Thissen, D., & Reeve, B. (2007). Developing tailored instruments: item banking and computerized adaptive assessment. Quality Life Research, 16, 95-108..
  • Carr, J.Z., Schmidt, A.M., Ford, J.K., & DeShon, R.P. (2003). Climate perceptions matter: A meta-analytic path analysis relating molar climate, cognitive and affective states, and individual level work outcomes. Journal of Applied Psychology, 88, 605-619.
  • Cella, D., Gershon, R., Lai, J.S., & Choi, S. (2007). The future of outcomes measurement: item banking, tailored short-forms, and computerized adaptive assessment. Quality Life Research, 16, 133-141.
  • Cheng, Y., Luh, W.M., & Guo, Y.L. (2003). Reliability and Validity of the Chinese Version of the Job Content Questionnaire (C-JCQ) in Taiwanese Workers International Journal of Behavioral Medicine, 10, 15-30.
  • Chien, T.W., Lai, W., Lu, Ch., Wang, W., Chen, S., Wang, H., & Su, S. (2011). Web-based computer adaptive assessment of individual perceptions of job satisfaction for hospital workplace employees. BMC Medical Research Methodology, 11, 1-8.
  • Chien, T.W., Wu, H.M., Wang, W.C., Castillo, R.V., & Chou, W. (2009). Reduction in patient burdens with graphical computerized adaptive testing on the ADL scale: tool development and simulation. Health and Quality of Life Outcomes, 5, 7-39. doi: 10.1186/1477-7525-7-39.
  • Chien, T.W., Wang, W.C., Wang, H.Y., & Lin, H.J. (2009). Online assessment of patients’ views on hospital performances using Rasch model’s KIDMAP diagram. 9. BMC Health Services Research, 31, 135.
  • Cole, L.L., Grubb, P.L., Sauter, S.L., Swanson, N.G., & Lawless, P. (1997). Psychosocial correlates of harassment, threats and fear of violence in the workplace. Scandinavian Journal of Work, Environment & Health, 23, 450-457.
  • Corral, S., & Pereña, J. (2010). Cuestionario de clima laboral. Manual [Work Climate Questionnaire. Manual] (2a ed.). Madrid: TEA Ediciones.
  • Cummings, L.L., & deCotiis, T.A. (1973). Organizational correlates of perceived stress in a professional organization. Public Personnel Management, 2, 275-282.
  • deCotiis, T.A., & Summers, T.P. (1987). A path analysis of a model of the antecedents and consequences of organizational commitment. Human Relations, 40, 445-470.
  • Delgado-Rico, E., Carretero-Dios, H., & Ruch, W. (2012). Content validity evidences in test development: an applied perspective. International Journal of Clinical and Health Psychology, 12, 449-460.
  • Denison, D.R. (1996). What is the difference between organizational culture and organizational climate? A native’s point of view on a decade of paradigm wars. The Academy of Management Review, 21, 619-654.
  • Downing, S.M., & Haladyna, T.M. (2006). Handbook of test development. Mahwah, NJ: Erlbaum.
  • Ekvall, G. (1996). Organizational climate for creativity and innovation. European Journal of Work and Organizational Psychology, 5, 105-123.
  • Fernández-Ballesteros., R., & Sierra, B (1984). Escalas de clima social: familia, trabajo, instituciones penitenciarias, centro escolar. Manual: investigación y publicaciones psicológicas [Social climate scales: Family, work, penitentiary institutions, school. Manual: Research and psychological publications]. Madrid: TEA Ediciones.
  • Fonseca-Pedrero, E., Menéndez, F.L., Paino, M., Lemos-Giráldez, S., & Muñiz, J. (2013). Development of a Computerized Adaptive Test for Schizotypy Assessment. PLoS ONE 8(9), e73201.
  • González-Romá, V., Tomás, I., Peiró, J.M., Lloret, S., Espejo, B., Ferreres, D., & Hernández, A. (1996). Análisis de las propiedades psicométricas del cuestionario de clima organizacional FOCUS-93 [Analysis of the psychometric properties of the organizational climate questionnaire FOCUS-93]. Revista de Psicología Social Aplicada, 6, 5-22.
  • Gibbons, R. D., Weiss, D. J., Kupfer, D. J., Frank, E., Fagiolini, A., Grochocinski, V. J., . . . Immekus, J. C. (2008). Using computerized adaptive testing to reduce the burden of mental health assessment. Psychiatric Services, 59(4), 361-368.
  • Haladyna, T.M. (2004). Developing and validating multiple-choice test item (3 rd ed.). Hillsdale, NJ: Erlbaum
  • Hambleton, R. K. (1989). Principles and Selected applications of item response theory. In R. L. Linn (Ed), Educational measurement (pp.147-200). New York, McMillan.
  • James, L.R., Choi, C.C., Ko, C.E., McNeil, P.K., Minton, M.K., Wright, M.A., & Kim, K. (2008). Organizational and psychological climate: A review of theory and research. European Journal of Work and Organizational Psychology, 17, 5-32.
  • Jette, A.M., Haley, S.M., Ni, P., Olarsch, S., & Moed, R. (2008). Creating a Computer Adaptive Test Version of the Late-Life Function and Disability Instrument. The Journals of Gerontology, 63, 1246-1256.
  • Jones, A.P., & James, L.R. (1979). Psychological climate: Dimensions and relationships of individual and aggregated work environment perceptions. Organizational Behavior and Human Performance, 23, 201-250.
  • Kopec, J.A., Badii, M., McKenna , M., Lima, V.D., Sayre, E.C., & Dvorak, M. (2008). Computerized adaptive testing in back pain: validation of the CAT-5DQOL. Spine, 33, 1384-1390.
  • Lazarus, R.S. (1982). Thoughts on the relations between emotion and cognition. The American Psychologist, 37, 1019-1024.
  • Lazarus, R.S., & Folkman, S. (1984). Stress, appraisal, and coping. New York: Springer-Verlag.
  • López-Fernández, L.A., Sánchez-Cantilejo, E., Calzas-Urrutia, D., Siles- Román, D., Sevilla-García, E., & de Oleaga-Usategui, I. (1988). Elaboración de un cuestionario para el estudio del clima organizacional de los centros de salud [Elaboration of a questionnaire to study the organizational climate of health centers]. Atención Primaria, 5(10), 531-535.
  • Lord, F. (1990). Applications of Item Response Theory to practical testing problems: Hillsdale, NJ: Erlbaum Associates.
  • Martin, M., Kosinski, M., Bjorner, J.B., Ware, J.E., Maclean, R., & Li, T. (2007). Item response theory methods can improve the measurement of physical function by combining the modified health assessment questionnaire and the SF-36 physical function scale. Quality Life Research, 16, 647-660.
  • Martínez-Arias, R. (1995). Psicometría: teoría de los tests psicológicos y educativos [Psychometrics: theory of psychological and educational tests]. Madrid, Síntesis.
  • Moreno, R., Martínez, R., & Muñiz, J. (2006). New guidelines for developing multiple-choice items. Methodology, 2, 65-72.
  • Muñiz, J., & Fonseca-Pedrero, E. (2008). Construcción de instrumentos de medida para la evaluación universitaria [Developing measurement instruments for university assessment]. Revista de Investigación en Educación, 5, 13-25
  • Muñiz, J., Peña-Suárez, E., de la Roca, Y., Fonseca-Pedrero, E., Cabal, A.L., García-Cueto, E. (2014). Organizational climate in Spanish Public Health Services: Administration and Services Staff. International Journal of Clinical and Health, Psychology, 14, 102-110.
  • Ostroff, C. (1993). The effects of climate and personal influences on individual behavior and attitudes in organizations. Organizational Behavior and Human Decision Processes, 56, 56-90.
  • Ostroff, C., Kinicky, A., & Tamkins, M.M. (2003). Organizational culture and climate. In W.C. Borman, D.R. Ilgen, & R.J. Klimoski (Eds.), Handbook of psychology (Vol. 12 Industrial and Organizational). Hoboken, NJ: Wiley and Sons.
  • Peña-Suárez, E., Muñiz, J., Campillo-Álvarez, A., Fonseca-Pedrero, E., & García-Cueto, E. (2013). Assessing organizational climate: Psychometric properties of CLIOR scale. Psicothema, 24, 137-144.
  • Petersen, M.A., Groenvold, M., Aaronson, N., Fayers, P., Sprangers, M., & Bjorner, J.B. (2006). Multidimensional computerized adaptive testing of the EORTC QLQ-C30: basic developments and evaluations. Quality Life Research, 15, 315-329.
  • Rasch, G. (1960). Probabilistic Models for Some Intelligence and Attainment Tests. Copenhagen, Denmark.
  • Rebollo, P., Castejón, I. , Cuervo, J., Villa, G., García-Cueto, E., Díaz-Cuervo, E., . . . Alonso, J. (2010). Validation of a computer-adaptive test to evaluate generic health-related quality of life. Health and Quality of Life Outcomes, 8, 147.
  • Reckase, M. D. (1979). Unifactor latent trait models applied to multifactor tests: Results and implications. Journal of Educational Statistics, 4, 207-230.
  • Revicki, D.A., & Cella, D.F. (1997). Health status assessment for the twenty-first century: item response theory, item banking and computer adaptive testing. Quality Life Research, 6, 595-600.
  • Rojas-Tejada, A. J., & Pérez-Meléndez, G. (2001). Nuevos modelos para la medición de actitudes. [New models for measuring attitudes]. Valencia: Promolibro.
  • Samejima, F. (1969). Estimation of ability using a response pattern of graded scores. Psychometrika Monograph 17.
  • Samejima, F. (1974). Normal ogive model for the continuous response level in the multidimensional latent space. Psychometrika, 39, 111-121.
  • Smits, N., Cuijpers, P., & van Straten, A. (2011). Applying computerized adaptive testing to the CES-D scale: A simulation study. Psychiatry Research, 188, 147-155.
  • Schnake, M.E. (1983). An empirical assessment of the effects of affective response in the measurement of organizational climate. Personnel Psychology, 36, 791-807.
  • Steel, R.P., Shane, G.S., & Kennedy, K.A. (1990). Effects of social system factors on absenteeism, turnover, and job performance. Journal of Business and Psychology, 4, 423-430.
  • Thissen, D. (1991). MULTILOG, multiple categorical item analysis and test scoring using Item Response Theory [Computer program]. Chicago, Scientific Software Inc.
  • Thumin, F.J., & Thumin, L.J. (2011). The measurement and interpretation of organizational climate. The Journal of Psychology, 145, 93-109.
  • Van der Linden, W. J. and C. A. W. Glas, Eds. (2001). Computerized Adaptive Testing: Theory and Practice. Dordrecht, The Netherlands: Kluwer Academic Publishers.
  • Vartia, M. (2008). The sources of bullying: Psychological work environment and organizational climate. European Journal of Work and Organizational Psychology, 5, 203-214.
  • Wainer, H.W., Dorans, N.J., Flaugher, R., Green, B.F., Mislevy, R.J., Steinberg, L., & Thissen, D. (1990). Computerized adaptive testing: A primer. Computerized adaptive testing: A primer. Hillsdale, NJ: Erlbaum.
  • Walter, O.B., Becker, J., Bjorner, J.B., Fliege, H., Klapp, B.F., & Rose, M. (2007). Development and evaluation of a computer adaptive test for ‘Anxiety’ (Anxiety-CAT). Quality Life Research, 16(Suppl 1), 143-155.
  • Walter, O. B. (2010). Adaptive Tests for measuring Anxiety and Depression. In W. J. van der Linden and A. W. G. Cees (Ed), Elements of Adaptive Testing, Statistics for Social and Behavioral Sciences (pp.123-136). New York Dordrecht Heidelberg London, Springer.
Data source: Dialnet