Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Hidden item variance in multiple mini-interview scores

Hidden item variance in multiple mini-interview scores The extant literature has largely ignored a potentially significant source of variance in multiple mini-interview (MMI) scores by “hiding” the variance attributable to the sample of attributes used on an evaluation form. This potential source of hidden variance can be defined as rating items, which typically comprise an MMI evaluation form. Due to its multi-faceted, repeated measures format, reliability for the MMI has been primarily evaluated using generalizability (G) theory. A key assumption of G theory is that G studies model the most important sources of variance to which a researcher plans to generalize. Because G studies can only attribute variance to the facets that are modeled in a G study, failure to model potentially substantial sources of variation in MMI scores can result in biased estimates of variance components. This study demonstrates the implications of hiding the item facet in MMI studies when true item-level effects exist. An extensive Monte Carlo simulation study was conducted to examine whether a commonly used hidden item, person-by-station (p × s|i) G study design results in biased estimated variance components. Estimates from this hidden item model were compared with estimates from a more complete person-by-station-by-item (p × s × i) model. Results suggest that when true item-level effects exist, the hidden item model (p × s|i) will result in biased variance components which can bias reliability estimates; therefore, researchers should consider using the more complete person-by-station-by-item model (p × s × i) when evaluating generalizability of MMI scores. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Advances in Health Sciences Education Springer Journals

Hidden item variance in multiple mini-interview scores

Download PDF
27 pages

Loading next page...
 
/lp/springer-journals/hidden-item-variance-in-multiple-mini-interview-scores-QtkN6ESq3Z
Publisher
Springer Journals
Copyright
Copyright © 2016 by Springer Science+Business Media Dordrecht
Subject
Education; Medical Education
ISSN
1382-4996
eISSN
1573-1677
DOI
10.1007/s10459-016-9706-5
pmid
27544387
Publisher site
See Article on Publisher Site

Abstract

The extant literature has largely ignored a potentially significant source of variance in multiple mini-interview (MMI) scores by “hiding” the variance attributable to the sample of attributes used on an evaluation form. This potential source of hidden variance can be defined as rating items, which typically comprise an MMI evaluation form. Due to its multi-faceted, repeated measures format, reliability for the MMI has been primarily evaluated using generalizability (G) theory. A key assumption of G theory is that G studies model the most important sources of variance to which a researcher plans to generalize. Because G studies can only attribute variance to the facets that are modeled in a G study, failure to model potentially substantial sources of variation in MMI scores can result in biased estimates of variance components. This study demonstrates the implications of hiding the item facet in MMI studies when true item-level effects exist. An extensive Monte Carlo simulation study was conducted to examine whether a commonly used hidden item, person-by-station (p × s|i) G study design results in biased estimated variance components. Estimates from this hidden item model were compared with estimates from a more complete person-by-station-by-item (p × s × i) model. Results suggest that when true item-level effects exist, the hidden item model (p × s|i) will result in biased variance components which can bias reliability estimates; therefore, researchers should consider using the more complete person-by-station-by-item model (p × s × i) when evaluating generalizability of MMI scores.

Journal

Advances in Health Sciences Education – Springer Journals

Published: Aug 20, 2016

References

  • Generalizability theory
    Brennan, RL
  • Generalizability of scaling gradient on direct behavior ratings
    Chafouleas, SM; Christ, TJ; Riley-Tillman, TC
  • Research methods in education
    Cohen, L; Manion, L; Morrison, K
  • I’m pickin’ up good regressions’: the governance of generalizability analyses
    Crossley, J; Russell, J; Jolly, B; Rickets, C; Roberts, C; Schuwirth, L; Norcini, J
  • Impact and causes of rater severity/leniency in appraisals without postevaluation communication between raters and rates
    Dewberry, C; Davies-Muir, A; Newell, S
  • The multiple mini-interview: How long is long enough?
    Dodson, M; Crotty, B; Prideaux, D; Carne, R; Ward, A; Leeuw, E
  • The multiple mini-interview in the UK context: 3 years of experience at Dundee
    Dowell, J; Lynch, B; Till, H; Kumwenda, B; Husbands, A
  • The interview in the admission process
    Edwards, JC; Johnson, EK; Molidor, JB
  • Predictive validity of the multiple mini-interview for selecting medical trainees
    Eva, KW; Reiter, HI; Trinh, K; Wasi, P; Rosenfeld, J; Norman, GR
  • An admissions OSCE: The multiple mini-interview
    Eva, KW; Rosenfeld, J; Reiter, HI; Norman, GR
  • Implications of three causal-models for the measurement of halo error
    Fisicaro, SA; Lance, CE
  • The effectiveness of academic admission interviews: An exploratory meta-analysis
    Goho, J; Blackman, A
  • The multiple mini-interview for selection of international medical graduates into family medicine residency education
    Hofmeister, M; Lockyer, J; Crutcher, R
  • Multilevel analysis: techniques and applications
    Hox, JJ
  • Robust Statistics
    Huber, PJ; Ronchetti, EM
  • Resolving the assessment center construct validity problem (as we know it)
    Kuncel, NR; Sackett, PR
  • Assessment of non-cognitive traits through the admissions multiple mini-interview
    Lemay, J; Lockyer, JM; Collin, VT; Brownell, AKW
  • Introduction to the Monte-Carlo method
    Manno, I
  • Industrial psychology
    McCormick, EJ; Ilgen, DR
  • A statistical model for the analysis of ordinal level dependent variables
    McKelvey, RD; Zavoina, W
  • The value and role of the interview in the student admissions process: A review
    Morris, JG
  • The multiple mini-interview (MMI) for student selection in health professions training—a systematic review
    Pau, A; Jeevaratnam, K; Chen, YS; Abdoul, AF; Khoo, C; Nadarajah, VD
  • Hierarchical linear models: applications and data analysis methods
    Raudenbush, SW; Bryk, AS
  • Reliability and validity of admissions tools used to select students for the health professions
    Salvatori, P
  • Variance components
    Searle, SR; Casella, G; McCulloch, CE
  • Psychometric properties of the multiple mini-interview used for medical admissions: Findings from generalizability and Rasch analyses
    Sebok, SS; Luu, K; Klinger, DA
  • Generalizability theory: A primer
    Shavelson, RJ; Webb, NM
  • Bootstrap estimates of standard error in generalizability theory
    Tong, Y; Brennan, RL
  • Enhancing the reliability of the multiple mini-interview for selecting prospective health care leaders
    Uijtdehaage, S; Doyle, LH; Parker, N
  • Variance in attributes assessed by the multiple mini-interview
    Zaidi, NB; Swoboda, C; Wang, LL; Manuel, RS