Access the full text.
Sign up today, get DeepDyve free for 14 days.
Y. Park, A. Hyderi, N. Heine, Win May, A. Nevins, Ming Lee, G. Bordage, R. Yudkowsky (2017)
Validity Evidence and Scoring Guidelines for Standardized Patient Encounters and Patient Notes From a Multisite Study of Clinical Performance Examinations in Seven Medical SchoolsAcademic Medicine, 92
R. Mink, A. Schwartz, B. Herman, D. Turner, M. Curran, A. Myers, Deborah Hsu, J. Kesselheim, C. Carraccio (2017)
Validity of Level of Supervision Scales for Assessing Pediatric Fellows on the Common Pediatric Subspecialty Entrustable Professional ActivitiesAcademic Medicine, 93
M. Govaerts, C. Vleuten, L. Schuwirth, A. Muijtjens (2007)
Broadening Perspectives on Clinical Performance Assessment: Rethinking the Nature of In-training AssessmentAdvances in Health Sciences Education, 12
S. Weingart, O. Yaghi, Matthew Wetherell, M. Sweeney (2018)
Measuring Medical Housestaff Teamwork Performance Using Multiple Direct Observation Instruments: Comparing Apples and ApplesAcademic Medicine, 93
K. Gergen, Ezekiel Dixon-Román (2014)
Social Epistemology and the Pragmatics of AssessmentTeachers College Record: The Voice of Scholarship in Education, 116
T. Greenhalgh, S. Thorne, K. Malterud (2018)
Time to challenge the spurious hierarchy of systematic over narrative reviews?European Journal of Clinical Investigation, 48
NK Denzin, YS Lincoln, NK Denzin, YS Lincoln (2008)
Introduction: The discipline and practice of qualitative researchStrategies of qualitative inquiry
Benjamin Kinnear, E. Warm, K. Hauer (2018)
Twelve tips to maximize the value of a clinical competency committee in postgraduate medical educationMedical Teacher, 40
D. Borsboom, Keith Markus (2013)
Truth and Evidence in Validity TheoryJournal of Educational Measurement, 50
M. Govaerts, bullet Wiel, bullet Schuwirth, bullet Vleuten, bullet Muijtjens, C. Vleuten, A. Muijtjens, M. Wiel, L. Schuwirth (2012)
Workplace-based assessment: raters’ performance theories and constructsAdvances in Health Sciences Education, 18
Su-Ting Li, D. Tancredi, A. Schwartz, Ann Guillot, A. Burke, Trimm Rf, Susan Guralnick, J. Mahan, K. Gifford (2018)
Identifying Gaps in the Performance of Pediatric Trainees Who Receive Marginal/Unsatisfactory RatingsAcademic Medicine, 93
A. Gingerich, S. Ramlo, C. Vleuten, K. Eva, G. Regehr (2016)
Inter-rater variability as mutual disagreement: identifying raters’ divergent points of viewAdvances in Health Sciences Education, 22
W. Tavares, R. Brydges, P. Myre, J. Prpic, Linda Turner, R. Yelle, Maud Huiskamp (2017)
Applying Kane’s validity framework to a simulation based assessment of clinical competenceAdvances in Health Sciences Education, 23
N. Denzin, Y. Lincoln (2005)
The Discipline and Practice of Qualitative Research
M. Govaerts, C. Vleuten (2013)
Validity in work‐based assessment: expanding our horizonsMedical Education, 47
M. Christensen, E. Lykkegaard, Ole Lund, L. O’Neill (2018)
Qualitative analysis of MMI raters’ scorings of medical school candidates: A matter of taste?Advances in Health Sciences Education, 23
P. Hicks, M. Margolis, Sue Poynter, Christa Chaffinch, Rebecca Tenney-Soeiro, T. Turner, L. Waggoner-Fountain, R. Lockridge, S. Clyman, A. Schwartz (2016)
The Pediatrics Milestones Assessment Pilot: Development of Workplace-Based Assessment Content, Instruments, and ProcessesAcademic Medicine, 91
V. Dory, C. Gomez-Garibello, R. Cruess, Sylvia Cruess, B. Cummings, M. Young (2018)
The challenges of detecting progress in generic competencies in the clinical settingMedical Education, 52
A. Cameron, L. Mackeigan, N. Mitsakakis, J. Pugsley (2017)
Multiple mini‐interview predictive validity for performance on a pharmacy licensing examinationMedical Education, 51
A. Kuper, S. Reeves, Mathieu Albert, B. Hodges (2007)
Assessment: do we need to broaden our methodological horizons?Medical Education, 41
M. Crotty (1998)
The Foundations of Social Research: Meaning and Perspective in the Research Process
J. Norcini, Brownell Anderson, V. Bollela, V. Burch, M. Costa, R. Duvivier, R. Galbraith, R. Hays, A. Kent, Vanessa Perrott, T. Roberts (2011)
Criteria for good assessment: Consensus statement and recommendations from the Ottawa 2010 ConferenceMedical Teacher, 33
F. Naumann, S. Marshall, B. Shulruf, Philip Jones (2016)
Exploring examiner judgement of professional competence in rater based assessmentAdvances in Health Sciences Education, 21
W. Tavares, M. Young, Geneviève Gauthier, Christina St-Onge (2019)
The Effect of Foregrounding Intended Use on Observers’ Ratings and Comments in the Assessment of Clinical CompetenceAcademic Medicine, 95
M. Kane (2013)
Validation as a Pragmatic, Scientific ActivityJournal of Educational Measurement, 50
S. Downing, R. Yudkowsky (2009)
Assessment in Health Professions Education
Timothy Wood, J. Chan, S. Humphrey-Murto, Debra Pugh, C. Touchie (2016)
The influence of first impressions on subsequent ratings within an OSCE stationAdvances in Health Sciences Education, 22
D. Borsboom, J. Heerden, G. Mellenbergh (2003)
Validity and truth
Su-Ting Li, D. Tancredi, A. Schwartz, Ann Guillot, A. Burke, F. Trimm, Susan Guralnick, J. Mahan, K. Gifford (2017)
Competent for Unsupervised Practice: Use of Pediatric Residency Training Milestones to Assess ReadinessAcademic Medicine, 92
Morag Paton, A. Kuper, Elise Paradis, Zac Feilchenfeld, C. Whitehead (2020)
Tackling the void: the importance of addressing absences in the field of health professions education researchAdvances in Health Sciences Education, 26
D. Cook, A. Kuper, R. Hatala, S. Ginsburg (2016)
When Assessment Data Are Words: Validity Evidence for Qualitative Educational AssessmentsAcademic Medicine, 91
(2016)
The milestones guidebook. Accreditation Council for Graduate Medical Education
J. Schoenherr, S. Hamstra (2015)
Psychometrics and its discontents: an historical perspective on the discourse of the measurement traditionAdvances in Health Sciences Education, 21
Hirotaka Onishi, Y. Park, Ryo Takayanagi, Y. Fujinuma (2018)
Combining Scores Based on Compensatory and Noncompensatory Scoring Rules to Assess Resident Readiness for Unsupervised Practice: Implications From a National Primary Care Certification Examination in JapanAcademic Medicine, 93
O. Cate, G. Regehr (2019)
The Power of Subjectivity in the Assessment of Medical Trainees.Academic Medicine
C. O'Brien, Sandra Sanguino, John Thomas, Marianne Green (2016)
Feasibility and Outcomes of Implementing a Portfolio Assessment System Alongside a Traditional Grading SystemAcademic Medicine, 91
Keng-Lin Lee, Shih-Li Tsai, Y. Chiu, M. Ho (2016)
Can student self-ratings be compared with peer ratings? A study of measurement invariance of multisource feedbackAdvances in Health Sciences Education, 21
K. Dore, H. Reiter, Sharyn Kreuger, G. Norman (2017)
CASPer, an online pre-interview screen for personal/professional characteristics: prediction of national licensure scoresAdvances in Health Sciences Education, 22
M. Dixon-Woods, Shona Agarwal, David Jones, B. Young, A. Sutton (2005)
Synthesising qualitative and quantitative evidence: a review of possible methods.Journal of health services research & policy, 10 1
M. Sandelowski (2010)
What's in a name? Qualitative description revisited.Research in nursing & health, 33 1
D. Borsboom (2005)
Measuring the mind: Conceptual issues in contemporary psychometrics
K. Hauer, J. Vandergrift, R. Lipner, E. Holmboe, Sarah Hood, Furman McDonald (2018)
National Internal Medicine Milestone Ratings: Validity Evidence From Longitudinal Three-Year Follow-upAcademic Medicine, 93
C. Roberts, C. Jorm, Stacey Gentilcore, J. Crossley (2017)
Peer assessment of professional behaviours in problem‐based learning groupsMedical Education, 51
R. Mislevy (2018)
Sociocognitive Foundations of Educational Measurement
Y. Park, M. Lineberry, A. Hyderi, G. Bordage, Kuan Xing, R. Yudkowsky (2016)
Differential Weighting for Subcomponent Measures of Integrated Clinical Encounter Scores Based on the USMLE Step 2 CS Examination: Effects on Composite Score Reliability and Pass–Fail DecisionsAcademic Medicine, 91
J. Norcini, M. Anderson, V. Bollela, V. Burch, M. Costa, R. Duvivier, R. Hays, Maria Mackay, T. Roberts, David Swanson (2018)
2018 Consensus framework for good assessmentMedical Teacher, 40
S. Ginsburg, C. Vleuten, K. Eva (2017)
The Hidden Value of Narrative Comments for Assessment: A Quantitative Reliability Analysis of Qualitative DataAcademic Medicine, 92
E. Guba, Y. Lincoln (1994)
Competing paradigms in qualitative research.
Sathya Naidoo, Safiatu Lopes, F. Patterson, Helen Mead, Sheona Macleod (2017)
Can colleagues’, patients’ and supervisors’ assessments predict successful completion of postgraduate medical training?Medical Education, 51
M Sandelowski (2000)
Focus on research methods-whatever happened to qualitative description?Research in Nursing & Health, 23
Saad Chahine, Bruce Holmes, Z. Kowalewski (2015)
In the minds of OSCE examiners: uncovering hidden assumptionsAdvances in Health Sciences Education, 21
J. Pearce (2019)
In defence of constructivist, utility‐driven psychometrics for the ‘post‐psychometric era’Medical Education, 54
Andrea Pool, M. Govaerts, D. Jaarsma, E. Driessen (2017)
From aggregation to interpretation: how assessors judge complex data in a competency-based portfolioAdvances in Health Sciences Education, 23
S. Kvale (1995)
The Social Construction of ValidityQualitative Inquiry, 1
W. Tavares, A. Kuper, K. Kulasegaram, C. Whitehead (2019)
The compatibility principle: on philosophies in the assessment of clinical competenceAdvances in Health Sciences Education, 25
ROYAL COLLEGE OF PHYSICIANS AND SURGEONS OF CANADA.
Canadian Medical Association journal, 23 5
A. Kuper, P. Veinot, Jennifer Leavitt, S. Levitt, Amanda Li, J. Goguen, M. Schreiber, Lisa Richardson, C. Whitehead (2017)
Epistemology, culture, justice and power: non‐bioscientific knowledge for medical trainingMedical Education, 51
J. Pearce, W. Tavares (2021)
A philosophical history of programmatic assessment: tracing shifting configurationsAdvances in Health Sciences Education, 26
William Roberts, John Boulet, J. Sandella (2017)
Comparison study of judged clinical skills competence from standard setting ratings generated under different administration conditionsAdvances in Health Sciences Education, 22
F. Biagioli, D. Elliot, R. Palmer, Carla Graichen, R. Rdesinski, K. Kumar, Ari Galper, J. Tysinger (2017)
The Electronic Health Record Objective Structured Clinical Examination: Assessing Student Competency in Patient Interactions While Using the Electronic Health Record.Academic Medicine
N. Bajwa, R. Yudkowsky, D. Belli, N. Vu, Y. Park (2017)
Improving the residency admissions process by integrating a professionalism assessment: a validity and feasibility studyAdvances in Health Sciences Education, 22
D. Cook, R. Brydges, S. Ginsburg, R. Hatala (2015)
A contemporary approach to validity arguments: a practical guide to Kane's frameworkMedical Education, 49
M. Grant, A. Booth (2009)
A typology of reviews: an analysis of 14 review types and associated methodologies.Health information and libraries journal, 26 2
A. Tashakkori, C. Teddlie (2003)
Handbook of mixed methods in social & behavioral research
Keith Markus, D. Borsboom (2013)
Frontiers of Test Validity Theory: Measurement, Causation, and Meaning
B. Young (2000)
Postpositivism and Educational Research
T. Kuhn, David Hawkins (1963)
The Structure of Scientific RevolutionsAmerican Journal of Physics, 31
M. Sandelowski (2000)
Whatever happened to qualitative description?Research in nursing & health, 23 4
Robin Demuth, Jon Gold, B. Mavis, Dianne Wagner (2017)
Progress on a New Kind of Progress Test: Assessing Medical Students' Clinical Skills.Academic Medicine
A. Kuper, M. D'Eon (2011)
Rethinking the basis of medical knowledgeMedical Education, 45
B. Hodges (2013)
Assessment in the post-psychometric era: Learning to love the subjective and collectiveMedical Teacher, 35
J. Norcini (2019)
What's Next? Developing Systems of Assessment for Educational Settings.Academic Medicine
M. Ross, Ian Walker, L. Cooke, M. Raman, P. Ravani, S. Coderre, K. McLaughlin (2017)
Are Female Applicants Rated Higher Than Males on the Multiple Mini-Interview? Findings From the University of CalgaryAcademic Medicine, 92
T. Turner, Vasudha Bhavaraju, Ulana Luciw-Dubas, P. Hicks, Sara Multerer, Amanda Osta, Jennifer McDonnell, Sue Poynter, D. Schumacher, Rebecca Tenney-Soeiro, L. Waggoner-Fountain, A. Schwartz (2017)
Validity Evidence From Ratings of Pediatric Interns and Subinterns on a Subset of Pediatric Milestones.Academic Medicine
S. Martin, Keme Carter, Noah Hellermann, L. Glick, S. Ngooi, Marika Kachman, Jeanne Farnan, V. Arora (2018)
The Consultation Observed Simulated Clinical Experience: Training, Assessment, and Feedback for Incoming Interns on Requesting Consultations.Academic Medicine
J. Sherbino, J. Frank, L. Flynn, L. Snell (2011)
“Intrinsic Roles” rather than “armour”: renaming the “non-medical expert roles” of the CanMEDS framework to match their intentAdvances in Health Sciences Education, 16
Assessment practices have been increasingly informed by a range of philosophical positions. While generally beneficial, the addition of options can lead to misalignment in the philosophical assumptions associated with different features of assessment (e.g., the nature of constructs and competence, ways of assessing, validation approaches). Such incompatibility can threaten the quality and defensibility of researchers’ claims, especially when left implicit. We investigated how authors state and use their philosophical positions when designing and reporting on performance-based assessments (PBA) of intrinsic roles, as well as the (in)compatibility of assumptions across assessment features. Using a representative sample of studies examining PBA of intrinsic roles, we used qualitative content analysis to extract data on how authors enacted their philosophical positions across three key assessment features: (1) construct conceptualizations, (2) assessment activities, and (3) validation methods. We also examined patterns in philosophical positioning across features and studies. In reviewing 32 papers from established peer-reviewed journals, we found (a) authors rarely reported their philosophical positions, meaning underlying assumptions could only be inferred; (b) authors approached features of assessment in variable ways that could be informed by or associated with different philosophical assumptions; (c) we experienced uncertainty in determining (in)compatibility of philosophical assumptions across features. Authors’ philosophical positions were often vague or absent in the selected contemporary assessment literature. Leaving such details implicit may lead to misinterpretation by knowledge users wishing to implement, build on, or evaluate the work. As such, assessing claims, quality and defensibility, may increasingly depend more on who is interpreting, rather than what is being interpreted.
Advances in Health Sciences Education – Springer Journals
Published: Dec 1, 2021
Keywords: Assessment; Validity; Philosophical positions; Paradigms; Clinical competence
Read and print from thousands of top scholarly journals.
Already have an account? Log in
Bookmark this article. You can see your Bookmarks on your DeepDyve Library.
To save an article, log in first, or sign up for a DeepDyve account if you don’t already have one.
Copy and paste the desired citation format or use the link below to download a file formatted for EndNote
Access the full text.
Sign up today, get DeepDyve free for 14 days.
All DeepDyve websites use cookies to improve your online experience. They were placed on your computer when you launched this website. You can change your cookie settings through your browser.