Assessment content and validity of medical students′ selection tools in an international perspective

doi:10.3760/cma.j.cn115259-20240131-00112

Abstract

Abstract: A variety of selection tools are used in undergraduate medical education admissions processes internationally. Through literature review and information gathering, it is found that the most widely used types of selection tools are academic records, aptitude tests, references and personal statements, interviews and multiple mini-interviews, and situational judgement tests. From the perspective of evaluation content, these selection tools each have their own emphasis and can be summarized as tools for assessing cognitive and non-cognitive abilities. From the perspective of effectiveness, cognitive ability assessment tools are better predictors of students′ academic performance and clinical examination results after enrollment. On the other hand, non-cognitive ability assessment tools are more effective in predicting later professional behavior, attrition rates, and diversity within the physician community. Based on research conclusions and combined with local context, it is suggested that China should learn from diversified international selection and evaluation tools for medical students, strengthen research on medical student selection tools and prediction effects, and construct a more scientific and reasonable medical student selection and evaluation system.

Key words: Students, medical, Cognitive ability assessment tools, Non-cognitive ability assessment tools, Assessment content, Predictive validity

CLC Number:

Zhong Yuxin, Wang Dan, Wu Hongbin. Assessment content and validity of medical students′ selection tools in an international perspective[J]. Chinese Journal of Medical Education, 2025, 45(6): 401-407, DOI: 10.3760/cma.j.cn115259-20240131-00112.

References

[1] Flexner A. Medical education in the United States and Canada: a report to the Carnegie Foundation for the advancement of teaching[R]. New York: Carnegie Foundation for the Advancement of Teaching, 1910.
[2] Kratzke I, Kapadia MR, Egawa F, et al.Medical student selection[J]. Surg Clin North Am, 2021,101(4):635-652. DOI:10.1016/j.suc.2021.05.010.
[3] de Visser M, Fluit C, Cohen-Schotanus J, et al.The effects of a non-cognitive versus cognitive admission procedure within cohorts in one medical school[J]. Adv Health Sci Educ Theory Pract, 2018,23(1):187-200. DOI:10.1007/s10459-017-9782-1.
[4] Patterson F, Knight A, Dowell J, et al. How effective are selection methods in medical education? A systematic review[J]. Med Educ, 2016,50(1):36-60. DOI:10.1111/medu.12817.
[5] Cunha F, Heckman J,Schennach S. Estimating the technology of cognitive and noncognitive skill formation[J]. Econometrica, 2010,78(3):883-931. DOI:10.3982/ECTA6551.
[6] Schreurs S,Cleutjens K, Cleland J, et al. Outcomes-based selection into medical school: predicting excellence in multiple competencies during the clinical years[J]. Acad Med, 2020,95(9):1411-1420. DOI:10.1097/ACM.0000000000003279.
[7] Greatrix R, Dowell J. UKCAT and medical student selection in the UK - what has changed since 2006?[J]. BMC Med Educ, 2020,20(1):292. DOI:10.1186/s12909-020-02214-1.
[8] Patterson F, Ferguson E,Zibarras, L. Selection into medical education and training.//Swanwick T, Forrest K, O′Brien BC.(ed.). Understanding medical education: evidence, theory, and practice[M]. Third edition. Oxford: Wiley-Blackwell&The Association for the Study of Medical Education, 2018: 375-388.
[9] Association of American Medical Colleges. Using MCAT ^® Data in 2023 medical student selection[R]. Washington, DC: Association of American Medical Colleges, 2022. https://www.aamc.org/media/18901/download.
[10] UK Clinical Aptitude Test Consortium. About the UCAT[EB/OL].[2023-06-01]. https://www.ucat.ac.uk/about-ucat/.
[11] Cambridge Assessment Admissions Testing. What is BMAT?[EB/OL].[2023-06-01]. https://www.admissionstesting.org/for-test-takers/bmat/.
[12] Australian Council for Educational Research. About GAMSAT[EB/OL].[2023-06-01]. https://gamsat.acer.org/about-gamsat.
[13] Kelly ME, O′Flynn S. The construct validity of HPAT-Ireland for the selection of medical students: unresolved issues and future research implications[J]. Adv Health Sci Educ Theory Pract, 2017,22(2):267-286. DOI:10.1007/s10459-016-9728-z.
[14] Meyer H, Zimmermann S,Hissbach J, et al. Selection and academic success of medical students in Hamburg, Germany[J]. BMC Med Educ, 2019,19(1):23. DOI:10.1186/s12909-018-1443-4.
[15] Busche K, Elks ML, Hanson JT, et al.The validity of scores from the new MCAT exam in predicting student performance: results from a multisite study[J]. Acad Med, 2020,95(3):387-395. DOI:10.1097/ACM.0000000000002942.
[16] Eva KW, Rosenfeld J, Reiter HI, et al. An admissions OSCE: the multiple mini-interview[J]. Med Educ, 2004,38(3):314-326. DOI:10.1046/j.1365-2923.2004.01776.x.
[17] McCarthy JM, Goffin RD. Improving the validity of letters of recommendation: an investigation of three standardized reference forms[J]. Mil Psychol, 2001, 13(4):199-222. DOI:10.1207/S15327876MP1304_2.
[18] Urlings-Strop LC, Stegers-Jager KM, Stijnen T, et al. Academic and non-academic selection criteria in predicting medical school performance[J]. Med Teach, 2013,35(6):497-502. DOI:10.3109/0142159X.2013.774333.
[19] Walsh K. 牛津医学教育教科书[M]. 王维民,主译. 北京:北京大学医学出版社, 2022: 437-464.
[20] Rosenfeld JM, Reiter HI, Trinh K, et al. A cost efficiency comparison between the multiple mini-interview and traditional admissions interviews[J]. Adv Health Sci Educ TheoryPract, 2008,13(1):43-58. DOI:10.1007/s10459-006-9029-z.
[21] Gröne O, Mielke I, Knorr M, et al. Associations between communication OSCE performance and admission interviews in medical education[J]. Patient Educ Couns, 2022,105(7):2270-2275. DOI:10.1016/j.pec.2021.11.005.
[22] Fröhlich M, Kahmann J,Kadmon M. Development and psychometric examination of a German video-based situational judgment test for social competencies in medical school applicants[J]. Int J Sel Assess, 2017(25):94-110. DOI:https://doi.org/10.1111/ijsa.12163.
[23] Amos AJ, Lee K, Sen Gupta T, et al.Systematic review of specialist selection methods with implications for diversity in the medical workforce[J]. BMC Med Educ, 2021,21(1):448. DOI:10.1186/s12909-021-02685-w.
[24] Mussel P,Gatzka T, Hewig J. Situational judgment tests as an alternative measure for personality assessment[J]. European Journal of Psychological Assessment, 2018, 34: 328-335. DOI:https://doi.org/10.1027/1015-5759/a000346.
[25] UK Clinical Aptitude Test Consortium. UCAT testformat[EB/OL].[2023-06-01]. https://www.ucat.ac.uk/about-ucat/test-format/.
[26] Powis D, Munro D, Bore M, et al.Why is it so hard to consider personal qualities when selecting medical students?[J]. Med Teach, 2020,42(4):366-371. DOI:10.1080/0142159X.2019.1703919.
[27] Kelly ME, Regan D, Dunne F, et al. To what extent does the Health Professions Admission Test-Ireland predict performance in early undergraduate tests of communication and clinical skills? An observational cohort study[J]. BMC Med Educ, 2013,13:68. DOI:10.1186/1472-6920-13-68.
[28] Lievens F, Sackett PR, De Corte W. Weighting admission scores to balance predictiveness-diversity:he Pareto-optimization approach[J]. Med Educ, 2022,56(2):151-158. DOI:10.1111/medu.14606.
[29] Bala L, Pedder S, Sam AH, et al. Assessing the predictive validity of the UCAT-a systematic review and narrative synthesis[J]. Med Teach, 2022,44(4):401-409. DOI:10.1080/0142159X.2021.1998401.
[30] O′Neill L, Hartvigsen J,Wallstedt B, et al. Medical school dropout--testing at admission versus selection by highest grades as predictors[J]. Med Educ, 2011,45(11):1111-1120. DOI:10.1111/j.1365-2923.2011.04057.x.
[31] Salem RO, Al-Mously N, AlFadil S, et al. Pre-admission criteria and pre-clinical achievement: can they predict medical students performance in the clinical phase?[J]. Med Teach, 2016,38 Suppl 1:S26-30. DOI:10.3109/0142159X.2016.1142511.
[32] Paton LW, Tiffin PA, Smith D, et al. Predictors of fitness topractise declarations in UK medical undergraduates[J]. BMC Med Educ, 2018,18(1):68. DOI:10.1186/s12909-018-1167-5.
[33] Adam J, Bore M, Childs R, et al.Predictors of professional behaviour and academic outcomes in a UK medical school: A longitudinal cohort study[J]. Med Teach, 2015,37(9):868-880. DOI:10.3109/0142159X.2015.1009023.
[34] Garrud P, McManus IC. Impact of accelerated, graduate-entry medicine courses: a comparison of profile, success, and specialty destination between graduate entrants to accelerated or standard medicine courses in UK[J]. BMC Med Educ, 2018,18(1):250. DOI:10.1186/s12909-018-1355-3.
[35] Association of American Medical Colleges. Changing the MCAT ^® exam[EB/OL].[2023-06-01]. https://students-residents.aamc.org/about-mcat-exam/changing-mcat-exam.
[36] Busche K, Elks ML, Hanson JT, et al.The validity of scores from the new MCAT exam in predicting student performance: results from a multisite study[J]. Acad Med, 2020,95(3):387-395. DOI:10.1097/ACM.0000000000002942.
[37] Ferguson E, James D, O′Hehir F, et al.Pilot study of the roles of personality, references, and personal statements in relation to performance over the five years of a medical degree[J]. BMJ, 2003,326(7386):429-432. DOI:10.1136/bmj.326.7386.429.
[38] Dore KL, Kreuger S, Ladhani M, et al.The reliability and acceptability of the Multiple Mini-Interview as a selection instrument for postgraduate admissions[J]. Acad Med, 2010,85(10 Suppl):S60-63. DOI:10.1097/ACM.0b013e3181ed442b.
[39] White J, Brownell K, Lemay JF, et al. ″What do they want me to say?″ The hidden curriculum at work in the medical school selection process: a qualitative study[J]. BMC Med Educ, 2012,12:17. DOI:10.1186/1472-6920-12-17.
[40] Amos AJ, Lee K, Sen Gupta T, et al.Systematic review of specialist selection methods with implications for diversity in the medical workforce[J]. BMC Med Educ, 2021,21(1):448. DOI:10.1186/s12909-021-02685-w.
[41] Posthuma RA, Morgeson FP, Campion MA.Beyond employment interview validity: a comprehensive narrative review of recent research and trends over time[J]. Pers Psycho, 2002, 55(1):1-81. DOI:https://doi.org/10.1111/j.1744-6570.2002.tb00103.x.
[42] Knorr M,Schwibbe A, Ehrhardt M, et al. Validity evidence for the Hamburg multiple mini-interview[J]. BMC Med Educ, 2018,18(1):106. DOI:10.1186/s12909-018-1208-0.
[43] Sahota GS,Taggar JS. The association between Situational Judgement Test(SJT) scores and professionalism concerns in undergraduate medical education[J]. Med Teach, 2020,42(8):937-943. DOI:10.1080/0142159X.2020.1772466.
[44] Husbands A, Rodgerson MJ, Dowell J, et al. Evaluating the validity of an integrity-based situational judgement test for medical school admissions[J]. BMC Med Educ, 2015,15:144. DOI:10.1186/s12909-015-0424-0.
[45] Rostom H, Watson R, Leaver L. Situational judgement tests: the role of coaching[J]. Med Educ, 2013,47(2):219. DOI:10.1111/medu.12070.
[46] Simon E, Walsh K, Paterson-Brown F, et al. Does a high ranking mean success in the Situational Judgement Test?[J]. Clin Teach, 2015,12(1):42-45. DOI:10.1111/tct.12239.
[47] Ahmed H, Rhydderch M, Matthews P. Can knowledge tests and situational judgement tests predict selection centre performance?[J]. Med Educ, 2012,46(8):777-784. DOI:10.1111/j.1365-2923.2012.04303.x.
[48] 于晨, 吴红斌, 吴美辰. 八年制医学博士的“弃医”选择——基于自我主导力理论的研究[J]. 中国高教研究,2020,(1):83-89. DOI:10.16298/j.cnki.1004-3667.2020.01.13.