基于概化理论的临床医学专业(本科)水平测试临床基本技能考试质量分析

doi:10.3760/cma.j.cn115259-20221114-01427

中华医学教育杂志 ›› 2023, Vol. 43 ›› Issue (5): 391-396.DOI: 10.3760/cma.j.cn115259-20221114-01427

基于概化理论的临床医学专业(本科)水平测试临床基本技能考试质量分析

周文静¹, 江哲涵², 欧阳劲樱¹, 王维民³

¹北京大学公共卫生学院卫生政策与管理学系 100191;
²北京大学医学教育研究所全国医学教育发展中心 100191;
³北京大学医学部 100191

收稿日期:2022-11-14 出版日期:2023-05-01 发布日期:2023-05-05
通讯作者: 王维民, Email: wwm@bjmu.edu.cn
基金资助:
国家自然科学基金青年项目(72104006)

Quality analysis of the clinical fundamental skills test for standardized competence test for clinical medicine undergraduates using generalizability theory

Zhou Wenjing¹, Jiang Zhehan², Ouyang Jinying¹, Wang Weimin³

¹Department of Health Policy and Management, School of Public Health, Peking University, Beijing 100191, China;
²Institute of Medical Education & National Center for Health Professions Education Development, Peking University, Beijing 100191, China;
³Peking University Health Science Center, Beijing 100191, China

Received:2022-11-14 Online:2023-05-01 Published:2023-05-05
Contact: Wang Weimin, Email: wwm@bjmu.edu.cn
Supported by:
National Natural Science Foundation Youth Project (72104006)

摘要/Abstract

摘要： 目的评价临床医学专业(本科)水平测试技能考试信度,为改进技能考试质量提供依据。方法应用概化理论对2020年完成临床医学专业(本科)水平测试技能考试的7 322名考生的成绩进行研究。根据考试设计,纳入的因素包括考生、考站和考题,计算各因素的方差百分比,分析影响考试信度的主要因素,计算考试的概括力系数和可靠性指数分析考试的信度。基于完成同一套试题的考生分数,通过调整考题和考站数量,得出不同概括力系数和可靠性指数以探讨考核方案的优化。结果考题的方差百分比为70.0%~76.0%,考站的方差百分比为17.6%~24.4%,考生的方差百分比为0.3%~0.8%。概括力系数范围为0.502~0.717,可靠性指数范围为0.052~0.223。当考站数量不变,考题数量增加1道时,概括力系数增加0.002,可靠性指数增加0.002,考题数量增加5道时,概括力系数增加0.007,可靠性指数增加0.004;当考站数量增加1站时,概括力系数增加0.025,可靠性指数增加0.010,考站数量增加2站时,概括力系数增加0.045,可靠性指数增加0.017。结论水平测试临床基本技能考试质量尚存在提升空间,考站与考题是影响测试信度的主要因素,通过增加考站与考题数量可以提高测试信度,但需综合考虑测试时间及考站、考题开发成本。

关键词: 医学生, 概化理论, 临床医学专业(本科)水平测试, 临床技能, 测试质量, 信度

Abstract: Objective To assess the reliability of the clinical fundamental skills test forstandardized competence test for clinical medicine undergraduates and provide an empirical basis for improving the quality of the test. Methods A generalizability theory was employed to examine the scores of 7 322 examinees who completed the clinical fundamental skills test for BSc in 2020. According to the test design, examinees, stations and items were included in the analysis.The percentage of variance for each facet was calculated to identify the main sources affecting test reliability. Generalizability (G) coefficient and phi index were calculated to analyze the reliability of the test. Based on the scores of examinees who completed the same items in the test, different G coefficients and phi indices were obtained to explore the optimization of the test reliability by adjusting the number of items and stations. Results The results from the generalizability theory showed that the percentage of variance for items was 70.0%~76.0%, the percentage of variance for stations was 17.6%~24.4%, and the percentage of variance for examinees was 0.3%~0.8%. G coefficients for the test ranged from 0.502 to 0.717, and phi indices ranged from 0.052 to 0.223. When one item was added, G coefficient increased by 0.002 and phi index increased by 0.002. When five items were added, G coefficient increased by 0.007 and phi index increased by 0.004. When one station was added, G coefficient increased by 0.025, and phi index increased by 0.010. When two stations were added, G coefficient increased by 0.045 and phi index increased by 0.017. Conclusions There is still room for improving the quality of clinical fundamental skills test. The stations and items are the main sources affecting the test reliability. The test reliability can be improved by increasing the number of stations and items, but the time and the cost of stations and items need to be considered in a comprehensive manner.

Key words: Students, medical, Generalizability theory, Standardized competence test for clinical medicine undergraduates, Clinical skills, Test quality, Reliability

中图分类号:

周文静, 江哲涵, 欧阳劲樱, 王维民. 基于概化理论的临床医学专业(本科)水平测试临床基本技能考试质量分析[J]. 中华医学教育杂志, 2023, 43(5): 391-396.

Zhou Wenjing, Jiang Zhehan, Ouyang Jinying, Wang Weimin. Quality analysis of the clinical fundamental skills test for standardized competence test for clinical medicine undergraduates using generalizability theory[J]. Chinese Journal of Medical Education, 2023, 43(5): 391-396.

参考文献

[1] 临床医学专业(本科)水平测试简介.国家医学考试网.[EB/OL]. (2021-06-03) [2021-06-03].https://www.nmec.org.cn/Pages/ArticleInfo-13-11592.html.
[2] Halman S, Fu A, Pugh D. Entrustment within an objective structured clinical examination (OSCE) progress test: bridging the gap towards competency-based medical education[J]. Med Teach, 2020,42(11):1283-1288. DOI: 10.1080/0142159X.2020.1803251.
[3] Oliveira JM, Spositon T, Cerci Neto A, et al. Functional tests for adults with asthma: validity, reliability, minimal detectable change, and feasibility[J]. J Asthma, 2022,59(1):169-177. DOI: 10.1080/02770903.2020.1838540.
[4] Bilgic E, Watanabe Y, McKendy KM, et al. Reliable assessment of performance in surgery: a practical approach to generalizability theory[J]. J Surg Educ, 2015,72(5):774-775. DOI: 10.1016/j.jsurg.2015.04.020.
[5] Brennan RL. Generalizability theory[M]. New York: Springer-Verlag, 2001:53-140.
[6] Andersen S, Nayahangan LJ, Park YS, et al. Use of generalizability theory for exploring reliability of and sources of variance in assessment of technical skills: a systematic review and meta-analysis[J]. Acad Med, 2021,96(11):1609-1619. DOI: 10.1097/ACM.0000000000004150.
[7] Clauser BE, Balog K, Harik P, et al. A multivariate generalizability analysis of history-taking and physical examination scores from the USMLE step 2 clinical skills examination[J]. Acad Med, 2009,84(10 Suppl):86-89. DOI: 10.1097/ACM.0b013e3181b36fda.
[8] 于惊涛. 我国临床执业医师资格考试的信度和效度研究[J].中国卫生事业管理,2008,25(5):349-350,354. DOI: 10.3969/j.issn.1004-4663.2008.05.029.
[9] 高超, 张革来, 王震, 等. 临床执业医师考试质量评价——以浙江考区为例[J].浙江医学教育,2021,20(2):4-6,22. DOI: 10.3969/j.issn.1672-0024.2021.02.002.
[10] Monteiro S, Sullivan GM, Chan TM. Generalizability theory made simple(r): an introductory primer to G-studies[J]. J Grad Med Educ, 2019,11(4):365-370. DOI: 10.4300/JGME-D-19-00464.1.
[11] Li G, Xie J, An L, et al. A generalizability analysis of the mobile phone addiction tendency scale for Chinese college students[J]. Front Psychiatry, 2019,10:241. DOI: 10.3389/fpsyt.2019.00241.
[12] Peeters MJ, Cor MK. Guidance for high-stakes testing within pharmacy educational assessment[J]. Curr Pharm Teach Learn, 2020,12(1):1-4. DOI: 10.1016/j.cptl.2019.10.001.
[13] 邹扬, 缪青, 芦开芳, 等. 本科和长学制毕业考试中客观结构化临床考试的应用[J].上海交通大学学报(医学版),2008, 28(z1): 71-75.
[14] Corrie K, Wiles M, Flack J, et al. A novel objective structured clinical examination for the assessment of transfusion practice in anaesthesia[J]. Clin Teach, 2011,8(2):97-100. DOI: 10.1111/j.1743-498X.2010.00404.x.
[15] Brannick MT, Erol-Korkmaz HT, Prewett M. A systematic review of the reliability of objective structured clinical examination scores[J]. Med Educ, 2011,45(12):1181-1189. DOI: 10.1111/j.1365-2923.2011.04075.x.
[16] Norman G, Bordage G, Page G, et al. How specific is case specificity?[J]. Med Educ, 2006,40(7):618-623. DOI: 10.1111/j.1365-2929.2006.02511.x.
[17] Clauser BE, Harik P, Margolis MJ, et al. The generalizability of documentation scores from the USMLE Step 2 Clinical Skills examination[J]. Acad Med, 2008,83(10 Suppl):S41-44. DOI: 10.1097/ACM.0b013e318183cd1d.
[18] Dory V, Gagnon R, Charlin B. Is case-specificity content-specificity? an analysis of data from extended-matching questions[J]. Adv Health Sci Educ Theory Pract, 2010,15(1):55-63. DOI: 10.1007/s10459-009-9169-z.
[19] Tsai TH, Shin CD, Neumann LM, et al. Generalizability analyses of NBDE Part II[J]. Eval Health Prof, 2012,35(2):169-181. DOI: 10.1177/0163278711425382.

基于概化理论的临床医学专业(本科)水平测试临床基本技能考试质量分析

Quality analysis of the clinical fundamental skills test for standardized competence test for clinical medicine undergraduates using generalizability theory

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	马兴彬, 史宁, 李彦敏, 于飞, 王伟, 胡营滨, 高桃, 胡宝光, 牛琼, 刘成霞. 基于成果导向教育理念的理实一体化教学方法在消化系统见习教学中的应用[J]. 中华医学教育杂志, 2023, 43(3): 199-202.
[2]	汪洁英, 李雪玲, 陈水云, 权戴, 曹莹, 王雪梅, 马永红, 门可. 虚拟仿真实验教学对医学生学习效果的影响研究[J]. 中华医学教育杂志, 2023, 43(3): 203-207.
[3]	吴妍平, 王立新, 王云文, 封秀娟, 卢羽彤, 张彦芳, 孙丽丽, 徐仁华. 中文版社区护理感知量表在职业院校护理学生中应用的信效度检验[J]. 中华医学教育杂志, 2023, 43(3): 236-240.
[4]	朱梓凝, 唐敏, 杨乔越, 张月映湖, 戚嘉婷, 施正清, 赵梅珍, 臧爽, 曾铁英. 医学生医学相关课程课堂学习偏差现象的质性研究[J]. 中华医学教育杂志, 2023, 43(2): 95-99.
[5]	靖颐雯韩叶. 课程思政融入军队医学院校英语教学的策略与反思[J]. 中华医学教育杂志, 2023, 43(2): 100-104.
[6]	高嘉徽, 刘晶晶, 洪夏芸, 严乾文, 郑文昊, 黄显清. 医学院校开展灾害医学救援培训的现状与思考[J]. 中华医学教育杂志, 2023, 43(1): 10-14.
[7]	郑方圆, 曾超美. 思维导图在八年制医学生新生儿疾病教学中的应用[J]. 中华医学教育杂志, 2023, 43(1): 15-18.
[8]	邓巍巍, 高玙, 靳梦蝶, 宋琰萍, 陈楠. 基于案例学习在儿科物理治疗学教学中的应用[J]. 中华医学教育杂志, 2023, 43(1): 23-26.
[9]	叶子兴, 雷曙槟, 伍晓倩, 纪志刚, 肖河. 医学生对外科教学巡诊需求的调查和分析[J]. 中华医学教育杂志, 2023, 43(1): 35-38.
[10]	罗昕泠, 詹剑. 麻醉学专业本科生实习期间麻醉法律风险认知的现状分析[J]. 中华医学教育杂志, 2023, 43(1): 39-41.
[11]	周鑫, 陶明, 崔龙, 王行雁, 张利. 器官系统整合课程教学模式在八年制临床医学专业学生消化系统临床见习中的应用[J]. 中华医学教育杂志, 2022, 42(9): 769-773.
[12]	张鹏程, 王小路, 李春. 黑龙江省5所高校医学生劳动素养及其影响因素研究[J]. 中华医学教育杂志, 2022, 42(9): 778-782.
[13]	翟晓艳, 臧玉芹, 井朋, 张石盼, 刘敬, 倪爱华. 新型冠状病毒肺炎疫情防控期间医学生心理状况及其影响因素分析[J]. 中华医学教育杂志, 2022, 42(9): 787-791.
[14]	李旭昶, 陈祥发, 刘博, 汪璐芸, 赖金胜, 项帅, 姚飞. 医学生视域下基于多学科诊疗的医学概论课程翻转课堂应用效果评价[J]. 中华医学教育杂志, 2022, 42(9): 796-800.
[15]	张勇, 常宁, 邱丹, 董沫含, 张艰. 临床思维导向式教学方式在呼吸与危重症医学科临床实习教学中的应用[J]. 中华医学教育杂志, 2022, 42(9): 801-805.