Application of generative large language models in improving the quality of basic medical science examination items

doi:10.3760/cma.j.cn115259-20250729-00842

Abstract

Abstract: Objective To investigate the effectiveness of generative large language models (LLMs) in assisting the improvement of basic medical science examination item quality. Methods A total of 2 520 items from the ″Comprehensive Basic Medical Science Examination″ administered at Air Force Medical University from 2013 to 2024 were selected. Based on the correct response rate, discrimination index, and distractor selection rate, 64 problematic items were identified. Three sets of prompt engineering protocols were designed using the Tree of Thoughts framework, and 121 new items were generated through the ChatGPT-4o model. After manual refinement by an expert panel, 48 items were randomly selected and incorporated into an actual examination for field testing. The Wilcoxon signed-rank test was used to compare differences in correct response rates and discrimination indices, and Fisher′s exact test was used to compare differences in distractor selection rates. Results The mean correct response rate of the generated items increased from 0.218 1 to 0.444 2 (P<0.001), and the mean discrimination index increased from 0.082 1 to 0.146 9 (P=0.019). The abnormal distractor structure was significantly improved: the proportion of items with no distractor selection rate exceeding 30% increased from 27.1% (13/48) to 54.2% (26/48), and the proportion of items with no distractor selection rate below 5% decreased from 41.7% (20/48) to 33.3% (16/48), all differences were statistically significant (all P<0.05). A total of 72.7% (88/121) of the generated items required manual refinement, primarily involving option design deficiencies [44.3% (39/88)] and stem wording issues [30.7% (27/88)]. Conclusions LLMs demonstrate promising application potential in basic medical science item development. Standardized prompt engineering design combined with manual refinement processes can effectively improve item quality, although in-depth involvement of subject matter experts in quality control remains essential.

Key words: Artificial intelligence, Basic medical science, Test item development, Generative large language models

CLC Number:

Liu Kan, Zheng Shuyuan, Wang Zhenyu, Liu Zidong, Yang Hai, Tian Tian. Application of generative large language models in improving the quality of basic medical science examination items[J]. Chinese Journal of Medical Education, 2026, 46(8): 628-635, DOI: 10.3760/cma.j.cn115259-20250729-00842.

References

[1] 刘侃, 于菲, 王振禹, 等. 中日临床医学专业水平测试的比较研究与启示[J]. 中华医学教育探索杂志,2024,23(2):280-284. DOI:10.3760/cma.j.cn116021-20230626-01516.
[2] 王维民. 新科技革命背景下的医学教育范式转型[J]. 中华医学教育杂志,2024,44(6):401-406. DOI:10.3760/cma.j.cn115259-20240405-00352.
[3] 钟京谕, 星月, 胡扬帆, 等. 中文大语言模型在放射医学中级专业技术考试中的正确率和答案解析质量[J]. 中华医学教育探索杂志,2025,24(2):145-149. DOI:10.3760/cma.j.cn116021-20231111-01961.
[4] 江哲涵, 奉世聪, 王维民. 生成式大语言模型在医学考试题库建设中的实践探索[J]. 中华医学教育杂志,2024,44(8):561-569. DOI:10.3760/cma.j.cn115259-20240524-00520.
[5] Lee P, Goldberg C, Kohane I. 超越想象的GPT医疗[M]. 芦义,译. 杭州:浙江科学技术出版社,2023:59-95.
[6] 刘侃, 张帅, 郑淑园, 等. 以生理学试题为例探索基础医学综合测试的定量分析方法[J]. 中华医学教育杂志,2018,38(3):388-393. DOI:10.3760/cma.j.issn.1673-677X.2018.03.017.
[7] Yao SY, Yu D, Zhao J, et al. Tree of thoughts: deliberate problem solving with large language models[C/OL]//Advances in Neural Information Processing Systems 36: 37th Conference on Neural Information Processing Systems, NeurIPS 2023,New Orleans,[2025-07-28]. https://proceedings.neurips.cc/paper_files/paper/2023/hash/271db9922b8d1f4dd7aaef84ed5ac703-Abstract-Conference.html.
[8] 李国建, 向阳, 何惧, 等. 临床思维能力测评框架构建和测评系统设计[J]. 中华医学教育杂志,2020,40(7):565-568. DOI:10.3760/cma.j.cn115259-20191212-01074.
[9] 张东奇, 李国建, 孙金婉, 等. 医学考试中基础医学融合型试题与非融合型试题的比较研究[J]. 中华医学教育杂志,2021,41(4):351-354. DOI:10.3760/cma.j.cn115259-20201112-01570.
[10] 张学. 新医科人才培养的思考与探索[J]. 中华医学教育杂志,2022,42(4):289-291. DOI:10.3760/cma.j.cn115259-20210816-01023.

[1]	Ou Jian, Hu Yueqiang, Huang Hongna, Luo Jun, Wang Chunling, Liang Jiyao. Construction and empirical research on a BOPPPS and large language models (LLMs) integrated teaching quality evaluation system [J]. Chinese Journal of Medical Education, 2026, 46(8): 615-621.
[2]	Li Xinhui, Wang Niqing, Jin Jingjing. Ethical governance and teaching optimization of generative artificial intelligence in the ″Anatomy and Physiology″ course [J]. Chinese Journal of Medical Education, 2026, 46(7): 502-508.
[3]	Cheng Wenjie, Lyu Feng, Min Su. The application and prospects of embodied artificial intelligence in medical education [J]. Chinese Journal of Medical Education, 2026, 46(7): 509-514.
[4]	Wang Chenghe, Gu Qian, Chen Lu, Li Yinglin, He Hongchao, Huang fang, Xu Danfeng. Design and implementation of an intelligent agent-assisted cystoscopy training curriculum for surgical residents [J]. Chinese Journal of Medical Education, 2026, 46(7): 534-538.
[5]	Zhang Kunsong, Zhang Junlong, Feng Shaoting, Chen Wei, Kuang Ming, Xiao Haipeng. Opportunities, challenges and integration paths of medical humanities education in the AI era [J]. Chinese Journal of Medical Education, 2026, 46(4): 241-246.
[6]	Chen Lihua, Liu Shuo, Xin Siyan, Wang Yu, Wu Hongbin. A study on the influencing factors of the use of generative artificial intelligence by clinical medicine students [J]. Chinese Journal of Medical Education, 2026, 46(3): 199-205.
[7]	Wen Deliang, Wei Xiaotong. Application progress and future trends of artificial intelligence in medical education [J]. Chinese Journal of Medical Education, 2026, 46(1): 1-7.
[8]	Wei Xiaofan, Wu Jianghua, Chi Xiaochun, Zhang Weiguang, Peng Yihong. Exploring the path, challenges, and prospects of artificial intelligence in empowering human morphology curriculum construction [J]. Chinese Journal of Medical Education, 2026, 46(1): 8-14.
[9]	Zhu Jing, Wang Xiaofei, Zhou Weibing, Liu Hengyu. Visual analysis of research on generative artificial intelligence in medical education based on scientific citation databases [J]. Chinese Journal of Medical Education, 2025, 45(9): 661-665.
[10]	Wei Fangze, Liu Qian. Bibliometric study on the current application status of artificial intelligence in surgical education [J]. Chinese Journal of Medical Education, 2025, 45(9): 666-670.
[11]	Rong Xiaoying, Xu Mao, Zhou Yang, Gu Shixian, Qu Yinyin. Application of ChatGPT-assisted CBL teaching in anesthesia clerkship teaching for undergraduate medical students [J]. Chinese Journal of Medical Education, 2025, 45(8): 610-615.
[12]	Zhang Ruyi, Zhou Yun′ao, Peng Yingchun. Exploring collaborative governance pathways for artificial intelligence empowered medical education [J]. Chinese Journal of Medical Education, 2025, 45(7): 496-501.
[13]	Ding Min, Liu Shengjun, Sun Xiaoliang, Ma Ruolin, Gu Haojie, Shan Qiyuan, Kang Baoli, Xu Zengguang, Xu Wei, Chen Chi. Construction and application of an intelligent information management platform for a medical simulation center [J]. Chinese Journal of Medical Education, 2025, 45(7): 508-513.
[14]	Wang Kun, Hu Nan, Lu Haoping, Liu Wenzheng, Liu Zuojing. Survey on the need and efficacy of artificial intelligence systems in enhancing young physicians′ ability to manage emergencies of hematemesis [J]. Chinese Journal of Medical Education, 2025, 45(6): 423-427.
[15]	Li Junxun, Zhang Fan, Li Runzhao, Cheng Jing, Tan Hongxia, Chen Peisong, Huang Bin, Ouyang Juan, Lyu Wange. Application of an artificial intelligence-based online learning platform in the internship teaching of hematological cell morphology for medical laboratory technology students [J]. Chinese Journal of Medical Education, 2025, 45(3): 204-209.