Research on virtual standardized patient application based on ChatGPT-4

doi:10.3760/cma.j.cn115259-20240307-00225

Abstract

Abstract: Acting with standardized patients is an important way to train the skills of medical history taking. The newly developed large language model ChatGPT has powerful language function and can communicate with human-like natural language. To further explore and develop the prompt engineering for ChatGPT-4 to act as virtual standardized patients (VSP), this study established, tested, and iterated the prompts from October 2023 to February 2024. The logs of ChatGPT-4′s interaction with participants and the evaluations of medical students participating in the test were collected. Qualitative text analysis was used to evaluate the influence of different versions of the prompt lexemes on improving the effectiveness of ChatGPT-4 as a VSP. The results showed that ChatGPT-4 had a better understanding of narrative history, mastered the interaction skills of SP, and effectively evaluated the consultation. After adjustment and optimization, the prompts can adapt to various medical cases. Students rated their performance as excellent, especially if there was emotional interaction. The establishment of the prompt engineering provides a new way for medical teachers to easily create VSPs, and creates a chance for medical students to practice medical history taking.

Key words: Medical history taking, ChatGPT-4, Standardized patients, Virtual standardized patients

CLC Number:

R-05

Chen Naping, Tang Luzhen, Huang Xiansheng, Zheng Jinbin, Fan Yizhou. Research on virtual standardized patient application based on ChatGPT-4[J]. Chinese Journal of Medical Education, 2025, 45(1): 44-49.

References

[1] Maicher K, Danforth D, Price A, et al. Developing a conversational virtual standardized patient to enable students to practice history-taking skills[J]. Simul Healthc, 2017,12(2):124-131. DOI: 10.1097/SIH.0000000000000195.
[2] Plaksin J, Nicholson J, Kundrod S, et al. The benefits and risks of being a standardized patient: a narrative review of the literature[J]. Patient, 2016,9(1):15-25. DOI: 10.1007/s40271-015-0127-y.
[3] Bosse HM, Nickel M, Huwendiek S, et al. Cost-effectiveness of peer role play and standardized patients in undergraduate communication training[J]. BMC Med Educ, 2015,15:183. DOI: 10.1186/s12909-015-0468-1.
[4] Gillette C, Stanton RB, Rockich-Winston N, et al. Cost-effectiveness of using standardized patients to assess student-pharmacist communication skills[J]. Am J Pharm Educ, 2017,81(10):6120. DOI: 10.5688/ajpe6120.
[5] Stevens A, Hernandez J, Johnsen K, et al. The use of virtual patients to teach medical students history taking and communication skills[J]. Am J Surg, 2006,191(6):806-811. DOI: 10.1016/j.amjsurg.2006.03.002.
[6] Peng S, Wang D, Liang Y, et al. AI-ChatGPT/GPT-4: an booster for the development of physical medicine and rehabilitation in the new era![J]. Ann Biomed Eng, 2024,52(3):462-466. DOI: 10.1007/s10439-023-03314-x.
[7] Lee P, Bubeck S, Petro J. Benefits, limits, and risks of GPT-4 as an ai chatbot for medicine[J]. N Engl J Med, 2023,388(13):1233-1239. DOI: 10.1056/NEJMsr2214184.
[8] GPT-4 technical report[EB/OL].[2024-03-01] . https://arxiv.org/pdf/2303.08774.
[9] Kuroiwa T, Sarcon A, Ibara T, et al. The potential of chatgpt as a self-diagnostic tool in common orthopedic diseases: exploratory study[J]. J Med Internet Res, 2023,25:e47621. DOI: 10.2196/47621.
[10] Topsakal O, Akinci TC, Celikoyar M. Evaluating patient and otolaryngologist dialogues generated by ChatGPT, are they adequate?[EB/OL].(2023-04-02)[2024-03-01] .https://www.researchsquare.com/article/rs-2719379/latest.
[11] Luke N, Taneja R, Ban K, et al. Large language models (ChatGPT) in medical education: embrace or abjure?[J]. Asia Pacific Scholar, 2023, 8(4): 50-52. DOI:10.29060/TAPS.2023-8-4/PV3007.
[12] Safranek CW, Sidamon-Eristoff AE, Gilson A, et al. The role of large language models in medical education: applications and implications[J]. JMIR Med Educ, 2023,9:e50945. DOI: 10.2196/50945.
[13] Milota MM, van Thiel G, van Delden J. Narrative medicine as a medical education tool: a systematic review[J]. Med Teach, 2019,41(7):802-810. DOI: 10.1080/0142159X.2019.1584274.
[14] Melvin RS, May W, Narayanan SS, et al. Creation of a doctor-patient dialogue corpus using standardized patients[EB/OL].[2024-03-01] . http://sail.usc.edu/publications/files/belvinlrec2004.pdf.
[15] Bickley L, Szilagyi PG. Bates′ guide to physical examination and history-taking[M]. Philadelphia: Lippincott Williams & Wilkins, 2012:65-106.
[16] Eysenbach G. The role of ChatGPT, generative language models, and artificial intelligence in medical education: a conversation with chatgpt and a call for papers[J]. JMIR Med Educ, 2023,9:e46885. DOI: 10.2196/46885.
[17] Boulet JR, van Zanten M, de Champlain A, et al. Checklist content on a standardized patient assessment: an ex post facto review[J]. Adv Health Sci Educ Theory Pract, 2008,13(1):59-69. DOI: 10.1007/s10459-006-9024-4.
[18] Bokken L, Linssen T, Scherpbier A, et al. Feedback by simulated patients in undergraduate medical education: a systematic review of the literature[J]. Med Educ, 2009,43(3):202-210. DOI: 10.1111/j.1365-2923.2008.03268.x.
[19] Makoul G. Essential elements of communication in medical encounters: the Kalamazoo consensus statement[J]. Acad Med, 2001,76(4):390-393. DOI: 10.1097/00001888-200104000-00021.
[20] Erby LA, Roter DL, Biesecker BB. Examination of standardized patient performance: accuracy and consistency of six standardized patients over time[J]. Patient Educ Couns, 2011,85(2):194-200. DOI: 10.1016/j.pec.2010.10.005.
[21] Danforth DR, Procter M, Chen R,et al. Development of virtual patient simulations for medical education[J]. J Virt Worlds Res, 2009, 2(2):1-11. DOI:10.4101/JVWR.V2I2.707.
[22] Talbot T, Sagae K, John B,et al. Designing useful virtual standardized patient encounters[EB/OL].[2024-03-01] . https://ict.usc.edu/pubs/Designing%20Useful%20Virtual%20Standardized%20Patient%20Encounters.pdf.
[23] Maicher KR, Stiff A, Scholl M, et al. Artificial intelligence in virtual standardized patients: combining natural language understanding and rule based dialogue management to improve conversational fidelity[J]. Med Teach, 2022, 45(3):279-285. DOI: 10.1080/0142159X.2022.2130216.
[24] Vishal K.ChatGPT-4 turbo, all you need to know[EB/OL].(2023-11-11)[2024-03-01] . https://medium.com/@kumar.vishal9626/chatgpt-4-turbo-all-you-need-to-know-e141e644bcf4.
[25] Hasle JL, Anderson DS, Szerlip HM. Analysis of the costs and benefits of using standardized patients to help teach physical diagnosis[J]. Acad Med, 1994,69(7):567-570. DOI: 10.1097/00001888-199407000-00013.
[26] Bearman M, Cesnik B, Liddell M. Random comparison of ′virtual patient′ models in the context of teaching clinical communication skills[J]. Med Educ, 2001,35(9):824-832. DOI: 10.1046/j.1365-2923.2001.00999.x.