Objective Applying item response theory (IRT) to analyze the quality of medical psychology course examination papers provides a reference for the quality of the evaluation of education and teaching. Methods In May 2022, we assessed the quality of 60 multiple-choice questions in the Medical Psychology test paper for 330 clinical medicine students in the class of 2016 at Shanxi Medical University. We tested the core assumptions of the IRT model and analyzed the model-data fit, as well as the item parameters of the test questions and the amount of information. Results The test data were consistent with the theoretical assumptions of IRT, and the model was well-fitted. The two-parameter logistic regression model was selected for parameter estimation. 59 (98.3%) of the test questions met the criteria for estimating the discrimination parameter, 54 (90.0%) of the test questions met the criteria for estimation of the difficulty parameter, 55 (91.7%) of the test questions were able to provide the minimum required amount of information(0.16), and 49 (81.7%) of the test questions were of satisfactory final quality, with the amount of information provided by the examination paper reaching 25. Conclusions The overall quality of the question papers was good, with the vast majority of the items being of high quality. However, it was necessary to modify or delete a small number of questions. This study can serve as a reference for the quantitative evaluation of examination papers in some university courses based on IRT.