Abstract: Objective This study aimed to conduct a preliminary investigation and explore methods to improve the homogeneity of examiners in clinical skill examination of standardized resident training. Methods In July 2021, the concordance correlation coefficient (CCC) was used to analyze the consistency of examiner scorings for history taking and physical examination, clinical thinking, and skills in the objective structured clinical examination (OSCE) of 105 residents at Sun Yat-sen University. A subgroup of 44 examiners received training on detailed interpretation of the rules and key criteria of scoring, and the consistency of examiner scorings was assessed before and after the training. Results Of the 63 examination items, 3 items (4.8%) had a CCC>0.7, 31 items (49.2%) had a CCC of 0.3-0.7, and 29 items (46.0%) had a CCC<0.3. Among the 17 items in clinical thinking station, 14 items had a CCC≥0.3. For the 25 items of history taking of the same candidate, the number of items with consistent examiner scores ≥80.0% (35/44) increased from 8 (32.0%) to 19 (76.0%) after intensive training. Conclusions The homogeneity of examiners in clinical skill examination of standardized resident training needs to be improved. The homogeneity of the examiners can be improved through implementing training, such as detailed interpretation of the rules and key criteria of scoring.

Key words: Educational measurement, Standardized resident training, Examiner scoring, Concordance correlation coefficient, Clinical skill examination, Objective structured clinical examination