Abstract: Objective Generalizability theory was utilized to determine optimal clinical skills assessment designs, which reduce errors and improve reliability.Methods Forty resident doctors acted as the study participants which were in their second year of residency training at the Sixth Affiliated Hospital of Guangzhou Medical University in 2018. Items in the clinical skills assessment included abdominocentesis, tracheal intubation, cardiac physical examination, debridement and suturing, and thoracentesis. Two scorers per station were calculated and analyzed using generalizability theory.Results The results from the multivariate analysis of variance and generalizability theory showed that the total variance was 268.106 and the variance component interpretation percentages were 39.9% (research subject, 106.936/268.106), 6.8% (assessment station, 18.190/268.106), 0.0% (scorer, 0.000/268.106), 41.9% (research subject and assessment station, 112.395/268.106), 0.0% (research subject and scorer, 0.000/268.106), 1.8% (assessment station and scorer, 4.838 /268.106), and 9.6% (residual, research subject and assessment station and scorer, 25.747/268.106). When adding test stations or scorers, their generalization coefficient and independent index values (these two indicators represent the reliability value of the test) increase until a certain point is reached; at that point, by adding test stations or scorers, the rate of increase of the generalization coefficient and the independent index value decrease.Conclusions To obtain the most reliable formative evaluation, eight test stations should be designed, with one scorer at each test station; 12 credential stations should be designed for the most reliable final evaluation, with two scorers at each test station.

Key words: Generalizability theory, Clinical skills assessment design, Reliability optimization