Objective By analyzing whether there is a correlation between the results of the annual clinic competency test and the final examination for residents in Beijing, to assess the validity of the annual clinic competency test, and to provide reference for early identification of residents with poor competency and targeted training. Methods Based on Kane's validity framework, 2 745 residents' results of the 2020 annual clinic competency test, which were expressed by the percentile of the national residents in the same specialty, and the 2021 final examination, which expressed by a pass or fail and including a theoretical assessment and an Objective structured clinical skills exam, were analyzed for the correlation by Cochran-armitage trend test and logistic regression model. Results The residents' national percentile of the annual clinic competency test score was 72.4% (43.4%, 90.8%), and pass rate of the final theory examination, skills examination and entirety were 95.9% (2 632/2 745), 96.2% (2 641/2 745) and 93.3% (2 560/2 745). After grouping according to the percentile of annual clinic competency test scores, pass rate of the theoretical examination, skill examination and entirety were statistically different between different groups [such as those pass rate of residents in the >90%-100% group vs. those in the 0-10% group: 99.9%(722/723) vs. 73.6%(89/121), 99.2%(717/723) vs. 85.1%(103/121), 99.0%(716/723)vs. 68.6%(83/121),all P＜0.001]. The results of logistic regression showed that, compared with the 0-10% group, the other groups had a significant increase in passing rate of the theoretical examination, skill examination and entirety, with statistical differences [such as the residents in the >90%-100% group, the probability of pass rate of theoretical examination, skill examination and entirety were 204.63 times and 18.02 times and 37.08 times of those in the 0-10% group, all P<0.001]. Conclusions There is a positive association between the results of the annual clinic competency test and the final examination for residents in Beijing,which means the annual business level test is a valid evaluation tool. More attentions should be paid to the application of the results of the annual clinic competency test, and the quality of training should be improved through targeted guidance and feedback.