Objective To predict the difficulty of questions used by physician qualifying examination and to accurately control the difficulty of the whole paper by using artificial intelligence (AI) deep learning technology. Methods By constructing the framework of attribute model and semantic model, the difficulty of the test questions was estimated. The results of AI prediction and experts' prediction were correlated and repeated-measures ANOVA were conducted with the actual test difficulty respectively for evaluation of feasibility and effectiveness of the AI model applied for difficulty prediction. Results For a given year's whole paper questions, the Pearson correlation coefficient between attribute model prediction and actual test difficulty was 0.266, which was slightly lower than the correlation coefficient of 0.356 between the experts prediction difficulty and the actual difficulty. There was a crossover between the confidence intervals of the two coefficients (P>0.05). The Pearson correlation coefficient between semantic model and actual difficulty was 0.512, which was higher than the correlation coefficient between the difficulty by experts' prediction and the actual test (0.356). There was no crossover in the confidence intervals of the two coefficients (P<0.05). The results of statistical analysis using one-way repeated measures ANOVA showed that there was no statistical difference only between sematic model prediction difficulty and actual test difficulty (P>0.05). Conclusions The difficulty of the test questions predicted by the semantic model is closer to the actual difficulty of the test than the difficulty predicted by the experts. So it may be applied to the pre-examination difficulty prediction, and combined with the results of the experts' prediction to jointly guide the development of test paper based a well-predicted difficulty.