|
内容記述 |
Purpose:To evaluate GPT-5 against GPT-4o on the Japanese National Examination for Radiological Technologists (2016–2025), assessing accuracy, repeatability, and factors influencing performance differences.Materials and methods:We analyzed 1992 multiple-choice questions involving text and images, spanning the medical and engineering domains. Both models answered all questions in Japanese under identical conditions across three independent runs. Majority-vote accuracy (correct if ≥ 2 of 3 runs were correct) and first-attempt accuracy were compared using McNemar’s test. Repeatability was quantified with Fleiss’ κ. Univariable and multivariable analyses were conducted to identify question-level factors associated with GPT-5 improvements.Results:Across all 10 examination years, GPT-5 achieved a majority-vote accuracy of 92.8 % (95 % CI: 91.5–93.8), consistently outperforming GPT-4o at 72.4 % (95 % CI: 70.4–74.4; P < .001). Repeatability was higher for GPT-5 (κ = 0.925, 95 % CI: 0.915–0.935) than for GPT-4o (κ = 0.904, 95 % CI: 0.894–0.914), with correct answers in all three runs for 88.2 % vs. 68.9 % of items. GPT-5 performed better than GPT-4o in text-based (96.5 % vs. 78.1 %) and image-based questions (72.6 % vs. 41.9 %). Significant improvements were observed for MRI, CT, and radiography images; however, performance improvements were smaller for clinically oriented ultrasound and nuclear medicine images. The greatest advantages were observed in calculation questions (97.3 % vs. 39.3 %) and engineering-related domains, consistent with external benchmarks highlighting GPT-5’s improved reasoning.Conclusion:GPT-5 demonstrated significantly higher accuracy and repeatability than GPT-4o over a decade of examination, with improvements in quantitative reasoning, engineering content, and diagram interpretation. Although improvements extended to medical images, performance in clinical image interpretation remains limited. |
