Confidence Can Be Measured and Calibrated
We read the article entitled “Medical Trainees and the Dunning–Kruger Effect: When They Don't Know What They Don't Know”1 published in a recent issue of the Journal of Graduate Medical Education. We agree with the author's identification of the problem of miscalibrated trainee confidence (of which the Dunning–Kruger Effect is a subset) and the serious shortcomings of self-assessment.
In addition to the suggestions by the author for improving calibration of trainee confidence, we would like to recommend quantitative methods for measuring confidence calibration at baseline and following specific interventions. The mathematical framework for these methods was first developed for improving the “diagnoses” of weather forecasters, but has been applied to physicians of all levels of training.2,3 We have summarized the medical application of these methods for quality improvement and education in 2 recent articles, with specific application to our field of practice (radiology).4,5
The use of these methods requires: (1) a diagnostic question with a reliable reference standard; (2) trainees who are willing to quantify their confidence in their diagnoses in terms of percentages (eg, “I am only 25% sure that this patient has condition X.”); and (3) access to a web browser. We have created an online tool for generating the metrics, along with descriptions of the metrics, at http://checkmyai.com.
