A 3-Year Longitudinal Study Evaluating Predictors of Overall Resident Performance
Setting and Problem
Pediatric residency programs have a vested interest in matching medical students who will make successful residents in their program. This is not a one-size-fits-all proposition, as the culture and values of programs may differ. Residency program directors indicate they value United States Medical Licensing Examination (USMLE) scores and letters of recommendation when selecting applicants to interview and rank in their programs, but the predictive value of these parts of the application are not well studied. Evaluation of residents in the era of competency-based assessment has contributed to the complexity of the issue, with residency programs reporting that their Clinical Competency Committees' milestone-level assessments of residents were influenced by local evaluation practice as much as, if not more than, actual resident competence. Residency applications are at an all-time high, creating work for program directors and expense for medical students. A useful system might incorporate program culture and values—constructs that are hard to measure (if measured at all) using milestones—to identify those students who will succeed in their program.
Intervention
To address this gap, residency program directors at our institution developed a Global Director Rating (GDR) to assess all residents in our program at the end of each academic year. We identified qualities shared by residents in our program who excelled during residency and were ideal colleagues. We grouped our ideas into 4 themes around the ability to: be a role model, build morale, demonstrate trustworthiness, and contribute to the overall health of the residency program. We used these 4 themes and created a GDR form to rate each resident in the program on a 9-point Likert scale (1, lowest score, to 9, highest score). We rated residents independent of their year of training (ie, an intern could be rated a 9 and a senior resident could be rated a 1). Our objective was to use this novel outcome measure to identify factors associated with resident performance, using predictor variables identified before and after starting residency, to gain insight into what makes a successful resident in our program and to focus recruitment efforts around these goals. This study was approved by the Institutional Review Board of the Medical College of Wisconsin.
Outcomes to Date
We assessed 203 pediatric residents at a moderate-sized program during the 2014–2017 academic years. Predictor variables associated with the GDR prior to starting residency (r = 0.3–0.6, P ≤ .05) were academic points (the vast majority of points from clerkship grades) and interview scores. USMLE/Comprehensive Osteopathic Medical Licensing Examination (COMLEX) scores were not associated with the GDR (P > .05). Predictor variables associated with the GDR after starting residency (r = 0.3–0.6, P ≤ .05) were milestone-level ratings for the competencies of patient care, practice-based learning and improvement, and interpersonal and communication skills. In-training examination scores and milestone-level ratings for the competencies of medical knowledge, professionalism, and systems-based practice were not (P > .05; table). Creating the database in a central location was time-consuming but has been easy to maintain and can be used for further research opportunity. The time to rate each resident using the GDR was minimal, and the only expense for this project was the time needed to develop our “mission statement” and create the database for analysis. We have used this data during recruitment and education of our intern selection committee. Relinquishing our focus on USMLE/COMLEX scores has opened our program doors to some exemplary residents who have thrived in our program.
Author Notes
