Choosing the Right Resident: Implications of the New Electronic Emergency Medicine Standardized Letter of Evaluation

“Good” is bad. “Solid” is worse. Is “outstanding” the same as “superior”? Residency program directors spend hours deciphering the hidden meanings of carefully chosen adjectives from a heterogeneous pool of letter writers. Applicants agonize over selecting ideal advocates to endorse their applications. In an era when many medical schools employ “pass/fail” grading, it is difficult to differentiate residency applicants, especially those below the elite echelon of their class. To discern the relative capability of applicants, the Council of Residency Directors in Emergency Medicine (CORD) developed and implemented a standardized tool for experienced emergency medicine (EM) educators to recommend and evaluate students seeking an EM residency position. This tool uses predefined comparative metrics and narrative comments.¹ In this issue of the Journal of Graduate Medical Education, Jackson and colleagues outline the historical development of the CORD Standardized Letter of Recommendation/Evaluation (SLOR/SLOE), the gold standard for EM, and report on the newly enacted electronic version (eSLOE) that integrates with the Electronic Residency Application Service (ERAS) system.²

Undoubtedly, the eSLOE has facilitated the selection process for program directors, although there are no definitive data to demonstrate if its metrics accurately predict residency or career performance. Additional eSLOE outcomes that are of interest to EM programs and other specialties considering a similar process include the impact on applicants, the conflicted role of the clerkship director when completing the eSLOE, and the implications of a “negative” eSLOE.

Do Standards-Based Forms Help or Hurt the Applicant?

The eSLOE categories are known to medical students so they have the advantage of knowing the metrics upon which their eSLOEs will be based. This level of transparency is not generally available to students whose letters are purely narrative. To be most effective, each category of the eSLOE should be completed accurately. As noted in the Jackson et al article, there remains a gap between the intended and actual use of the categories. In 2016–2017, 18% of applicants were placed in the “top 10%” category and an additional 35% were placed in the “top one-third” category in Global Assessment.² Since EM is a competitive specialty, candidates who fall into lower categories may not match into the field. SLOE writers may be reluctant to assign these categories for fear that their students will be overlooked, especially with an electronic tool where computerized screening could occur. In this instance, carefully crafted narratives highlighting applicants' strengths and putting negative discreet variables into perspective may go unnoticed.

Is the Clerkship Director an Agent of the Residency Selection Process or the Student's Advocate?

The clerkship director's primary duty is to provide a comprehensive educational experience for medical students in the discipline of EM. Typically, this includes planning curriculum, clinical experiences, assessment, and career advising. One could argue that the categorical nature of the eSLOE limits the ability of clerkship directors to advocate solely on behalf of medical students by forcing them to address every aspect of a candidate's worthiness, rather than choosing to focus only on positive attributes. As reported by Jackson and colleagues, letter writers in the 2011–2012 cycle assigned 40% of the applicants to the “top 10%” category and an additional 43% to the “top one-third” in Global Assessment.² By not adhering to the guidelines, it is difficult to differentiate among applicants.

The emergence of a cohort of experienced, trained letter writers was likely instrumental in increasing the utility of the eSLOE, as the reported metrics improved significantly in the 2016–2017 analysis. Since clerkship directors are typically among those trained in the use of the eSLOE, they can use it as an educational tool for medical students. For example, they can explain to students what it means to “develop and justify an appropriate differential and a cohesive plan” during the course orientation and provide formative feedback throughout the rotation to guide students toward the highest category. Ultimately, clerkship directors are responsible to their own programs and national colleagues to accurately report the relative abilities of each applicant.

Should Applicants in the Lower One-Third of the eSLOE Pursue EM?

Considering that the specialty is competitive, assignment to one of the lower categories may be an indicator that EM is not the right fit. The clerkship director can collaborate with the medical school dean's office to use this evidence to guide students to a more realistic choice that matches their inherent skill sets. Although it may be a disappointment to students, the honest assessment reflected on the eSLOE may serve to circumvent challenges that could be encountered if they match into a residency program whose demands exceed their capabilities. Occasionally, students show great improvement on subsequent rotations, and because of the standardized nature of the eSLOE, this upward trend can be noted and compared directly in the screening and ranking processes.

Future Directions

Any specialty can benefit from standards-based evaluations. Many of the CORD eSLOE categories are broadly applicable (eg, knowledge, commitment to the specialty, work ethic, communication of a caring nature to patients). Specialties considering a SLOE should define and include their own integral skills. For EM, time-based decision making for critical events with limited prior knowledge is vital. Thus, the timely “ability to develop and justify an appropriate differential and cohesive plan” is a crucial category. Because of the unpredictable environment in the emergency department, applicants must have the “ability to work with a team.” Narrative letters often incorporate the concept of “hardworking” when describing applicants. While not listed explicitly in the eSLOE, this concept is captured in 2 important questions: “How much guidance do you predict this applicant will need during residency?” and “Given the necessary guidance, what is your prediction for success for the applicant?” Many program directors would gladly take a “middle-third” candidate whose potential for success is “outstanding” over one who would need minimal guidance for only a “good” outcome. This is an example of how an “objective” evaluative instrument accommodates subtle messages.

The success of the EM eSLOE also results from the dedication of a core group of educators who desired a meaningful tool to aid in the selection of residents. With continuous quality improvement metrics, the CORD task force identified obstacles and devised solutions by cultivating a national cohort of trained letter writers and ensuring technical compatibility with ERAS.

It will be important to investigate whether the eSLOE correlates with or predicts residency and career performance. As noted by Jackson and colleagues, some students had eSLOEs submitted on their behalf but did not apply to EM.² Analyzing characteristics of these students and their eSLOE data may inform future advisement of similar students. Researchers could perform within-subjects comparisons of letter writers, evaluate effects of using eSLOE as a teaching tool by clerkship directors for students, or develop a profile of the ideal EM applicant incorporating all ERAS application components (eg, USMLE scores, MSPE data, other letters of recommendation, entries in the curriculum vitae, etc).

The eSLOE is completed by trained faculty and provides a uniform evaluation platform for EM applicants. It focuses on rating preidentified characteristics and skills that faculty believe will lead to success in residency. Students and faculty may also use these criteria to guide learning and career choice. When combined with the varied components of the ERAS application, the eSLOE may help ensure a good fit for applicants to residency programs.