Introduction

Residency programs must ensure their trainees are capable of handling a number of medical emergencies before graduation. Status epilepticus (SE) is a neurological emergency with an incidence of up to 41 per 100 000 patients and a 20% mortality rate.¹ Timely and appropriate treatment of SE improves morbidity and mortality.² Yet, a review of multiple studies demonstrated deficiencies in the treatment of SE by neurologists in academic medical centers.³ Treatment is often hindered due to delays in recognition, administration of incorrect first- and second-line anti-seizure drugs (ASDs), and suboptimal doses of correct ASDs.^3–5 Depending on the hospital setting, SE is often managed by physicians from various specialties, including neurology, emergency medicine, internal medicine, family medicine, and intensive care. Therefore, it is imperative that residency programs focus efforts on teaching residents about SE diagnosis and management beyond lecture series and chance clinical experiences.^6,7

In 2015, the American Board of Psychiatry and Neurology formed a joint initiative with the Accreditation Council for Graduate Medical Education (ACGME) to develop Milestones that provided a framework for the assessment of neurology residents.⁸ The Milestones are scored on a 5-point behaviorally based developmental rating scale, with level 1 roughly equivalent to a novice to advanced beginner in neurology and level 5 as an aspirational goal (someone who has been in practice for several years). Level 4 is the targeted level for residency graduation, although this is not a requirement for graduation. Specific Milestones address neurology residents' ability to diagnosis and manage patients with epilepsy and other neurological emergencies. Neurology residents achieve a level 3 for the epilepsy Milestone if they can diagnose and manage convulsive and non-convulsive SE. Residents reach level 4 for the management/treatment Milestone when they can independently direct management of patients with neurological emergencies (such as SE).

Despite guidance on what needs to be assessed at each level of residency, the Milestones do not provide a description for how residency programs should perform assessments. Residency programs use Milestones to evaluate the progression and advancement of their residents over time. They often use end-of-rotation or shift evaluations as a guide to help assign Milestone levels. End-of-rotation evaluations may be unreliable due to variation in opinions by supervising faculty.^9–13 Accurate assignment of Milestone levels require a sizeable number of direct observations by faculty of a vast array of clinical experiences.^14,15 However, this is problematic because end-of-rotation/shift evaluations often rely on chance clinical experiences and lack direct observations of infrequent clinical scenarios.^6,7,14–16 Milestone evaluations of high-stakes clinical skills such as diagnosis and treatment of SE and other neurological emergencies should not be based solely on subjective evaluations made during chance clinical experiences.

We previously developed an SE simulation-based mastery learning (SBML) curriculum at Northwestern University Feinberg School of Medicine.¹⁷ During the SE SBML curriculum, residents are assessed rigorously with a skills checklist on SE diagnosis and management and required to meet or exceed a minimum passing standard (MPS). The checklist helps guide the neurology program director in assessing a resident's ability to achieve the levels 3 and 4 Milestones in epilepsy and management/treatment. The current study has 2 aims: first, to evaluate the simulated SE identification and management skills of graduating (postgraduate year [PGY]-4) neurology residents and how their performance correlated with clinical experience and self-confidence, and second, to compare residents' simulated SE skills with their ACGME level of Milestones attained for epilepsy and management/treatment.

Methods

Results

Sixteen of 21 (76%) eligible residents participated in the study (4 of 6 eligible residents from site 1; 6 of 7 eligible residents from site 2; and 6 of 8 eligible residents from site 3). Eighteen residents came for the simulated sessions. One declined to participate (from site 1), and one other was excluded because he had already participated in the design of the study (from site 2). Demographic, clinical experience, and self-confidence of residents can be found in Table 1. The majority of graduating residents experienced 8 or fewer clinical episodes where they identified and managed convulsive or non-convulsive SE; one graduating resident experienced no clinical episodes.

Table 1 Demographics, Clinical Experience, and Self-Confidence of Neurology Resident Participants

View Fullscreen

No graduating neurology residents were able to meet or exceed the MPS during testing. The average SE checklist score was 15.6 of 26 checklist items correct (60%, SD 12.2). There were no significant correlations between resident checklist performance and number of cases performed during residency and self-confidence (Table 2). The average participant's level of Milestones for epilepsy and management/treatment were 4.3 of 5 (SD 0.4) and 4.4 of 5 (SD 0.4), respectively. Only one resident achieved a score of 3.5 (below the residency graduation threshold). There were no significant associations between checklist skills performance and level of Milestones assigned at graduation (Table 3). The Figure shows checklist performance of residents at each institution.

Table 2 Pearson's Correlation Coefficients Between Simulation-Based Skills Checklist Performance and Experience and Self-Confidence

View Fullscreen

Table 3 Neurology Residents Achieving Epilepsy and Management/Treatment Level of Milestones at Graduation and Average Simulation-Based Status Epilepticus Skills Checklist Score by Milestone Level (N = 16)

View Fullscreen

View Figure

• Download as Powerpoint
• Download Figure

Table 4 shows the performance on each checklist item by graduating residents. Only 11 of 16 (69%) residents properly identified the possibility of non-convulsive SE occurring after convulsive SE. While all residents were properly able to identify a first-line ASD, less than half properly ordered a second-line ASD and only half ordered relevant lab work. No residents ordered a post-ASD administration drug level.

Table 4 Abbreviated Status Epilepticus Checklist¹⁷ and Number (%) of Each Graduating Neurology Resident Getting Each Specific Item Correct

View Fullscreen

Discussion

We identified a significant gap in the skills of graduating residents consistent with prior work showing identification and management of SE is less than ideal.^3–5 To our knowledge, this is the first study to objectively evaluate graduating residents' skills using simulation compared to final level of Milestones assigned at graduation. Our findings are also consistent with our previous work showing graduating neurology residents performed lumbar puncture with highly variable skills.¹⁸ Additionally, and maybe more importantly, we revealed that despite the gap in SE identification and management skills as measured by the checklist, participating residents still received “ready to graduate” level Milestones for epilepsy and management/treatment on end-of-rotation evaluations.

Our study suggests that neurology training programs should not rely solely on end-of-rotation assessments submitted by faculty and staff to ensure residents' competency in high-stakes clinical events such as SE. End-of-rotation evaluations are often based on chance experiences, and it is unlikely that residents are observed frequently in their management of neurological emergencies such as SE.^6,7,14–16 This is evidenced by the variable number of self-reported graduating resident experiences identifying and managing convulsive and non-convulsive SE (Table 3), which did not correlate with their performance measured objectively. This lack of correlation between experience and measured performance is also consistent with many other studies using objective assessments of less commonly observed clinical skills.^19–23 Instead, we recommend that residency programs use competency-based assessment tools for direct observation using simulation for difficult to observe high-stakes events such as SE. Anesthesiology and emergency medicine residency programs have already successfully used objective structured clinical examinations or simulation-based assessments to help assess level of Milestones for clinical events that may be difficult to observe by faculty.^24–27 Additionally, information gathered from the simulated sessions helps programs identify and address gaps in trainee skills and areas for improvement in existing curricula and clinical workflows.

Equally problematic is the relatively moderate to high reported self-confidence identifying and managing SE, which did not correlate with the performance on the simulated assessment. This is also a common finding among studies comparing self-reported confidence and performance measured objectively.^{19–23,28,29} Because learners cannot reliably self-identify gaps in their knowledge, it is unlikely that they will seek further training or clinical experiences. Although we did not directly assess the effect of SBML on metacognitive skills, we believe the process of deliberate practice (a key component of SBML) improves residents' metacognition.³⁰

At Northwestern University Feinberg School of Medicine, we have successfully used SBML to help internal medicine, neurology, and surgical residency program directors guide Milestone assessments for clinical skills, including advanced cardiac life support,³¹ paracentesis,²² thoracentesis,²³ central venous catheter insertion,³² lumbar puncture,¹⁸ and laparoscopic surgery.³³ We also used SE SBML to help evaluate neurology residents in epilepsy and management/treatment Milestones.¹⁷ Similar SBML curriculum can be developed to help evaluate residents' performance in other high-stakes scenarios such as stroke codes, intracerebral hemorrhage, gastrointestinal bleeding, sepsis, and anaphylaxis.

Our study has several limitations. First, it was performed in a relatively small cohort of residents from only 3 academic neurology programs. Therefore, our study was underpowered to detect significant differences for small correlation coefficients. In a post-hoc power analysis, our study had 80% power to detect an rho of 0.65 with 16 participants (P < .05). We would have needed 47 participants to detect an rho of 0.40. However, this should not take away from the finding that no residents were able to meet or exceed the MPS on the skills checklist at residency graduation despite getting high Milestone-level ratings. We did not do an a priori power calculation because we used the entire cohort of graduating residents available for our study. Second, it is possible that participating neurology residents did not understand the expectations resulting in incorrect decisions during the simulation-based assessments. Additionally, the checklist was designed based on the protocols at one institution, which may be different from those learned at the participants' institutions. Although this may affect performance for some skills checklist items, this does not take away from the fact that several graduating residents did not properly identify non-convulsive SE, send appropriate labs, or correctly order second-line ASDs in the management of SE. Finally, residents were only provided one assessment (observation). We recommend using multiple assessments to determine final Milestone levels for residents. It is important to note that Milestones were not designed to be used as assessment tools, but rather to help residency programs evaluate the progress of their residents over time.³⁴

Conclusions

In conclusion, our study suggests that end-of-rotation evaluations are inadequate as the sole data source for assigning level of Milestones for high-stakes clinical skills such as identification and management of SE.