Introduction

Over the past decade, we have witnessed a transformation of medical education and training to a competency-based model. This new paradigm emphasizes a contextual and developmental approach to medical education and training with the promise of more proficient physicians and ultimately improved safety and quality of patient care.^1–5 In the movement toward competency-based medical education, each specialty developed subcompetencies in each of the 6 Accreditation Council for Graduate Medical Education (ACGME) competency domains with milestones levels that represented a blueprint for the development of knowledge, skills, and attitudes germane to their field.⁶ While 51 subcompetencies were developed in pediatrics, 21 were identified for tracking during residency training.^7,8 It was never the intent that these theoretical milestones would serve as an assessment tool, but rather they represented a shared mental model to track trainee progress over time, and serve as a roadmap for individual improvement.^6,9 Many programs have nevertheless employed the subcompetency milestones as a direct assessment tool, as these milestones ratings are the accepted ACGME standard in the required semiannual reporting of resident competence. Challenges with this approach include the milestones' lengthy descriptions, wide range of applicability, and room for interpretation.^6,10–13 Most importantly, the subcompetency milestones are a theoretical construct, and while they provide a roadmap for moving from novice to expert, they do not necessarily reflect specific observable behaviors.¹³

Entrustable professional activities (EPAs) require observable behaviors and skills that support decisions to trust a trainee to independently perform within a particular field.^13–16 In the context of competency-based medical education, EPAs represent an attractive assessment tool for several reasons: they are easily observed and reliably assessed^4,9,10,17; entrustment incorporates supervision and safety considerations^18–20; and competence is implicit in the eventual entrustment of trainees to perform EPAs.^13,21,22 Because EPAs are directly observable, they may serve as a potential link between the theoretical framework of subcompetencies and point-of-service clinical practice.^13,23–25 From a practical perspective, if multiple subcompetency milestones ratings are correlated to single entrustment decisions, an approach that utilizes EPAs as assessment tools may greatly simplify our approach to competency-based assessment.⁵

Recognizing the promise of EPAs in assessment, many have explored their use as an assessment tool. These studies have demonstrated successful mapping of EPA assessments to the subcompetency milestones.^9,26–30 However, there has been no study to date that has investigated validity evidence for this mapping strategy.

The primary objective of our study was to determine the interitem correlation of resident assessment on global faculty-of-resident rotation assessments by parallel and concurrent EPA rating and primary subcompetency milestones level rating. A secondary objective was to determine evidence of relations to other variables' validity for the EPA entrustment scale used for this project.

Methods

This prospective study was conducted across 3 diverse pediatric training programs: University of Vermont (21 residents), University of South Alabama (39 residents), and Children's National Medical Center in Washington, DC (117 residents). The sites included a children's hospital within a larger medical center, a free-standing children's and women's hospital, and a large free-standing children's hospital, respectively. Pediatric residents across the 3 years of training were assessed in 4 separate core rotations during the 2014–2015 academic year. These rotations included the pediatric intensive care unit (PICU), neonatal intensive care unit (NICU), general inpatient pediatrics, and continuity clinic representing a mix of inpatient and outpatient rotations and general pediatrics and subspecialty rotations.

Common rotation-specific EPAs were developed for each of these core rotations by adapting existing rotation-specific goals and objectives from the 3 training programs. Consensus was achieved by using a modified Delphi method in which iterative cycles of verbal and written feedback from the program directors, rotation directors, and rotation-specific supervising faculty at each of the institutions were conducted until consensus was reached.³¹ Numerous pediatric subcompetencies were mapped to each rotation-specific EPA based on utilizing a similar modified Delphi method with the same personnel. The decision for each mapping was based on whether the observation of the knowledge, skills, and attitudes embedded in the EPA could inform the performance of that individual in a particular subcompetency domain. In order to quantitatively compare the evaluation of entrustment to a subcompetency milestone level, we employed a 5-level entrustment scale (Box), which was adapted from the literature and mirrored the Dreyfus 5-level novice-to-expert scale that is embedded within the subcompetency milestones.^16,32,33

New faculty-of-resident global rotation assessments for the 4 rotations were assembled with 2 primary sections: an entrustment section and a subcompetency section (provided as online supplemental material). The entrustment section included the rotation-specific EPAs with the corresponding 1 to 5 entrustment scale, including half-point graduations. The subcompetency section included all correspondingly mapped subcompetencies and their respective 1 to 5 milestones levels, including half-point graduations. Half-point intervals in the entrustment scale were used to afford scoring flexibility when summating multiple observations of discrete clinical situations and, in the case of composite assessments, discrepancies in faculty entrustment decisions based on their particular observations. Identical assessments were built in the respective electronic management systems of the 3 programs (New Innovations, Uniontown, OH, and MedHub, Minneapolis, MN). Faculty development was carried out at each of the institutions by the program directors and rotation directors to optimize understanding and compliance with the assessment system and milestones ratings. While rotations across the institutions used a similar end-of-rotation assessment strategy, some rotation assessments were completed by an individual attending physician while other rotations employed a composite assessment approach in which multiple attending physicians jointly completed single assessments.

For each completed assessment, the entrustment level assigned for each rotation-specific EPA was then compared to the milestone level assigned for each of the subcompetencies mapped to that EPA to determine level of correlation (Figure 1). Each comparison between entrustment level and milestone level represented a data point. A REDCap database (Vanderbilt University, Nashville, TN) was utilized to collect data across the 3 programs. Pearson correlation coefficients were calculated for each rotation-specific EPA and subcompetency milestones pair. Multivariate linear regression produced R-squared values for each EPA, which measured the strength of the relationship between the primary milestones ratings for the collective subcompetencies mapped to that EPA.

View Figure

• Download as Powerpoint
• Download Figure

The study received Institutional Review Board exemption from each of the participating institutions.

Results

Using the modified Delphi method, rotation-specific EPAs were generated for each of the 4 rotations: 6 for PICU, 8 for NICU, 6 for inpatient pediatrics team, and 7 for continuity clinic (provided as online supplemental material). With judicious mapping, 16 subcompetencies spanning the ACGME 6 competency domains were mapped to the 6 PICU EPAs, 16 to the 8 NICU EPAs, 16 to the 6 inpatient pediatrics team EPAs, and 17 to the 7 continuity clinic EPAs (provided as online supplemental material; Table 1). Of the 21 reportable pediatric subcompetencies, 18 were mapped to the 4 sets of rotation-specific EPAs. There were 630 assessments completed on these 4 rotations at the 3 participating institutions during the study period, comprising 29 143 paired data points distributed across the rotation-specific EPAs and subcompetency milestones included in the study.

Table 1 Pearson Correlation Coefficients for Each Mapping of Subcompetency Milestone to Rotation-Specific Entrustable Professional Activity (EPA)

View Fullscreen

Pearson correlation coefficients were calculated for all pairs of rotation-specific EPAs and their mapped subcompetency milestones. Statistical significance (P < .001) was demonstrated for all pairs (Table 1). Nearly all Pearson coefficients ranged from 0.70 to 0.90, indicating high correlation. Mean correlation among the mapped milestone-entrustment pairs for the 4 rotations were 0.86 for PICU, 0.79 for NICU, 0.87 for inpatient pediatrics team, and 0.86 for continuity clinic.

Multivariate linear regression modeling assessing the percentage of the variance for each EPA entrustment rating that was predicted by the group of mapped subcompetency milestones ratings yielded statistically significant R² values (median R² = 0.81; interquartile range 0.73–0.83; P < .001) for all EPAs (Table 2). Hence, the milestones scores assigned to that resident on each rotation explained a consistently high proportion of the variance in entrustment ratings.

Table 2 Assessing Percentage of Variance for Each Entrustable Professional Activity (EPA) Entrustment Rating Predicted by Mapped Subcompetency Milestones Ratings

View Fullscreen

Both EPA level (entrustment rating) and subcompetency milestones rating increased linearly with level of training (P < .001). The overlapping lines with similar slopes provide evidence of relations to other variables' validity for the entrustment and milestones scales used (Figure 2).

View Figure

• Download as Powerpoint
• Download Figure

Discussion

The results of this study demonstrate that rotation-specific EPAs correlate with multiple mapped subcompetency milestones. The statistically significant correlations appear to hold across institutions and rotations. This implies that graduate medical education programs may be able to simplify their faculty-of-resident assessment strategies by replacing numerous individual subcompetencies and associated milestones with rotation-specific EPAs derived from stated rotation-specific goals and objectives.

In comparing average correlations across rotations, it appears some experiences lend themselves better to rotation-specific EPAs. This implies that the setting or the specific wording of the EPA may affect the validity evidence for using EPAs as a predictor of subcompetency milestones levels. Additionally, as EPAs represent observable skills, a rotation where the attending and the resident work more closely would presumably afford a more reliable EPA assessment than a rotation where the resident works more indirectly with the supervisor.

Because rotation-specific EPAs include a number of subcompetencies, they are holistic in nature, and their assessment represents a more global screening assessment of trainees.^5,10,13,34 If a learner is assessed to be at a high level on an EPA, then it follows that they are generally proficient in the subcompetency milestones mapped to it. However, when a weakness is identified using an EPA assessment, alternative ways to assess learners (eg, objective structured clinical examination or another form of direct observation) are needed in order to deconstruct and tease out the problem area(s). This represents a natural way of assessing learners—to screen and then verify with additional assessment tools. Therefore, while using rotation-specific EPAs can be a valuable screening assessment, this does not preclude the need for other assessment methods.

The use of 4 rotations at 3 institutions within 1 specialty (pediatrics) limits generalizability. The potential presence of faculty who developed the EPA-subcompetency mapping and who were also responsible for completing the end-of-rotation resident assessments might have unduly positively skewed correlations between the EPAs and subcompetencies. However, we estimate there was less than 5% overlap between these faculty groups. Participating faculty also expressed concern about the length of the assessment forms; thus, assessment fatigue in which an assessor chooses the same level throughout the assessment form to reduce cognitive load could have positively skewed the results. We acknowledge that high-quality EPAs should focus on professional tasks, not individual qualities of a learner.³⁵ In reviewing our developed EPAs, the final rotation-specific EPAs (PICU 6, NICU 8, inpatient team 6, and continuity clinic 7) do include an aspect of personal development; however, these EPAs also include activities that are directly relevant to patient care. There may have been variability in the manner in which individual versus composite assessments were completed, which may have affected entrustment decisions. However, as different rotations at the 3 institutions used individual versus composite assessments, this factor is less likely to produce systematic bias. We also recognize that greater time devoted to faculty development and iteratively revising the rotation-specific EPAs verbiage might have improved correlations between entrustment and competency.

Further standardization and characterization of EPA and entrustment scale content and construct should be broadly explored and studied.^34–36 Additionally, furthering faculty development strategies will be important to mitigate unwanted variability in assessment of trainee performance, especially around the use of entrustment decision-making tools.^34,37,38

Conclusions

In this study of 3 pediatric residency programs over an entire academic year, we found a strong association between entrustment-level assessments using rotation-specific EPAs and concurrent subcompetency milestones assessments in global end-of-rotation assessments by faculty. Our results lend support to the use of EPAs as an observable component of global faculty-of-resident evaluation assessments.