Introduction

The concept of entrustable professional activities (EPAs) is increasingly being adopted in postgraduate medical education (PGME). EPAs are components of professional practice that can be entrusted to trainees once they have demonstrated the required competence, and the approach facilitates a stepwise progression to unsupervised practice for trainees.¹

Awareness that investment in residents' training is required to enhance optimal quality and cost-effectiveness of health care in the Netherlands has stimulated innovations in PGME. For the country's 17 million inhabitants, about 1000 new residents are admitted to PGME programs to become medical specialists yearly. Programs are organized in 8 teaching regions, each headed by a University Medical Center (UMC). Dutch PGME programs comprise workplace-based learning for 4 to 6 years, and use the CanMEDS competencies. Nearly all residents train partly in UMCs and in large general teaching hospitals.

Progress of pediatric residents is monitored in a specific Electronic Portfolio Assessment Support System (EPASS), comprising a variety of mandatory information, for example, Mini-Clinical Evaluation Exercise (Mini-CEX), feedback, Critical Appraisal of a Topic (CAT), rotation reviews, progress reports, mandatory training courses, test results, and entrustment decisions.

Optimal education and quality and safety of patient care benefit from individualized training with a robust assessment of residents' progress and level of functioning on their trajectory toward unsupervised practice.² An EPA-based training program provides a framework that enables stepwise reduction of the level of supervision as warranted by the progress of individual residents.³ When residents achieve competence at supervision level 4 (are qualified as “competent”),^4,5 a summative entrustment decision signifies the starting point of unsupervised practice.^6,7

Use of EPA-based summative entrustment decisions is intended to enhance the quality of these high-stakes decisions. An individual resident's progress for a particular EPA should be based on a valid, reliable, and reproducible procedure.⁶ In practice, this is not yet the case, and the content and processes used by clinical competency committees (CCCs) making EPA-based summative entrustment decisions are not yet well-established.^8,9

Our objective was to evaluate a national structured process for assessment, aimed at attaining uniform summative entrustment decisions, using an innovative national pediatric training program in which EPAs are embedded.¹⁰

Methods

Dutch pediatric program directors, residents, and educationalists designed and implemented an EPA-based residency training program, referred to as TOP2020 (Training Optimization for Pediatrics in 2020).¹⁰ This entailed a limited number of generic EPAs with a structured group-based summative entrustment decision. The concepts, experience, evaluations, and other documents are available for broader application, as commissioned by the Royal Dutch Medical Association to stimulate PGME innovations in other specialties.

Design of the Summative Entrustment Process

To attain uniform entrustment decisions, we developed a standardized process to prepare for and conduct CCC meetings, termed the Evaluation and Assessment of Residents by Supervisors (EARS) procedure. In the EARS procedure at least 5 staff members are assigned at the start of a resident's rotation to provide feedback regarding their progress over a 3- to 6-month period to monitor their longitudinal progress, deliberately including some subjective judgment (“gut feeling”¹³) by multiple supervisors the trainee has been exposed to during rotations. Residents are encouraged to collect frequent feedback from any other supervisor they work with (eg, during on-call shifts). For each EPA the requirements are described, and a sample EPA “Outpatient clinic for common problems” is shown in table 1. Two weeks before a CCC meeting, the assigned staff supervisors complete an electronic form to provide structured input (see box 2). Residents complete an electronic self-assessment and a checklist of all mandatory data requirements for an EPA. Data are added to the residents' e-portfolios. The final step of the EARS procedure is a CCC meeting, with the timing synchronized with individual residents' progress meetings (2 to 4 times per year). The rotation coordinator or residency program director presents a summary of the aggregated EARS data. All supervisors have an opportunity to explain their input, followed by a succinct discussion that focuses on similarities and differences in the independent assessments. If a resident has requested an entrustment decision (level 4), and the independent judgments of staff supervisors show disagreement, the discussion should result in a consensus. When a resident's request for entrustment is declined, the trainee receives specific recommendations on what to additionally accomplish to be entrusted with the EPA. The results are discussed with the resident during a regular progress meeting held within a week after the CCC meeting. A summary of the EARS procedure is depicted in the figure and in a 9-minute instructional video.¹⁴

table 1 Example of a Standardized EPA Format^10,11

View Fullscreen

View Figure

• Download as Powerpoint
• Download Figure

Results

Between January 2017 and October 2018, 125 Dutch pediatric residents started their training according to TOP2020. In late 2017 we conducted a web-based survey, collecting information from 37 national representatives of program directors and attending staff, and web-based survey data from 112 residents (88 [79%] were female and 88 [79%] had started in 2017 or later). Using a scale of 1 to 10, satisfaction with the new EPA-based curriculum and the EARS procedure scored 7.3 (mean, SD 0.4) and 7.2 (SD 1.2) among attending staff and program directors, respectively. The actual reduction of supervision after an EPA was achieved and was rated 5.3 (SD 1.9) by residents. A summary of the most important advantages and challenges of the program as assessed by staff and program directors at the invitational conferences is presented in box 3.

In total, 51% of resident respondents (57 of 112) completed 2 or more EARS procedures per year, 56% (63 of 112) received comprehensive feedback after CCC meetings, and for 75% (84 of 112) the level of supervision was always or often adjusted for training year, skills, and knowledge (table 2). CCC meetings were attended by 7 staff members (mean, SD 2.7). When a summative entrustment decision was requested by a resident, on average the initial input of 6.0 (SD 2.8) staff members who attended the CCC meeting agreed and 1.1 (1.7) disagreed with the resident's request. In subsequent discussions during the CCC meeting consensus was always obtained. At the time of the survey, 56% (49 of 88) of the prospective residents had achieved 1 or more EPA entrustment decisions. The evaluation of EPAs by residents on 0 to 5 Likert scale is summarized in table 3.

table 2 Resident Questionnaire of Experience With EARS Procedure (n = 112)

View Fullscreen

table 3 Resident Experience With EPAs on 0 to 5 Likert Scale^a

View Fullscreen

Discussion

Our study shows encouraging experiences following the implementation of a national structured process for summative entrustment decision-making using EPAs. Residents generally appreciated the added value of EPAs, and most were aware of the required criteria for summative entrustment decisions. They welcomed the opportunity of stepwise progression toward unsupervised practice and felt more motivated and focused to work on concrete learning objectives, putting them more in charge of their own training, including feeling empowered to set rotation-specific goals related to EPAs.

Although the attending staff was generally well aware of the resident's training level, once an EPA was obtained residents seemed to perceive less reduction of supervision than might be expected. This may reflect staff needing time to get accustomed to the new procedure and provide supervision only on request. We found frequent hesitation to acknowledge the EPAs obtained in another clinical unit, department, or hospital. We expect this to decrease over time, but acknowledge that transition to a different context may require some initial supervisory checks.

Staff were positive about the EARS procedure. Two years following the official introduction, the EARS procedures are used in all centers. Early prospective assignment of staff for future assessment of residents appeared to contribute to the quality of summative entrustment decisions. Staff also acknowledged that this procedure, with independent individual input before a CCC meeting, improved and enriched discussion on resident progress, reducing the tendency of reiteration that is often seen in less structured procedures.¹⁵ Holding staff accountable to provide timely input and to attend scheduled CCC meetings remains a challenge in busy clinical settings.

An inherent issue is that generic EPAs do not have exact determined endpoints. Nevertheless, it is important that the EARS procedure provides ground rules for the process (discussion, based on individual experience of and independent input by staff) and outcome (consensus, regarding an entrustment decision). Program directors have informally reported that even when no uniform recommendation regarding a requested EPA decision is initially provided as part of the EARS procedure, consensus regarding the trustworthiness of an individual resident is (almost) always achieved during the CCC meeting. As the presence of residents could inadvertently affect the frankness and the quality of discussing assessments by staff, residents are not invited to CCC meetings. This policy has support from the national resident representatives.

Supervisors had concerns about legal challenges if EPAs are used as a starting point for unsupervised practice for residents, even with remote supervision available. For physicians in the Netherlands legal accountability is uniform and based on the MD license, irrespective of further experience as a resident or (sub)specialist. There are no legal restrictions for stepwise transfer of responsibility from supervisor to resident as facilitated by certification for an EPA at level 4, but remote supervision remains available while the resident is in training.

The American Board of Pediatrics outlined 17 pediatric EPAs,¹⁶ some of which align with the 9 Dutch EPAs (eg, Manage patients with acute, common diagnosis in an ambulatory emergency or inpatient setting; Demonstrate competence in performing the common procedures of the general pediatrician). However, there also are differences, particularly due to the different health care systems. In the Netherlands, pediatricians provide hospital-based secondary and tertiary care, and this explains the broader scope of the US pediatric EPAs.

Limitations of this study include that our results are preliminary and based on questionnaires and invitational conferences. Second, our program is dedicated to pediatrics, which may limit transferability to other specialties, although monitoring through the EARS procedure could be used as a broader template for robust entrustment decisions across specialties.

Conclusion

An EPA-based national PGME program with a structured process for summative entrustment decisions (the EARS procedure) was found to support the individualized stepwise progression of residents toward unsupervised practice. The results are preliminary, and further research is needed. Broader application of these concepts will require adaptations to accommodate differences in health care systems and training programs.