The Role of Generic Competencies in the Entrustment of Professional Activities: A Nationwide Competency-Based Curriculum Assessed
ABSTRACT
Background
Entrustable professional activities (EPAs) seek to translate essential physician competencies into clinical practice. Until now, it is not known whether EPA-based curricula offer enhanced assessment and feedback to trainees.
Objective
This study examined program directors' and senior residents' justifications for entrustment decisions and what role generic, cross-specialty competencies (such as communication skills, collaboration, and understanding health care systems) play in these decisions.
Methods
Entrustment decisions for all Dutch obstetrics and gynecology residents between January 2010 and April 2014 were retrieved from their electronic portfolios. Justifications for entrustment were divided into 4 categories: the resident's experience, his or her technical performance, the presence of a generic competency, and training. Template analysis was used to analyze in depth the types of justifications, which play a role in entrustment decisions.
Results
A total of 5139 entrustment decisions for 375 unique residents were extracted and analyzed. In 59% of all entrustment decisions, entrusting a professional task to a resident was justified by the experience of the resident. Generic competencies were mentioned in 0.5% of all entrustment decisions. Template analysis revealed that the amount of exposure and technical skills are leading factors, while the quality of the performance was not reported to be of any influence.
Conclusions
Entrustment decisions only rarely are based on generic competencies, despite the introduction of competency frameworks and EPAs. For program directors, a leading factor in entrustment decisions is a resident's exposure to an activity, and the quality of a resident's performance appears to play only a minor role.
Introduction
One of the major challenges for medical training programs all over the world is to comply with society's demand for specialists with a holistic view and a patient-centered approach.1,2 This has resulted in the introduction of non–specialty-specific, generic competencies, such as communication skills, collaboration, and understanding health care systems; these are implemented in training programs with the help of competency frameworks like the Accreditation Council for Graduate Medical Education competencies or the CanMEDS framework.3,4 In these frameworks, competencies are often described as a mix of a professional's knowledge, skills, and attitudes, and they are seen as descriptors of a physician's personal quality.5 By describing these personal qualities, medical educators hope to create a basis for training programs that lead to medical professionals who are fit-for-purpose6 and ready for changes that affect society.7 Implementing curricula based on these competencies has been a challenging exercise, and various approaches have been developed to support this.8,9 A popular approach in recent years makes use of entrustable professional activities (EPAs).
EPAs help to translate competencies into clinical practice.10 EPAs are independently executable, observable, and measurable in their process and outcome, and their use is facilitated in entrustment decisions once residents have reached sufficient competence.11,12 Furthermore, EPAs may contribute to patient safety by making more transparent the issue of who is capable of performing specific tasks, which could prevent situations in which tasks are done by residents who are not fully capable.
Part of the success of competency-based education and a curriculum based on EPAs depends on how well they succeed in incorporating generic competencies in entrustment decisions.13 Decision making about entrustment is complex. Ideally, entrustment is based on evidence of a resident's competence (ie, it is granted if a resident has proven to have sufficient competence to act independently in a specific EPA).14 In theory, evidence of competence may come from several factors: level of performance observed by a supervisor, scores on workplace-based assessments (WBAs), experience in an activity, and training completed by a resident.4,12,15 In practice, entrustment decisions likely are influenced by informal assessments, supervisor characteristics, and the demands of the clinical situation.16 Yet, little research has been done about how entrustment decisions are reached in daily practice17 and whether generic competencies play a role. Consequently, there is a lack of evidence showing whether the introduction of EPAs has expanded the role of generic competencies in these decisions and in feedback provided to residents. This study examined entrustment decisions for Dutch obstetrics and gynecology (ob-gyn) residents who learn in an EPA-based postgraduate curriculum, to answer 2 questions: what do program directors and residents consider important information for an entrustment decision; and to what extent are generic competencies included in entrustment decisions?
Methods
Setting
Curricula based on EPAs have been a part of Dutch medical training programs since the concept was first introduced in 2005. Ob-gyn was 1 of the first specialties to adopt the EPA concept.18,19 Dutch ob-gyn training consists of 6 residency years, which must be spent in at least 2 different teaching hospitals. Training uses a national blueprint based on the CanMEDS-based curriculum. The blueprint also lists all EPAs entrusted to a resident, as well as the generic competencies that should be covered: collaboration, communication, scholarship, health advocacy, management, and professionalism.
One of the requirements for using EPAs is a system for gathering information about the resident's performance (ie, a portfolio system).20 The portfolio is used to gather relevant information about the tasks a resident has worked on, such as WBAs, relevant exposure to clinical tasks, and feedback.21 The information in the portfolio helps the program director make entrustment decisions.
The present research focused on the EPAs related to obstetric care. Obstetric EPAs include technical activities, such as performing a caesarean section, and more generic ones, such as delivering bad news. The study encompassed all 18 obstetric activities used in ob-gyn training (provided as online supplemental material). It excluded 56 gynecology-oriented EPAs, since these are more technical-focused activities, and we expected to pay less attention to the generic competencies.
Data Collection
The data for this study were extracted from the electronic portfolio system used by all Dutch ob-gyn residents, and encompassed all entrustment decisions recorded between January 2010 and April 2014. Residents were responsible for requesting an entrustment level for a specific EPA, using the electronic portfolio. Residents indicated the EPA for which they wanted to be entrusted and the desired level, and provided a brief justification for the request in the portfolio system. This information was sent to the program director, who decides, possibly after consultation with faculty, whether the entrustment request is warranted. The program director fills out a short response to the resident's justification. The exposure to relevant professional activities and WBAs is documented in the portfolio.
Ethical approval for this study was granted by the Netherlands Association for Medical Education Ethical Review Board.
Data Analysis
The justification of the resident and the response by the program director are taken as 1 unit of analysis. In the following analysis, we will talk about justifications, meaning the justification of the resident with the response of the program director as one. All justifications were analyzed to establish which information was crucial for program directors in accepting an entrustment request.
Justifications were divided into 4 categories, selected based on literature about crucial information in entrustment decisions.4,11,13,14 One justification could contain several arguments for entrustment, and therefore could be assigned to more than 1 category. SPSS version 22 software (IBM Corp, Armonk, NY) was used for calculating the descriptive statistics.
To explore whether there are categories of importance in entrustment decisions, beyond those reported in the literature, we performed a qualitative analysis.22 The first step was to select and discuss categories the literature identified as important aspects of an entrustment decision. These included experience, technical performance, presence of a generic competency, and training.4,12,15,16 Experience relates to residents' experience in performing a specific task; technical performance refers to their skill in performing the activity; generic competency focuses on whether the justification for entrustment pays attention to 1 of the generic competency domains; and training relates to justifications based on skills training that a resident has had for the activity for which entrustment is requested. This initial list was adjusted during data sampling to develop the final template. Data were analyzed separately by the first author (K.v.L.) and 3 other researchers (2 PhD students and 1 postdoctoral researcher). Labels were compared, and for differences in labeling, the justifications were reviewed by all researchers until agreement was reached.
Results
From January 2010 to April 2014, a total of 5139 entrustment requests for 375 residents were granted for the 18 obstetric EPAs, and all were included in the analysis. Entrustment requests were granted by 90 program directors working in 46 hospitals and were distributed over all program years (table 1).
Of the 5139 accepted requests, 655 provided a justification that contained more than 1 reason for entrustment, resulting in 5828 components of justifications that were categorized (table 2).
Of the justifications, 59% (3031 of 5139) were categorized into the “experience” group, 20% (1018 of 5139) referred to the resident's technical performance, 9% (449 of 5139) focused on the training a resident had attended, and only 0.5% (26 of 5139) mentioned the presence of a generic competency. Approximately 25% (1304 of 5139) of records did not provide any justification related to the entrustment decision.
For the qualitative analysis, all justifications were labeled and analyzed using the final template developed by the authors (table 3). The following is an overview of the main themes that were found, illustrated by examples.
Exposure
An important factor in the justifications of entrustment requests was whether a resident had previously performed an activity. Comments referred to the number of times the activity had been performed, rather than to the quality of the performance.
Residents also claimed to deserve an entrustment level because they had seen a certain activity performed a number of times. Program directors trusted a resident because he or she had “seen this activity often and should be able to do this independently,” suggesting that some supervisors expected residents to acquire skills required for entrustments solely by seeing an activity performed a number of times. No fixed number was mentioned for the times an activity had to be witnessed for a resident to be entrusted, and the threshold appeared to differ for various activities.
Activities that were rarely performed in practice were replaced by training programs and simulation trainings: “After the resident's last simulation training she should be able to perform this in real practice as well. Entrustment is agreed.” In the entrustments for activities rarely performed in practice, such as a breech extraction, resident participation in a specific training program appeared to be of high importance. The quality of the performance during training was not mentioned, and justifications for whether a resident had participated were just noted.
Competence
“I know how to do this well and I do this flawlessly” or “Skills of this resident are sufficient, knows all the steps for performing this activity safely” are examples of justifications about the level of residents' technical skills. The technical skill of the resident and the knowledge of how and when to use the technique were seen as important reasons to trust a resident to perform an activity independently. In contrast, performance based on generic competencies was rarely mentioned. Justifications that did mention a generic competency mostly referred to the communications domain: “Resident communicates well with nurses and patients and is trusted to enact the activity without supervision.” For activities in which communication is an important aspect, such as “giving bad news,” justifications mostly mentioned the resident's experience and not about their performance on this competency.
Occasionally, program directors only entrusted residents under certain conditions. “Resident is performing okay on this activity, but a supervisor should stay close. . .” Such arguments were commonly given when residents were entrusted in error-prone activities, or those only rarely encountered in practice, such as breech deliveries or multiple babies.
The Role of WBAs
The most commonly used WBAs were the mini-clinical evaluation exercise (mini-CEX)23 and the objective structured assessment of technical skills (OSATS).24 Both played a prominent role in the justifications for entrustment. Unsurprisingly, OSATS were important for the more surgically oriented EPAs, while mini-CEXs were favored for activities that required interaction with patients. Justifications concerning WBAs were found in the following examples: “I have 10 completed mini-CEXs for this activity,” and “enough WBAs to grant the resident the next level of entrustment.” These justifications did not mention the quality of the outcome of the WBAs, only the frequency.
In a few cases, the decision for entrustment was made after requesting additional input from faculty: “All supervisors found the resident capable of doing this alone.” In these cases, it was unclear if the justifications were about the resident's technical skills and/or generic competencies, but it was clear that several supervisors judged the resident to be capable of performing the activity. Arguments about the opinion of other faculty members are to be expected, since the curriculum blueprint stipulates that decisions about entrustment are to be made with faculty input. However, in most justifications in this study, faculty opinion was not mentioned.
Discussion
In this analysis of written justifications for entrustment decisions by Dutch ob-gyn program directors, decisions were mainly based on residents' experience with the given clinical activity. This is despite the aims of an EPA-based curriculum to move the entrustment process focus on experience and technical skills to considering WBAs and the achievement of generic competencies. After experience, written arguments for entrustment mentioned residents' technical performance, while achievement of generic competencies was rarely mentioned. Although present in the residents' portfolios, it is not clear that the quality of residents' performance and/or the assessment data were used in entrustment decisions.
Having an EPA-based curriculum is expected to contribute to assessment and documentation of a resident's performance in generic competencies,25,26 and we expected entrustments based on generic competencies. In contrast, our findings suggest that an EPA-based curriculum does not guarantee the desired attention to generic competencies during entrustment decisions. For instance, the EPA “cesarean section” asks for residents who are capable of performing the incision, leading the surgical team, communicating clearly with the patient about the procedure, and are aware of the newest techniques in the literature. However, this research showed very little mention of these generic competencies. Even technical performance was mentioned less than expected in entrustment justifications. The main focus in entrustment decisions appeared to be the number of times residents performed a procedure, regardless of how well they performed it.
One possible explanation is that the imposed competencies were introduced in the outlines of the training, but program directors, residents, and faculty were uncertain about how to use them in daily training activities.27,28 This uncertainty is in line with our understanding of the implementation hurdles of innovations, which can be subject to unintended, undesired, and unexpected effects.17 The successful implementation of an innovation depends on users' affinity with the innovation and its incorporation into workplace routines.29 The lack of full incorporation of the generic competencies in the daily workplace could have resulted in the limited use of generic competencies during entrustment decisions.
Another explanation could be that assessing generic competencies in practice is still problematic, resulting in little useful data to inform entrustments.15 For some time, assessment has focused predominantly on psychometric approaches, which are useful for assessing technical skills, but less suitable when it comes to measuring complex and relatively less well-defined domains like the generic competencies.30
Faculty development has been suggested as a means to overcome some of these barriers, to deal with the observation that faculty members often are insufficiently prepared for implementing competency-based medical education using EPAs and assessing generic competency domains.31,32 In addition, the design of the portfolio system could also have influenced how faculty executed entrustment decisions. For instance, the large number of blank argumentations highlights flaws in the design of the portfolio. Redesigning the way entrustment decisions are being requested could stimulate added attention on generic competencies.
Another possible explanation for the minor role of generic competencies in entrustment decisions could be that they may need to fit current practice. Several recent studies have suggested a need for reconsidering current competency frameworks.33–35 Adjusting the generic competencies to clinical practice and thereby increasing users' affinity could increase their use in entrustment decisions.
Limitations of this study include the fact that it was solely based on documentation in residents' portfolios, which does not show all inputs that influence entrustment decisions. In addition, decisions not to entrust a resident were not represented in the portfolio, and therefore could not be taken into account. Another limitation of this study was that the conclusions do not make a distinction between residents' and supervisors' arguments. The final limitation is that the data are from a single specialty, reducing the ability to generalize. Further research is needed to investigate if supervisors take generic competencies into account during entrustment, even if they are not mentioned in entries in resident portfolios.
Conclusion
For Dutch ob-gyn residents, generic competencies such as communication or collaboration received limited attention in written justifications for entrustment decisions. The transition to full competency-based training appears to need more than implementation of EPAs alone. Supervisors and residents should be trained to enhance their use of generic competencies in daily activities, and in giving and receiving feedback,36 to ensure entrustment decisions are based on competencies and not perceptions of experience or exposure.
Author Notes
Funding: The authors report no external funding source for this study.
Conflict of interest: Mr. van Loon and Dr. Scheele have been involved in the creation of the curriculum plans for the Dutch Obstetrics and Gynecology training. Dr. Driessen has been involved in the creation of the portfolio system Epass.
The authors would like to thank Joyce Moonen for retrieving all relevant entrustment decisions from the Epass database.
Editor's Note: The online version of this article contains a list of 18 entrustable professional activities of the Dutch obstetrics and gynecology training in random order.
