Using the Entrustable Professional Activities Framework in the Assessment of Procedural Skills
ABSTRACT
Background
The entrustable professional activity (EPA) framework has been identified as a useful approach to assessment in competency-based education. To apply an EPA framework for assessment, essential skills necessary for entrustment to occur must first be identified.
Objective
Using an EPA framework, our study sought to (1) define the essential skills required for entrustment for 7 bedside procedures expected of graduates of Canadian internal medicine (IM) residency programs, and (2) develop rubrics for the assessment of these procedural skills.
Methods
An initial list of essential skills was defined for each procedural EPA by focus groups of experts at 4 academic centers using the nominal group technique. These lists were subsequently vetted by representatives from all Canadian IM training programs through a web-based survey. Consensus (more than 80% agreement) about inclusion of each item was sought using a modified Delphi exercise. Qualitative survey data were analyzed using a framework approach to inform final assessment rubrics for each procedure.
Results
Initial lists of essential skills for procedural EPAs ranged from 10 to 24 items. A total of 111 experts completed the national survey. After 2 iterations, consensus was reached on all items. Following qualitative analysis, final rubrics were created, which included 6 to 10 items per procedure.
Conclusions
These EPA-based assessment rubrics represent a national consensus by Canadian IM clinician educators. They provide a practical guide for the assessment of procedural skills in a competency-based education model, and a robust foundation for future research on their implementation and evaluation.
Introduction
Over the past decade, there has been a paradigm shift in health professions education toward a competency-based education (CBE) model.1,2 One of the challenges in CBE has been the increased need to provide robust assessment of competencies achieved by trainees, especially for workplace, criterion-based assessment.3
A key element of CBE is the expectation that graduates become competent in performing procedures, with certifying boards in Canada and the United States having specific procedural skills requirements.4,5 Still, many trainees do not master these skills during residency and report feeling ill-prepared to perform procedures,6–8 and this may be partly due to limited opportunities for formal assessment during training.9
Studies of assessment methods for procedural skills have largely focused on technical ability, and have not considered nontechnical skills that are equally critical for successful procedural performance.10–13 These skills include the ability to communicate effectively (eg, obtaining informed consent); to collaborate with other professionals; to act in a professional manner; and to exercise clinical reasoning and judgment.14 There have been attempts to assess procedural skills in context,15–17 but it remains unclear which nontechnical skills are most important to assess.
The entrustable professional activities (EPAs) framework is 1 approach to assessment in a CBE model.18,19 EPAs are observable, holistic activities that require the integration of knowledge, skills, and attitudes across competency domains,20 and they also ground competencies within a physician's day-to-day work. EPAs differ from competencies in that they refer to the components of physicians' work, while competencies refer to the abilities of physicians.21 Examples of EPAs include managing intravenous fluids, disclosing a medical error,22 or performing a lumbar puncture.
Although there are published reports on how to develop EPAs,22–26 there is relatively little guidance on how to use them in assessment. One approach entails use of entrustability-aligned anchor scales,27 which reflect the continuum of abilities ranging from passive observer to being able to perform an activity independently and provide supervision to others. Rating scales that use entrustability anchors produce more reliable scores than conventional anchors.28
To date, there is only a small amount of literature to guide the assessment of procedural skills within an EPA framework. Conceptually, using entrustability-aligned anchors makes sense in the assessment of procedural skills because they rely on an expert's holistic judgment regarding whether or not a trainee can be entrusted to perform a procedure (or components thereof). When compared with traditional assessment methods, which tend to focus on a series of checklist items and/or ratings, an entrustment framework may better reflect the realities of clinical practice.
In other words, a supervisor may not trust the abilities of a trainee even if they have received a high score on an examination of procedural skills (eg, if they missed 1 key step, or if the examination did not assess all the competencies required in performing the procedure). Instead, the supervisor makes an overall judgment about a trainee's ability based on a number of sources of information related to the essential skills required to successfully complete the procedure.29 For EPAs to be used in a program of assessment, the essential skills necessary for entrustment to occur must first be identified. In this study, we aimed to (1) define the essential skills required for entrustment to occur for 7 bedside procedures, and (2) develop an EPA-based assessment tool for procedural skills.
Methods
In this multisite study, we defined EPAs for 7 bedside procedures: central venous catheter insertion, lumbar puncture, peripheral arterial catheter insertion, paracentesis, endotracheal intubation, thoracentesis, and knee arthrocentesis. The process for defining the components (ie, essential skills) of the procedural EPAs was divided into 3 phases.
Phase 1: Defining EPA Components
Using a purposive, maximum variation sampling strategy (aimed at capturing a wide range of perspectives),30 the investigators recruited subject matter experts (SMEs) at each of the 4 participating sites (University of British Columbia, University of Calgary, University of Toronto, and University of Ottawa) to participate in focus groups. The SMEs were clinicians with expertise in performing the procedure of interest, who were involved in teaching and/or assessment of residents' procedural skills. To ensure a sample with maximum variation, SMEs included physicians from a variety of specialties, who practiced in both academic and community settings, and who performed procedures in different contexts (eg, intensive care unit versus ambulatory clinic).
Data Collection—Focus Groups
Although the essential skills or components required for procedures may be generic, we opted to define the components for all bedside procedures separately and to explore potential differences. We assembled 7 focus groups (1 for each procedure) and asked them to define the components for procedural-specific EPAs. The focus groups were divided among the participating sites (ie, 1 or 2 focus groups per site).
Participants were asked to generate a list of the essential skills required for entrustment to occur (to allow a resident to perform a procedure without supervision). Essential skills required for each EPA were defined using the nominal group technique.31,32 Three iterations were required to achieve consensus on all items. From this information, a final list of EPA components was compiled.
Data Analysis—Focus Group Field Notes
After generating the final list of EPA components, the field notes from each focus group were analyzed using a framework approach,33 with a focus on participants' rationales for accepting or rejecting potential EPA components. This analysis helped to inform the development of the EPA component lists for a national survey.
Phase 2: National Survey
To ensure representation from eventual end users, we invited SMEs representing all Canadian internal medicine (IM) and general IM training programs to complete a web-based survey (SurveyMonkey, FluidSurveys, Ottawa, Ontario, Canada). Using a snowball sampling technique, program directors were contacted and asked to identify 8 to 12 practicing physicians involved in their procedural skills curriculum (development, teaching, or assessment) and 8 to 12 residents with experience in performing the procedure of interest, including IM or general IM residents (postgraduate year 3 and above).
National Survey
A survey was developed comprising the final list of EPA components for each procedure developed in phase 1. The survey was designed and revised by all coauthors to capture consensus on the proposed EPAs based on small group sessions. Responses indicating that items “should not be included” were followed by a text box allowing respondents to expand on their rationale for disagreement (see online supplemental material for a copy of the survey). For each EPA component that they believed should be excluded from the final list, they were asked to justify their decision with written comments. Demographic information (staff versus resident and university affiliation) was also captured for reporting purposes. The survey was pilot tested by the authors.
Experts identified by IM program directors were sent a link to the web-based survey. For each EPA component, agreement was defined as at least 80% consensus. A follow-up survey was distributed that included only items for which consensus had not yet been reached.
Analysis
Response and consensus percentages rates were calculated for each procedure. Qualitative survey comments were analyzed using a framework approach,33 with a particular focus on exploring participants' rationale for the exclusion of potential EPA components.
Phase 3: EPA Rubric Development
In the development of a final EPA rubric for each procedure, the initial lists of EPA components were modified by the investigators based on comments from participants in the national survey. The final lists of EPA components were then used to create the assessment rubrics.
Ethics approval was obtained from each of the 4 participating sites (the universities of British Columbia, Calgary, Toronto, and Ottawa).
Results
Five groups had 8 participants each, 1 group had 7 participants (thoracentesis), and 1 group had 10 participants (paracentesis). Participants (N = 57) were from the specialties of IM, anesthesiology, emergency medicine, and surgery. Eight participants were residents (fourth and fifth years), 1 was a respiratory therapist, and the remainder were practicing physicians, including 3 practicing in a community setting.
The qualitative analysis resulted in refinements of the lists by either subdividing or collapsing items. In addition, stylistic changes (eg, using a verb at the beginning of each component) were made to help standardize the lists. Following this, the revised lists included a range of 12 (thoracentesis) to 21 (central venous catheter insertion) components for each EPA.
A request to distribute a link to the web-based survey was sent to all Canadian IM (n = 17) and general IM (n = 16) program directors. A total of 111 physicians responded (50 practicing physicians and 61 residents). Responses were received from 14 different schools, with respondents from each university ranging from 1 to 18, with a mean of 8.
Consensus was reached on 101 of 102 items (99%) with the first iteration of the survey. For 20 items, there was 100% consensus. By the second iteration (54 respondents), consensus was reached on the 1 outstanding item. In all instances, the consensus was to include the item.
Following this, the qualitative analysis of comments was used to inform any further changes to the final lists. Themes (table) were identified independently by 2 of the coinvestigators (D.P. and M.M.). Using this analysis, final lists of EPA components were modified to produce a rubric to guide the assessment of procedural competence in IM residents. Modifications included adding headings for competencies34 (ie, medical expert, communicator, and professional) and grouping items that were thought to be linked (eg, performing the technical aspects and maintaining sterility).
One item (“knowing how to rewire a line”) was removed from the central line list because 8 respondents identified it as being below the standard of care. For the item “ensuring the availability of an assistant” for an arterial line insertion, 7 respondents noted that an assistant was not always necessary, and the item was revised to include “if required.” Two items (“demonstrating concern for patient comfort and well-being” and “demonstrating an ability to communicate and behave professionally”) were removed based on comments from participants, noting these skills (ie, professionalism, empathy) would apply to all patient interactions.
Finally, we standardized wording across procedures for common items (eg, “obtaining informed consent”) and merged related items for knee arthrocentesis into a single item of “knowing indications/contraindications.”
Entrustability-aligned anchors (ranging from “no task execution” to “supervision may be provided to junior learners”) were then added to the rubric, as adapted from previously published tools.23,25 Final rubrics included 6 to 10 items per procedure (see online supplemental material for a copy of the rubrics).
Discussion
Our study used an EPA framework to develop rubrics to assess IM residents' performance of 7 bedside procedures. These procedural EPAs represent, to our knowledge, the first consensus-driven framework that conceptualizes and delineates the different skills necessary for procedural competence (ie, beyond the technical checklists currently in use). The solicitation of feedback from stakeholders was part of a broader knowledge translation approach that emphasizes the inclusion of knowledge users. The acceptability of an assessment tool is an important factor in determining its use.35 By engaging eventual end users, the final rubrics are more likely to be accepted and incorporated as a tool across programs.
This study used the nominal group technique and a modified Delphi procedure to define the essential skills required for entrustment of residents performing procedures. All items included in the initial lists of EPA components developed by local focus groups were rated as being important by the majority of respondents in a national survey. This adds value, because insights from a diverse group ultimately influenced rubric development and provided an important source of validity evidence (content evidence)36 that contributes to the acceptability of the final product.
An important question when developing EPAs relates to how detailed they should be. EPAs are meant to be relatively broad; therefore, it is necessary to provide some guidance about what, exactly, the EPA encompasses. EPAs can be disaggregated into a list of essential skills that can be mapped to competencies.24,29 However, as EPAs are deconstructed, the risk is that they may become too checklist-driven. In this study, the risk was mitigated by grouping the EPA components into categories so that the number of items in each final EPA rubric was kept to a manageable number.
Our rubrics for assessing procedural skills are to be used as a tool to guide entrustment decisions as residents progress through training.24 The use of entrustability scales allows the level of supervision required to be made explicit, thus helping to ground entrustment decisions. Although they are meant to be used when skills are directly observed, they can be used, or adapted for use, in multiple settings, including when trainees perform procedures on part task models, hybrid settings, or real patients. Importantly, the rubrics allow learners to be assessed along a continuum as they grow in competence. This allows for evaluation of the progression of competency for a particular trainee and facilitates teaching, assessment, and eventual remediation.
Our study serves as a template for the development of other EPA-based competency assessment tools. Although this study focused on the development of EPA-based tools for the procedures performed by internists, the methods used could be applied to other procedural skills, and the developed tools could be adapted to procedures performed by other specialties.
Limitations of this study include the inherent subjectivity of the modified Delphi approach. In addition, the survey data should be interpreted with caution as it is possible that participants' responses were influenced by the lists provided (ie, confirmation bias).
Conclusion
As accreditation bodies in medical education move toward a CBE model, new assessment frameworks are needed to ensure progression and attainment of competency in different domains. The EPA-based assessment rubrics developed in this study represent a national consensus by Canadian clinician educators in IM. The rubrics provide a practical guide for the assessment of procedural skills within a CBE model and a robust foundation for future studies of their implementation and evaluation.
Author Notes
Funding: This research was supported by grants from the Medical Council of Canada and the Department of Innovation in Medical Education, University of Ottawa.
Conflict of interest: The authors declare they have no competing interests.
This study was presented as a poster at the International Conference on Residency Education, Niagara Falls, Ontario, Canada, October 1, 2016.
Editor's Note: The online version of this article contains the assessment rubrics and the EPA survey.
