Monitoring Competency-Based Medical Education Uptake: Analysis of Entrustable Professional Activity Submission, Expiration, and Assessment Scores
ABSTRACT
Background Program directors need concrete indicators to monitor uptake of competency-based medical education (CBME). Entrustable professional activity (EPA) observation completion rates offer practical measures of CBME adoption.
Objective In this study, we used residents’ EPA observation data in clinical departments, specifically the submission and expiration of EPA observation forms and assessment scores, to explore the uptake of CBME practices across departments. Our research question asked: What are the patterns and contributing factors (department group, resident year, calendar year, program size) associated with EPA observation submission rates, expiration rates, and assessment scores?
Methods We conducted exploratory analysis of de-identified EPA observation data (n=233 176) from residents’ electronic portfolios (n=2110) across 45 programs in 12 departments at one Canadian institution from 2018 to 2023. Descriptive statistics summarized submission, expiration, and score distributions. Spearman correlations and logistic regression examined 4 predictors: department group, resident year, calendar year, and program size.
Results EPA submission rates (81.0%), expiration rates (7.7%), and assessment O-scores (M=4.4 out of 5) did not differ significantly by training department. Calendar year increased odds of an independent or full score by 26.3% per year (OR, 1.263; 95% CI, 1.259-1.267) while resident year (OR, 0.818; 95% CI, 0.813-0.825) and program size (OR, 0.995; 95% CI, 0.994-0.996) decreased those odds.
Conclusions EPA submission, expiration, and scoring patterns are consistent across departments and correlate with implementation year, resident training stage, and program size.
Introduction
Despite a national rollout in Canada of competency-based medical education (CBME) in 2017, there are limited quantitative measures for evaluating the adoption of key aspects of CBME implementation, yet such data could inform continuous quality improvement (CQI). Completion rates of entrustable professional activity (EPA) observations serve as direct, actionable metrics for CQI.1
To address this gap, we examined 3 indicators of CBME uptake—EPA submission rates, expiration rates, and assessment scores—across 12 departments. We explore how these metrics vary by department group, resident year, calendar year, and program size. These factors are chosen for their relevance to supervisory accountability structures and program resources. The research question that leads the study is: What are the patterns and contributing factors (department group, resident year, calendar year, and program size) associated with EPA observation submission rates, expiration rates, and assessment scores?
Methods
Participants
We used routinely collected resident assessment data for secondary analysis. EPA observations (n=233 176) from residents’ electronic portfolios (n=2110) were collected from 45 residency programs within 12 departments at one Canadian institution from 2018 to 2023. Data were de-identified by data stewards in the Office of Postgraduate Medical Education who were outside the study team. Each year there are approximately 875 residents. The number of residents per program ranged from 2 to 436. Programs included in this study launched CBME between 2017 to 2023. Results are reported in 4 groups of related departments to ensure anonymity as described in Table 1. While the initiation/completion of EPA observations involve participation of both the resident and the supervisor, supervisors are often the rate-limiting source.2 Supervisors in this institution are accountable to their department chairs, rather than residency program leadership, for their teaching engagement and quality. For these reasons, the supervisors’ department was chosen as the unit of analysis rather than the residents’ program. The number of residents (2018 to 2023) in each department group includes acute care: 322; diagnostic: 94; surgical: 629; and medical: 1065.
Materials
The institution’s Office of Postgraduate Medical Education provided program-level details: program name, annual enrollment, CBME implementation start year, and stage duration benchmarks. Each EPA record included initiation and submission time stamps, expiration status, and an assessment score on a 1-5 entrustment scale (“O-score,” 1=fully supervised to 5=independent). In 2020, saved forms expired after 30 days; in 2021, this window was reduced to 14 days. Cancelled forms were excluded from analysis.
Data Analysis
We calculated the EPA submission rate as the proportion of submitted forms to initiated forms and the expiration rate as the proportion of expired forms to the sum of expired and submitted forms. Descriptive statistics, including 95% confidence intervals, summarized submission rates, expiration rates, and mean O-scores by department group. To examine factors associated with EPA scoring, we performed Spearman correlation analyses between mean O-scores and 4 predefined predictors: department group, resident year in training, calendar year of data collection, and program size (number of residents). We then used logistic regression to model the odds of achieving an independent score (O-score=5) based on these same predictors.
This study was approved by the institutional research ethics board (#Pro00140071).
Results
Table 1 summarizes EPA submission rates, expiration rates, and mean assessment scores by department group. Overall, the submission rate was 81.0% (95% CI, 80.8-81.2), with departmental rates ranging from 77.6% to 83.3% (difference=5.7 pp; 95% CI, 5.5-5.9 pp). Expiration rates averaged 7.7% (95% CI, 7.6-7.8), ranging from 4.6% to 9.2% (difference=4.6 pp; 95% CI, 4.4-4.8 pp). Mean O-scores were 4.40 (95% CI, 4.39-4.41), with department means between 4.36 and 4.48 (difference=0.12; 95% CI, 0.11-0.13).
Correlations
We examined 4 predictors—calendar year, resident year in training, program size, and department group—and their association with mean O-scores. As the Figure shows, Spearman correlations were strongest for calendar year (ρ=0.48; 95% CI, 0.47-0.49) and weaker for resident year (ρ=-0.13; 95% CI, -0.14 to -0.12) and program size (ρ=-0.10; 95% CI, -0.11 to -0.09).
Citation: Journal of Graduate Medical Education 17, 5; 10.4300/JGME-D-25-00152.1
Predictors of Independent Scores
Logistic regression (Table 2) indicated that each additional calendar year increased the odds of an independent O-score by 26.3% (OR, 1.26; 95% CI, 1.25-1.26), whereas each additional resident training year reduced those odds by 18.2% (OR, 0.81; 95% CI, 0.81-0.82) and each additional program resident reduced them by 0.5% (OR, 0.99; 95% CI, 0.99-0.99). Submission completion time showed a nonsignificant association (OR, 0.99; 95% CI, 0.99-1.00).
Discussion
While department grouping did not significantly affect EPA submission, expiration, or assessment score metrics, later implementation years were associated with higher submission rates and faster completion times. Additionally, residents received fewer “independent” EPA scores as they progressed, reflecting the increasing complexity of later-stage EPAs.
Several prior studies have debated whether procedural versus diagnostic specialties experience smoother CBME adoption.3,4 Our cross-departmental analysis extends this work by demonstrating minimal variation across specialty clusters, suggesting that program-level systems (eg, electronic portfolios, supervisory accountability) promote uniform adoption.
The staged Royal College of Physicians and Surgeons of Canada CBME framework provides a conceptual basis for our progression findings.1 Early stages emphasize simpler EPAs achieved rapidly, whereas the longer core stage contains complex activities requiring more time. Our regression results—showing each additional training year reduced odds of an independent score (OR, 0.82; 95% CI, 0.81-0.83)—support the need for stage-specific benchmarks in CQI dashboards.
Limitations include reliance on a single institution’s data and binary categorization of scores, which may oversimplify competence decisions. Future multicenter studies incorporating narrative feedback and committee deliberations would deepen understanding of CBME uptake.
Conclusions
Overall, our findings demonstrate that EPA submission rates, expiration rates, and assessment scores serve as consistent, department-agnostic metrics of CBME adoption and are significantly associated with implementation year, resident training stage, and program size.
Correlation Among Variables
Author Notes
