Influence of Clinical Competency Committee Review Process on Summative Resident Assessment Decisions
ABSTRACT
Background
Clinical Competency Committees (CCCs) are charged with making summative assessment decisions about residents.
Objective
We explored how review processes CCC members utilize influence their decisions regarding residents' milestone levels and supervisory roles.
Methods
We conducted a multisite longitudinal prospective observational cohort study at 14 pediatrics residency programs during academic year 2015–2016. Individual CCC members biannually reported characteristics of their review process and Accreditation Council for Graduate Medical Education milestone levels and recommended supervisory role categorizations assigned to residents. Relationships among characteristics of CCC member reviews, mean milestone levels, and supervisory role categorizations were analyzed using mixed-effects linear regression, reported as mean differences with 95% confidence intervals (CIs), and Bayesian mixed-effects ordinal regression, reported as odds ratios (ORs) and 95% credible intervals (CrIs).
Results
A total of 155 CCC members participated. Members who provided milestones or other professional development feedback after CCC meetings assigned significantly lower mean milestone levels (mean 1.4 points; CI –2.2 to –0.6; P < .001) and were significantly less likely to recommend supervisory responsibility in any setting (OR = 0.23, CrI 0.05–0.83) compared with CCC members who did not. Members recommended less supervisory responsibility when they reviewed more residents (OR = 0.96, 95% CrI 0.94–0.99) and participated in more review cycles (OR = 0.22, 95% CrI 0.07–0.63).
Conclusions
This study explored the association between characteristics of individual CCC member reviews and their summative assessment decisions about residents. Further study is needed to gain deeper understanding of factors influencing CCC members' summative assessment decisions.
Introduction
With the advent of milestone-based assessment, the Accreditation Council for Graduate Medical Education (ACGME) has required programs to convene Clinical Competency Committees (CCCs).1 The literature on CCCs to date has largely focused on how-to approaches for designing a CCC2–5 and potential best practices for the CCC review process.6–15
While studies have not yet closely examined the work performed by CCCs, 2 aspects of their efforts stand out. First, CCCs focus on making summative assessments in many cases without direct observation of residents in the clinical environment.1,2,16 In contrast, much of the milestone and entrustment literature has focused on frontline assessors.17–23 Second, while milestones and CCCs could represent a new framework for planned, vetted reviews of resident performance, recent discussions suggest that in actuality, milestones may serve as the impetus to create a more intentionally designed program of resident assessment.24 Yet, relatively little is known about the summative assessment work of CCCs. We conducted a multi-institutional study to explore associations between characteristics of individual CCC member reviews and the summative assessment decisions they make about resident physicians, namely ACGME milestone levels and recommended supervisory role categorizations (ie, whether or not residents may serve in supervisory capacities and in what settings).
Methods
Setting and Participants
This longitudinal prospective observational cohort study was conducted during the 2015–2016 academic year. Fourteen pediatrics residency programs in the Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN) participated (participant information provided as online supplemental material). Programs were chosen for their representative range of size, program type, and geographic location. Geographic location was due to the possibility of regional differences in how groups of programs may approach several aspects of residency education.
Eligible study subjects at each program included current CCC members, all categorical pediatrics residents, and program directors. For feasibility purposes, programs were given the option to report data for all pediatrics residents or a defined subset of residents whose performance spanned the full range of performance.
Intervention
Data collection occurred during biannual CCC reviews and milestone reporting periods of the academic year (winter 2015 and spring 2016), with data submitted to the research network between December 2015 and August 2016. Site leads were asked to recruit individual CCC members at their program via e-mail. These participants provided details about their review processes through an electronic survey (provided as online supplemental material).
Outcomes
During each review cycle, individual CCC members were asked to provide 2 study variables for the residents reviewed (figure 1): ACGME milestone levels (1–5 scale, by 0.5 increments) selected for the 21 ACGME reporting competencies and their recommended supervisory role categorization, choosing from 5 categories (no settings to all settings), including inpatient and outpatient general pediatrics and intensive care (figure 2). This entrustment inference (“readiness to supervise others”) is 1 of several defined by the Pediatrics Milestones Assessment Collaborative (an effort of APPD LEARN, the American Board of Pediatrics, and the National Board of Medical Examiners). It embraces a broader view of entrustment22,23,25 than focusing on individual specific entrustable professional activities to determine the level of entrustment germane to the specialty.26–29
Citation: Journal of Graduate Medical Education 10, 4; 10.4300/JGME-D-17-00762.1
Citation: Journal of Graduate Medical Education 10, 4; 10.4300/JGME-D-17-00762.1
One program did not have a review process where CCC members were assigned residents, and the program director reported the consensus decisions of the group. Program directors also provided information about how many CCC members prereview residents prior to full CCC meetings. Study questions were developed, reviewed, and edited by a group of 12 residency and medical education research leaders through an iterative process without field testing.
The Institutional Review Board (IRB) at Cincinnati Children's Hospital Medical Center (lead site) granted exempt status to this study, and at each participating program, the IRB reviewed and either approved or exempted the study.
Analysis
Frequencies and descriptive statistics (mean, range, interquartile range) for the self-reported CCC member review characteristics were calculated using R version 3.1 (The R Foundation for Statistical Computing, Vienna, Austria).30 The ACGME milestone levels were averaged across the 21 competencies for each resident to construct a mean milestone level, called a summative milestone profile (ranging from 1 to 5). Our goal in developing this summative profile was to aggregate milestone ratings into a single summative assessment for the purpose of comparison. For 3 pediatrics milestones, levels are intended to range from 1 to 4, but because many programs submitted at least 1 level of 4.5 or 5 on 1 competency, we did not transform levels on these competencies before averaging them. A sensitivity analysis that excluded these competencies from the summative milestone profile did not show a change in the results.
We examined the relationship between characteristics of individual CCC member reviews and the summative milestone profile by fitting a set of linear mixed models to mean milestone levels with resident year, with each review characteristic variable as a fixed-effect predictor, and random effects for resident, program, and CCC member, using the lme4 and lmerTest packages.31,32 We used histograms of the residuals to visually confirm roughly normal distributions of error in the mixed models, and report both unadjusted P values and P values adjusted for multiple comparisons using the Holm-Sîdak procedure.33 We entered characteristics with univariate regression coefficients significant at P < .02 into a single multivariate linear mixed model, along with resident year and random effects.
We also examined the relationship between characteristics of individual CCC member reviews and recommended supervisory role categorizations, excluding 8 reports of “unable to determine.” We collapsed the supervisory role categorizations into 3 categories: “may not serve,” “may serve in some settings,” and “may serve in all settings.” This was done to simplify the results for easier understanding. We then fit a set of mixed-model ordinal (continuation ratio) regressions to the supervisory role categorization, with resident year as a fixed nominal effect predictor, each single review characteristic variable as fixed, category-specific, nominal effect predictor, and random effects for resident, program, and CCC member, using full Bayesian inference with an uninformative prior and multi-chain Monte Carlo sampling to provide unbiased estimates and credible intervals via the Stan system34 and the brms R package.35
In analyses of the relationship between types of information reviewed and summative assessment decisions, we only analyzed options reported by the majority of CCC participants because 31 of 35 types of information reviewed were reported only by a small number of participants. We report effect sizes and 95% confidence or credible intervals for each review characteristic predictor.
Results
Across the 14 sites, 155 of 192 CCC members and all 14 program directors participated in this study. Over 2 review cycles (midpoint and end of academic year), participants reported milestone assignments and supervisory role categorizations for 463 of 852 residents at the study sites (307 both cycles; 34 fall only; 122 spring only). Supervisory role categorizations were distributed as follows: able to supervise in all settings (level 5, n = 512), all settings but borderline (level 4, n = 56), some settings (level 3, n = 47), some settings but borderline (level 2, n = 80), not able to serve as a supervisor (level 1, n = 67), and unable to assign a level (n = 8).
Individual CCC members reviewed a median of 8 residents per cycle (interquartile range [IQR] 4–15, table 1). Most members (97%, 150 of 155) indicated that they reviewed residents before CCC meetings. Based on program director reports, residents were reviewed by 1 CCC member before a full CCC meeting at 8 programs. Three of the programs had additional members review select residents with identified concerns before or during the primary review. The remaining 6 programs had 2 or more CCC members prereview all residents prior to a full CCC meeting. When serving as the primary reviewer assigned to residents, individual CCC members spent a median of 1 hour (range 0–8, IQR 0.5–1.5) reviewing a resident and a median of 0.5 hours (range 0–3, IQR 0.25–1) when serving as a secondary reviewer. As table 1 and the box show, individual CCC members reviewed a variety of assessment data types, predominated by written comments (eg, narrative assessment data) in end-of-rotation assessments (95%, 147 of 155). Most CCC members provided milestones or other professional development feedback to all (44%, 68 of 155) or some (26%, 41 of 155) of the residents reviewed. Finally, most CCC members (70%, 109 of 155) had been part of more than 2 CCC review cycles.
Association Between Characteristics of CCC Member Reviews and Summative Milestone Profiles
After adjusting for multiple comparisons and controlling for resident year and resident clustering in CCC member, program, and review cycle, CCC members assigned lower summative milestone profiles when they provided post-CCC meeting milestones or other professional development feedback to all reviewed residents, compared with members who did not provide such feedback (regression coefficient –0.55; 95% CI 1.28–0.19; table 2), along with other significant predictors. These CCC members assigned an average milestone level 1.4 levels lower than faculty who did not provide such feedback. We left the characteristic “first CCC review cycle” out of the multivariate model as it was linearly dependent with the “total number of CCC review cycles” predictor, which we retained. In the multivariate model, all significant univariate predictors remained significant and had similar coefficients (table 2, right column).
Association Between Characteristics of CCC Member Reviews and Recommended Supervisory Role Categorizations
Reviewing more residents during the current review cycle was associated with individual CCC members being significantly more likely to place residents in the “some settings” category compared with “all settings” (odds ratio [OR] = 0.96; 95% credible interval [CrI] 0.94–0.99; table 3).
Completing reviews of residents prior to the full CCC meeting was significantly associated with individual members recommending residents in categories allowing for less supervisory responsibility, with both “no settings” being more likely than “some settings” and “some settings” being more likely than “all settings.”
When reviewers provided post-CCC meeting milestones or other professional development feedback to all residents reviewed, they were more likely to place them in the “no settings” rather than “some settings” category (OR = 0.23, 95% CrI 0.05–0.83).
Finally, being involved in more biannual CCC processes in the past was associated with being more likely to place a resident in the “some settings” category compared with “all settings” (OR = 0.22, 95% CrI 0.07–0.63).
Discussion
In this study, we found that individual CCC members who provide milestones or other professional development feedback to residents assigned lower milestone levels and recommended less supervisory role responsibility. In addition, reviewing more residents during a given cycle, being involved in more biannual CCC cycles, and completing reviews of residents prior to full CCC meetings were all associated with CCC members recommending residents be granted less supervisory responsibility.
This study suggests that individuals' experience and attributes of their review process influence the summative assessment decisions they make as members of the CCC, including that reviewing more residents may lead individual CCC members to be more stringent in assigning summative assessment decisions. This raises the question of whether they are more discerning based on their vantage point of reviewing a larger number of composites of residents' performance, or whether they may satisfice judgments due to time pressures or other factors. The CCC members completing their first cycle also assigned lower milestone ratings. Explanations may exist for these seemingly contradictory findings, such as CCC members being more cautious when completing their first cycle, and thus more likely to assign lower milestones.
Individual CCC members who completed resident reviews prior to full CCC meetings were similarly more stringent when recommending supervisory roles. This finding highlights the importance of further study of CCC member decisions made before, compared with during, full CCC meetings. Future efforts should seek to elucidate the role and value of individual CCC member review versus group-level decisions, including potential sources of bias.2,36
Stringency in milestone ratings and supervisory roles was also observed when individual CCC members provided milestone or other professional development feedback to all residents they reviewed. This finding warrants further study to determine if providing performance feedback develops relationships with residents that allow for more honest and accurate summative assessment, whether providing such feedback happens more frequently with lower-performing residents, or if other explanations exist.
This study has limitations. We did not gather data on faculty training relevant to assessment. We did not test our survey questions for validity evidence, and respondents may not have interpreted questions as intended. Data were reported by individual CCC members without objective measures of assessment programs or review processes, and we did not include an objective measure of resident performance for comparison to the summative milestone profiles and supervisory role categorizations. This study was conducted in 1 specialty, and its results may not generalize. Finally, we considered only the role that review characteristics of CCC members played in their summative assessment decisions.
Conclusion
This study found that individual CCC members who reviewed more residents during a given CCC review cycle were involved in more biannual CCC review cycles, completed reviews of residents prior to full CCC meetings with all or most CCC members present, and provided milestones or other professional development feedback to residents assigned lower summative milestone ratings and supervisory roles to residents.
Study Data Collection Process
Supervision Categories
Author Notes
APPD LEARN CCC Study Group Members: Kathleen Bartlett, MD, Associate Professor and Associate Program Director, Department of Pediatrics, Duke University; Natalie Burman, DO, MEd, Assistant Professor and Associate Program Director, Department of Pediatrics, Naval Medical Center San Diego; Sharon Calaman, MD, Associate Professor and Pediatric Program Director, Department of Pediatrics, Drexel University College of Medicine, St. Christopher's Hospital for Children; John G. Frohna, MD, MPH, Professor, Departments of Pediatrics and Internal Medicine, University of Wisconsin School of Medicine and Public Health; Lynn C. Garfunkel, MD, Associate Program Director, Department of Pediatrics, University of Rochester; Caren Gellin, MD, Assistant Professor and Associate Program Director, Department of Pediatrics, University of Rochester; Su-Ting Li, MD, MPH, Professor, Vice Chair of Education, and Pediatric Program Director, Department of Pediatrics, University of California Davis; Sue Poynter, MD, MEd, Associate Professor and Pediatric Program Director, Department of Pediatrics, Cincinnati Children's Hospital Medical Center/University of Cincinnati; Shannon E. Scott-Vernaglia, MD, Assistant Professor, Department of Pediatrics, Massachusetts General Hospital and Harvard Medical School; Daniel Sklansky, MD, Assistant Professor and Associate Program Director, Department of Pediatrics, University of Wisconsin School of Medicine and Public Health; Lynn Thoreson, DO, Assistant Professor and Associate Program Director, Department of Pediatrics, The University of Texas at Austin Dell Medical School.
Editor's Note: The online version of this article contains the characteristics of study programs and the CCC member reporting survey.
Funding: This study was provided in-kind support from the Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network.
Conflict of interest: The authors declare they have no competing interests.
The views expressed in this article are those of the authors and do not necessarily reflect the official policy or position of the Department of the Navy, Department of Defense, or the US government.
