Investigating the Roles and Impact of Clinical Competency Committees in Graduate Medical Education: A Narrative Review
Background Although Clinical Competency Committees (CCCs) were implemented to facilitate the goals of competency-based medical education, implementation has been variable, and we do not know if and how these committees affected programs and assessment in graduate medical education (GME).
Objective To explore the roles CCCs fulfill in GME and their effect on trainees, faculty, and programs.
Methods We conducted a narrative review of CCC primary research with the following inclusion criteria: all articles must be research in nature, focused on GME and specifically studying CCCs, and published in English language journals from January 2013 to November 2022.
Results The main results are as follows: (1) The primary role of the CCC (decision-making on trainee progress) is mostly described in “snapshots” (ie, focusing on a single aspect of this role at a single point in time); (2) CCCs are taking on secondary roles, some of which were anticipated (eg, remediation, feedback) whereas others were “unanticipated” (eg, use of CCC data to validate trainee self-assessment, predict trainee performance in other settings such as certifying examinations, investigate gender bias in assessment); and (3) Articles briefly mentioned short-term outcomes of CCCs at the level of the trainees, faculty, and programs. However, most studies described interventions to aid CCC work and did not specifically aim at investigating short-term (eg, curriculum changes) or long-term outcomes (eg, improved patient outcomes).
Conclusions CCCs fulfill a range of roles in assessment beyond their intended purpose. A more systematic approach is needed to investigate the outcomes of CCC implementation on GME.
Introduction
The primary role of Clinical Competency Committees (CCCs) in graduate medical education (GME) is to render judgements about trainees’ performance against a set of criteria and standards (eg, competency Milestones in the United States) and to make recommendations to program directors about advancement.1 In addition to this primary role, there has been a recognition of evolving secondary roles that these committees can play in residency training.2 Although neither a mandate nor rarely made explicit, these secondary roles result from CCCs being privy to “seeing” and synthesizing all their trainees’ assessment data and may include remediating trainees, critiquing the quantity and quality of assessment data, providing faculty development, identifying curricular gaps and redundancies, and serving a quality improvement function for the assessment system.2 To date, however, there has been no synthesis of the existing literature to describe roles actually fulfilled by CCCs and their effect on GME and assessment systems in particular. Without such exploration, we risk not providing CCC members with the time allotment, faculty development, and administrative support needed, thus limiting their ability to ensure that trainees are ready for graduation and capable of providing safe and effective care. Therefore, the purpose of this narrative review is to provide an overview of the published research on the roles CCCs fulfill in GME and their effect on residency training and assessment.
Methods
We chose to conduct a narrative review because they are particularly useful for exploring topics that are complex and under-studied, and they allow the inclusion of a wide variety of studies.3 Our approach was informed by the recommendations published by Sukhera, which offer guidance on key decisions in the review process.3 According to Sukhera, narrative reviews are (1) “flexible” in which the initial scope may change through the review process,” (2) well suited for topics that are “under-researched,” (3) “require a meaningful synthesis of research evidence that may be complex or broad,” and (4) “require detailed, nuanced description and interpretation.”3 Our study met each of these criteria. First, although we started the review broadly (eg, including perspectives and policy documents), we realized that we needed to focus on research articles to gather data about how CCCs are actually being implemented and studied. Second, our focus on the roles of CCCs and their associated outcomes is indeed an “under-researched” area in GME. Third, the CCC literature to date is complex (ie, represented by varying implementation approaches).4 Fourth, we provided a “nuanced”3 analysis of the descriptions offered in the literature of CCCs as detailed below.
Search Strategy
To our knowledge, the first mandate for CCCs was put forth in January 2013, when the Next Accreditation System went into effect. Thus, we searched PubMed, Scopus, ERIC, and Google Scholar databases from January 1, 2013 to November 1, 2022. During the study, the search was updated twice because there was a delay in our process due to the COVID-19 pandemic and the emergence of pertinent CCC literature. We used “Clinical Competency Committee” as a search term to identify publications where the CCC was mentioned in the title and/or abstract. The search strategy was conducted under the guidance of a librarian at the leading author’s institution.
Publication Selection Process
The Figure summarizes the search process. One author (A.E.) performed a preliminary review of the search results. One hundred seventeen articles were initially identified and, after removal of duplicates, 111 were selected for “title and abstract review.” Two researchers (A.E. and either M.G. or S.H.) independently screened 60 abstracts for inclusion criteria and subsequently resolved discrepancies through discussion. This approach proved to be critical to the richness of our discussions centered on developing definitions for some of the key concepts involved in conceptualizing our work, such as “secondary roles” and “impacts.” A.E. subsequently reviewed all remaining abstracts. Following full-text review of selected abstracts, ultimately, 84 articles were included for data extraction.5-88
Citation: Journal of Graduate Medical Education 16, 6; 10.4300/JGME-D-24-00017.1
Inclusion/Exclusion Criteria
Inclusion criteria were the following: all articles must be research in nature, focused on GME and specifically studying CCCs (ie, the term CCC was used in the abstract and/or title), and published in English language journals from January 2013 to November 2022. Articles were excluded if they focused on other decision-making groups such as education committees, were written as perspectives, or situated in undergraduate medical education. The authors discussed these criteria at length, opting to focus the review on only GME, as the process for using CCCs in undergraduate medical education is still evolving.
Data Extraction
A data extraction form was created in Microsoft Excel. The categories (codes) for data extraction included demographic program information, description of the primary and secondary role(s) of CCCs, and the effect of these roles at various levels of the assessment system and context (eg, on the trainee, training program, specialty). Using the preliminary coding framework, 3 authors (A.E., M.G., S.H.) independently reviewed 14 randomly selected articles. Preliminary findings were discussed in the entire research team (A.E., M.G., S.H., E.H.) to resolve any discrepancies and to further refine the coding framework. Using the final coding framework, A.E. subsequently reviewed the remaining articles, while the other authors (M.G., S.H., E.H.) “spot checked” the extracted data for accuracy and completeness. Any articles co-authored by one or more members of the research team were reviewed independently by a colleague (Lisa Conforti or Raghdah Al-Bualy) with content expertise in CCCs. Table 1 presents a taxonomy of themes capturing both primary and secondary CCC roles.
| Studies (Authors, Year)/Themes | Participants/Data Sources | Type of Study/Methods | Key Finding/Contribution to CCC Role |
|---|---|---|---|
| CCC Primary Role | |||
| Scoring and rating patterns | |||
| Beeson et al, 20179 | Emergency medicine residents’ Milestones ratings | National study reporting descriptive analysis of straight line scoring (SLS) | SLS was found in 20% of emergency medicine programs with no significant difference between the fall and spring scores. |
| Schwartz et al, 202010 | Pediatric residents’ assessments | Multi-institutional study to investigate response process validity (Clinical Competency Committee [CCC] process) | Interns from programs that were competency-based and time-variable were found to have the same level of competence as those from traditional programs. CCC decisions may involve undocumented information. |
| Schumacher et al, 201911 | Pediatric residents’ CCC ratings | Multi-institution exploratory study to determine association between average Milestones ratings (AMRs) and competencies, and that between AMRs and CCC’s supervisory role recommendations | Modest positive correlation between AMRs and supervisory role and moderate to large positive correlation between Milestones ratings for individual competencies and AMRs. Results suggest that performance can be assessed by measuring only 1 or 2 aspects of performance, thus simplifying the process. |
| Schumacher et al, 202012 | Pediatric residents’ assessments by CCCs | Multi-institutional prospective cohort study of clinical performance using entrustable professional activities (EPAs) | 90% of pediatric residents in the study achieved “unsupervised practice” by graduation for only 8 of 17 EPAs. |
| Dehon et al, 201513 | Emergency medicine resident assessment data (end-of-shift evaluations [ESEs] consisting of emergency medicine Milestones) and other data were used by CCCs | Single institution retrospective analysis of assessment data | Poor correlation between CCC ratings and ESE scores. ESE (which incorporated emergency medicine Milestones) did not distinguish level of training. Faculty rated residents as not achieving a Milestone only 8% of the time. ESEs overestimated performance. |
| Schumacher et al, 201814 | CCC members of pediatric programs | Multi-institutional prospective observational cohort study to investigate the association between the review process CCC members use and their summative decisions | CCC members who provided feedback after CCC meetings were more likely to assign lower Milestones ratings and less likely to recommend supervisory responsibility. Those who reviewed more residents also recommended less supervisory responsibility. |
| Hamstra et al, 201915 | Urology, emergency medicine, and diagnostic radiology (DR) interns’ Milestones data | National study to determine whether CCCs are consistent in assigning their ratings | Average ratings decreased over time for 32 of 56 subcompetencies (no change was noted with the others). For instance, in DR 8 of 12 subcompetencies had a downward trend (with effect sizes 0.01-0.03). |
| Kelleher et al, 202116 | Internal medicine residents’ narrative data | Single institution qualitative study to compare narrative assessments for struggling and satisfactory performing residents | Thematic analysis of narrative data for the cohorts revealed 2 types of themes: explicit issues centered on resident performance (not paying attention to details, deficits in communication, inability to “see the big picture”) and implicit comments based on the assessors description such as the types of phrasing used to differentiate both levels of performance. |
| Data dilemmas | |||
| Pack et al, 20195 | Multispecialty CCCs at single institution (Canada) | Single institution, multispecialty, constructivist grounded theory study to determine how CCCs interpret assessment data | Although CCC members expected to have “high quality” assessment data, they were often in the position of trying to interpret “problematic” data instead. However, their willingness to engage in “effortful” discussions about such data led to meaningful decision-making. |
| Tam et al, 20206 | CCC members, pediatric subspecialty (Canada) | Single institution instrumental case study to investigate the use of previously undocumented data in CCC decision-making | CCC members found formal assessment data lacking and perceived undocumented data (such as personal anecdotes, hearsay, impressions, and contextual information about trainees’ situations) to be invaluable to their decision-making. |
| Schauer et al, 202251 | Internal medicine residents’ assessment data | Single institution observational study to describe the development of an expected entrustment score for a work-based assessment (ie, observable practice activity) | The expected score is a prediction of performance for a given resident which takes into account the context in which assessment occurs. Analysis of 10 years of data revealed that many residents’ observed entrustment scores “oscillated closely” to their expected scores. However, the authors provide examples of residents whose actual scores deviate from the expected score, and in some cases presents an opportunity for CCCs to intervene early. |
| Roshan et al, 202152 | General surgery resident assessment data (narrative comments) | Single institution retrospective analysis to determine the effect of rotation setting on the quality of narrative comments | The McMaster Narrative Comment Rating Scale was used to rate narrative comments generated over approximately a 5-year period. There was variability in the quality of the narrative comments both between and within rotations. Academic sites produced comments that were more corrective and actionable compared to other sites. Urban tertiary rotation comments were of lower quality across the rating categories. |
| CCC process | |||
| Donato et al, 201653 | Internal medicine CCC members | Single institution study of a program’s approach to developing a developmental CCC (ie, all residents are discussed) | Authors describe the 2-step presentation process for each resident (using a debate-like set up). Findings included: (1) the importance of faculty development (eg, led to more actionable feedback to residents), and (2) the feasibility of implementing a developmental approach. |
| Abbott et al, 202154 | General surgery residents’ end-of-rotation and CCC assessments | Single institution study to determine whether natural language processing (NLP) estimates CCC Milestones ratings | There were 594 end-of-rotation assessments and 97 CCC assessments. The mean for area under the curve was 0.84 for models with non-NLP predictors, 0.83 for those with only NLP predictors and 0.87 for those with both NLP and non-NLP predictors. Authors concluded that NLP could be used to estimate Milestones ratings. |
| Rozenshtein et al, 201555 | Radiology program directors | National observational cross-sectional study via surveys to determine approaches to teaching and assessing radiology residents | Response rates ranged from 35% to 39%. At the time of the study, the development of CCCs was expanding with program directors serving as the chair in most instances. The most used assessment was the end-of-rotation evaluation (96% of respondents). 60% performed preliminary ratings of the Milestones prior to the CCC meeting. |
| Hauer et al, 20157 | Internal medicine residency program directors | Multi-institutional qualitative study conducted via semistructured interviews to determine how CCCs are implemented | Two approaches were revealed in the process of resident assessment: (1) “developmental” (assisting each resident in their learning trajectory) and (2) “problem identification” (focusing on struggling learners). In this study, most of the programs used a problem identification approach. |
| Lloyd et al, 202056 | Program directors, associate program directors in psychiatry, psychiatry residents’ assessment data | Multi-institutional study examining the utility of various assessment tools in CCC decision-making | Investigated how CCCs reach decisions (including via the use of actual and mock CCCs) and provided guidelines for these committees. Found end-of-rotation evaluations were the most informative tool. Patient care and medical knowledge were easiest to rate, whereas systems-based practice and practice-based learning and improvement were the most challenging. |
| Sebesta et al, 201957 | Urology program directors | Multi-institutional online survey to determine perceived usefulness of the Milestones | 48% did not find the Milestones helpful in assessing their residents. They also felt that the Milestones did not help differentiate clinical performance. 56% of participants had not made any changes to their assessment system based on the Milestones and 80% had not made any curricular updates. |
| Smit et al, 201958 | Pediatric residents and faculty (the Netherlands) | National online survey and conference regarding the implementation of a structured process for rendering summative judgements about resident performance in an EPA-based system | The program was well received by the residents. 75% felt it supported their development with the appropriate level of supervision. 56% received “comprehensive” feedback based on CCC meetings. Faculty also expressed satisfaction with the program (7.3 on a scale of 1 to 10). |
| Duitsman et al, 201959 | CCC members in pediatrics (the Netherlands) |
Single institution design-based study to determine how CCCs perform their work via the development of a prototype | Design principles used to develop a CCC could be implemented in practice (eg, structured discussions, a shared mental model, multiple assessment tools). In this study, residents were not satisfied with the feedback they received from the CCC, but this was not addressed prior to the next cycle. |
| Ekpenyong et al, 202160 | CCC chairpersons (multiple specialties) | Multi-institutional cross-sectional web-based survey to identify resources CCCs need | CCC chairpersons supported the notion of institutions offering faculty development to CCC members (81.2%) and workshops for program coordinators (87%). 93.7% did not received protected time for their participation CCC work. |
| Ekpenyong et al, 20224 | CCC members, program directors (multiple specialties) | Multi-institutional ethnographic study of CCC processes | CCC practices were described with respect to 3 themes—membership, CCC roles, and processes such as decision-making and resident presentation to the committee. There was significant variation noted in the approaches used by the CCCs observed. |
| Ekpenyong et al, 201761 | Internal medicine CCC members | Single program mixed methods study to determine how CCCs use different types of assessment data | CCC faculty rated prioritized different types of assessment data: resident rotations, faculty rotation comments, their personal experience with residents (from highest to lowest). They identified 3 types of problems in the assessment process: (1) “design issues” (eg, problematic data or lack thereof), (2) “synthesis issues” (eg, those factors that affect Milestones ratings/decision-making), and (3) “impact issues” (eg, the way the Milestones ratings are used). |
| Shumway et al, 201562 | Hematology/oncology fellowship assessment data (EPAs) | Multi-institutional study of the implementation process for EPAs, CCCs | CCCs spent approximately 10 minutes in their decision-making process per fellow. For 5 continuity clinic EPAs, entrustment increased with progression in training. This was the first study of the implementation process for EPAs/CCCs for an internal medicine fellowship. Authors offer recommendations for this process. |
| Nabors et al, 201733 | Internal medicine CCC members, resident assessment data | Single institution study of CCC process outcomes with the implementation of a “rapid deliberation approach” | In the new process, faculty reviewed resident data prior to the CCC meeting and assigned Milestones ratings. These faculty subsequently presented their assigned residents to the CCC and after deliberation consensus was reached. With the new approach, deliberation time was reduced from 25 minutes to 9 minutes per resident and members were more satisfied with the rating process. |
| Chen et al, 201763 | Anesthesiology CCC members | Single institution post-hoc pilot study to evaluate a new process to review resident assessment data | A new process was created using key performance indicators to assist with rating residents on their Milestones. CCC members were involved in the redesign. This CCC assessed “one Milestone at a time, instead of one resident at a time.” This process reduced the length of the meeting time (from 8 to 3.5 hours). These CCC members preferred this approach. |
| Van Osdol et al, 201464 | General surgery residents | Single institution study to enable interns to take home call via institution of a curriculum covering multiple core competencies. An innovative Accreditation Council for Graduate Medical Education project | CCCs made the decision as to whether the interns could take home call. Medical knowledge and clinical performance were assessed via nurse mock pages, oral and written examinations. All residents were deemed ready to take home call. Authors outline resident experience including number of calls per shift, duty hours logs, willingness to call for back up from a supervisor, etc. |
| Kearney et al, 202265 | Plastic surgery CCC chairpersons | National survey to determine CCC membership and decision-making process | 69.5% response rate. 73% had 1 faculty complete the Milestones ratings and provide feedback to the residents. End-of-rotation assessments and in-service examination scores were used most of the time to determine ratings. 64% felt that Milestones did not facilitate mentorship. |
| Schumacher et al, 202066 | CCC members of a pediatric program | Single institution constructivist grounded theory study to determine how CCC members use resident-sensitive quality measures (RSQMs) | Five dimensions for the use of RSQMs by CCCs were identified: (1) orientation to RQSMs, (2) willingness to use RQSMs, (3) ability to interpret RQSMs, (4) use of RQSMs for assessment decisions, and (5) fairness regarding the use of RQSMs. Authors generated 3 profiles for CCC members based on their engagement with RQSMs: (1) eager, (2) willing, and (3) disinclined. |
| Goyal et al, 201867 | Emergency medicine residents and faculty | Single program study to determine agreement between 3 approaches to decision-making regarding Milestones ratings: (1) traditional or “gold standard” full Milestone assessment (FMA) using all available data, (2) faculty ad-hoc Milestone assessment based on memory (AMA), and (3) resident self-Milestone assessment (SMA) | AMA took less time than FMA but there was agreement between AMA and SMA on only 8 of 23 subcompetencies and between FMA and SMA for 1 subcompetency. |
| Kou et al, 202268 | Pediatric emergency medicine (PEM) fellowship program directors (US and Canada) | Multi-institutional cross-sectional web-based survey to describe if PEM programs entrust their fellows to work independently | There was variability in whether programs allowed their fellows to practice independently. 31 of 68 allowed them to work independently at some point in their fellowship. Programs had certain metrics that had to be met prior to granting this entrustment (eg, Milestones levels attained, number of clinical hours completed). |
| Schumacher et al, 201969 | Pediatric program directors’ and CCC members’ decisions about residents serving in a supervisory role | Multi-institutional study to explore how program directors “justify” their entrustment decisions about residents’ supervisory roles | Authors compared CCC and program director supervisory role ratings. 801 categorizations were made by both. At the 2 time periods studied there was agreement 77% and 91% of the time. In cases of discrepancy, program directors were more likely to assign a lower level of entrustment. Authors list participants’ justifications for moving residents either up or down on the entrustment scale. Resident performance played a larger role than their experience when moving residents up on the entrustment scale and the converse when moving them down. |
| Schumacher et al, 201970 | CCC members of pediatric programs | Multi-institutional qualitative study to explore the key factors CCCs use in making decisions about residents being assigned a supervisory role | Via thematic analysis the authors found 4 factors that play a role in CCCs’ determination of the type of supervisory role residents are assigned to: (1) trustworthiness demonstrated by the resident, (2) combination of clinical experience and performance, (3) ability to lead a team, and (4) factors beyond the resident such as availability of back-up supervisors for those residents who were deemed “borderline,” patient volume, patient acuity, etc. |
| Schumacher et al, 202071 | Pediatric CCC-generated assessment data (ie, expected narrative performance levels [ENPLs]) | Multi-institutional study to determine narrative performance levels (per EPA) that denotes readiness for graduation and compare these expectations with actual narratives describing actual performance | In 94.4% of cases, actual initial entrustment scores were the same or higher than the expected performance. For 8 EPAs, ENPLs for graduation and entrustment were the same. For 5 EPAs, the ENPLs for graduation were lower for graduation than for entrustment. Narrative performance at levels 3 and 4 were the primary choices to describe residents at graduation and entrustment. |
| Stehman et al, 201972 | Emergency medicine CCC chairpersons and program directors | National cross-sectional survey to assess performance on the professionalism Milestones | 121 programs (65.4%) completed the survey. Faculty shift evaluations (89.7%) and CCC opinions (86.8%) were the most commonly used methods of assessment. Over 95% of graduating residents achieved mastery in accountability (37% of programs) and professional values (42% of programs). |
| Implementation of the Milestones/EPAs | |||
| Schumacher et al, 202173 | Pediatric program leaders and CCC members | Multi-institutional constructivist grounded theory study using interviews to describe the experience of implementing EPAs | Themes generated regarding EPA implementation: (1) recognizing the EPAs as an acceptable framework in “alignment” with current approaches, (2) identifying curricular gaps, (3) recognizing the importance of faculty participation in the process, and (4) recognizing how CCC structure and process impacts the process. Within each of these 4 themes there were factors that facilitated or inhibited the EPA implementation process. |
| Yaghmour et al, 202274 | CCC chairs, residents, program directors (emergency medicine, internal medicine, family medicine, pediatrics) | Multi-institutional qualitative study using interviews and template analysis to explore experience in implementation of Milestones | Authors found 3 stages of Milestones implementation: (1) “resource intense” (eg, lack of engagement of residents and faculty, negative perception of the Milestones), (2) “transition” state (eg, mentions of some improvements in the process), and (3) “fine tuning” stage (eg, faculty development, continuous program improvement). |
| Balach et al, 202275 | Orthopedic surgery program leaders (eg, program director or CCC chairperson) and residents | National qualitative study of orthopedic surgery programs to assess implementation of their Milestones | 57 (23%) programs participated in phone interviews. 2 themes emerged: implementation and impact of the Milestones via which benefits and challenges were identified. |
| Nabors et al, 201376 | Internal medicine faculty, program leaders, residents, CCC members | Single institution study describing the implementation of a Milestones-based evaluation process. | Faculty and residents agreed (somewhat or strongly) that the Milestones promoted a shared mental model of expectations at various points in training and enhanced feedback efforts. Faculty and residents also thought that Milestones promoted “fairness and uniformity.” 8 of 12 faculty indicated they would recommend a Milestones-based evaluation system to other institutions. |
| Maranich et al, 202277 | CCC members from transitional year programs from 7 specialties (neurology, emergency medicine, ophthalmology, internal medicine subspecialty, pediatrics, internal medicine, and pathology) | Multi-institutional study using a constructivist lens performed a thematic analysis of interview data to explore participants’ understanding and application of the Milestones | Found 5 themes: (1) lack of formal training regarding their role as CCC members, (2) perspective of Milestones as a helpful tool for assessing residents, but not referencing the other uses of Milestones (eg, for program evaluation or quality improvement), (3) the existence of a process to convert assessment data into a Milestone rating, (4) use of both formal and informal narrative comments in reaching judgements, and (5) assigning residents an “average” rating in the absence of data (eg, for the systems-based practice Milestones). |
| Hanson et al, 202378 | Advanced gastroenterology/hepatology fellows and faculty | Multi-institution study to describe the EPA implementation process | Entrustment increased over the course of a year. Overall, viewed as positive by fellows and faculty. |
| Lewis et al, 202179 | Program directors, associate program directors, CCC chairpersons (19 specialties and subspecialties in a children’s hospital) | Single institution collective case study to describe the Milestones implementation process | Thematic analysis of meeting notes revealed varied assessment methods and tools to address the 6 core competencies. Three categories of assessment methods emerged: (1) direct (eg, direct observations, in-training examinations), (2) indirect (eg, rotation evaluations), and (3) multisource (eg, 360-degree assessments). Among other issues, the primary challenge identified by participants was the need for alignment between the assessment methods/tools and the Milestones. |
| Yamazaki et al, 202280 | Obstetrics and gynecology (OB/GYN) residents’ Milestones ratings | National study to investigate the internal structure validity of the OB/GYN Milestones | Milestones ratings increased as residency training progressed. Apart from the 6 general core competency domains, CCCs further identified 3 factors of the patient care competency. |
| Identification of struggling residents | |||
| Schumacher et al, 201881 | Pediatrics program directors and CCC members | Multi-institutional, qualitative study using survey methodology and thematic analysis to examine how struggling residents are identified | Found 2 groups of themes by which a resident can be identified for concern: (1) pathways by which a resident reaches a threshold (eg, accumulation of isolated data points, narrative comments from rotation evaluations, developmental), and (2) how CCCs and program directors make decisions about performance concerns (eg, CCC used norm and/or criterion referenced approaches and made judgements as to the quality of the assessment data reviewed including the type of rater [eg, “hawk or dove”]) |
| Conforti et al, 201882 | Program directors, residents, and CCC chairs of neurosurgery programs | Multi-institutional grounded theory study to examine the process of using the Milestones to assess neurosurgery residents | Exploration of the Milestones implementation process led to 2 types of responses: (1) “outcomes” (eg, identifying struggling trainees earlier, curriculum revision, continuous quality improvement), and (2) “facilitators” (eg, clarified the core competencies) and “barriers” (eg, not being accepting of the Milestones as a framework). |
| Dashboards/displays/apps | |||
| Yen and Lewis, 202224 | Radiology residents, faculty, and program coordinator | Single institution survey analysis of the implementation of a web-based resident management system | The online platform was well received by residents (82% agreed or strongly agreed that it was a useful addition, 53% agreed that it was easy to navigate). Faculty had a similar positive response. Program coordinator preparation time (for CCC meetings) was reduced by ∼90% and that of faculty by at least 50%. |
| Stahl et al, 202025 | Residents and faculty, general surgery, emergency medicine, hospital medicine | Single institution, multi-specialty study of EPA implementation via mobile app. Feedback was obtained from participants about their experience using the app. | Resident self-assessments and faculty assessments were compared (66% were matched, ie, resident and faculty data on the same assessment). Assessments increased after resident and faculty development sessions. Residents and faculty found the app easy to use but identified barriers and offered suggestions for improvement. |
| Friedman et al, 201626 | CCC members, internal medicine | Single institution study of the development and implementation of a dashboard to compile both quantitative and qualitative assessment data | Dashboards were created in Excel for each resident (5 minutes per resident). CCC members took 30 minutes per resident assessing their data. Dashboards facilitated a structured and standardized process to CCC decision-making. |
| Almufarrej et al, 202227 | Plastic surgery faculty, residents’ data, CCC members | Single institution prospective pilot study to determine feasibility of the use of a smartphone-based evaluation system | Pre- and post-surveys were conducted. CCC members were also interviewed to determine the effectiveness of feedback obtained via the new system. CCC scored the new app highly (4.65 of 5, compared to a rating of 2.82 of 5 with the previous assessment tool). |
| Dougherty et al, 202228 | CCC members, general surgery | Single institution implementation study of a “resident scorecard” to facilitate CCC decision-making | Scorecard is a single page document containing both quantitative and qualitative assessment data. Due to the implementation of the score cards, total time of the CCC meeting was reduced from 126 to 106 minutes. Time spent per page increased from 1 to 5 minutes (although total pages per resident decreased from 9 to 1). |
| Johna and Woodward, 201529 | General surgery resident assessment data | Single institution study using a newly developed dashboard to better align existing curricula goals and objectives with the Milestones to enhance CCC deliberations | Developed a dashboard using Microsoft Excel linking data from assessments to each of the Milestones. This reduced the time spent per resident from 45-60 minutes to 5-10 minutes. The dashboard also included the use of radar plots. |
| Input from allied health professionals | |||
| Regan et al, 201830 | Emergency medicine residents’ CCC data, end-of-shift (EOS) and end-of-rotation (EOR) data | Single institution study to determine if 2 new assessment methods, EOS and EOR, facilitate CCC decision-making. | EOS and EOR were collected from doctors (d) and nurses (n). EORn and EORd scores were correlated with CCC scores. The strongest correlation was between CCC scores and EORd (r 0.71 to 0.84). EOSn and EORn were not as strongly correlated with CCC scores (r 0.49 to 0.62) and (r 0.4 to 0.73) respectively. Authors also found that levels of proficiency assigned by the CCC were highly correlated with the levels that were assigned in the previous 6 months (suggesting a possible bias by these previous decisions). |
| Bedy et al, 201931 | Emergency medicine resident data, CCC members | Single institution comparison of pharmacists’ and CCC’s ratings. | Pharmacists assessed residents on subcompetency patient care (PC)-5 ie, “pharmacotherapy.” Authors found 100% agreement between the pharmacists’ ratings and those of the CCC. |
| New assessment tools/examinations/curricula | |||
| Mamtani et al, 201518 | Emergency medicine residents and CCC members | Single institution study using survey methodology to elicit perceptions from residents and CCC members regarding introduction of a novel curriculum (“EM debates”) for systems-based practice and practice-based performance improvement content | 71% of participating residents the curriculum improved their performance. 100% of those that led a debate felt they could use an evidence-based approach to a particular topic. CCC members felt that as a result of the new curriculum they could better assess performance on the associated subcompetencies. |
| Cooney et al, 201619 | Plastic surgery resident assessment data | Single institution study of the implementation process of a web-based tool (comprehensive observations of resident evolution “CORE”) | CORE distinguished novice from experienced residents. 38% of residents reported more timely feedback about their operative performance. |
| Mount et al, 20148 | Internal medicine residents and faculty | Single institution study of the development of an oral examination for use by the CCC | Validity evidence was obtained. Faculty spent less than 0.1 annual full-time equivalent on this assessment. 54 (86%) residents passed it on their first attempt. The posttest survey of residents and faculty revealed that the examination was perceived as fair. |
| Bryant, 202122 | Pathology residents, faculty, pathologists’ assistants, CCC members at a single program | Single institution study using survey methodology to determine feasibility of an EPA for frozen section training | 100% of faculty and CCC members agreed that the EPA was easy to understand. 80% of CCC members thought it would help them determine residents’ readiness for taking call. 75% of residents found it easy to use and helpful for understanding the frozen section process; 50% thought it provided useful feedback. |
| Rassbach and Blankenburg, 201823 | Pediatric residents and faculty | Single institution study to determine if a coaching program could improve the quality of feedback residents receive and faculty members’ confidence in providing feedback | 99% of residents agreed or strongly agreed that their coaches were well trained in providing feedback. Coaches reported more confidence in providing feedback and teaching other clinical skills compared to non-coaches. Non-coaches also may have benefited from this program as they reported providing equal or more feedback. |
| Secondary CCC roles | |||
| Anticipated | |||
| Feedback to trainees and programs | |||
| Pack et al, 202032 | CCC members, multispecialty (Canada) | Multispecialty, single institution constructivist grounded theory study to identify CCC roles | CCC members were observed and interviewed. CCCs went beyond their primary role to identify and offer solutions to problems at both the level of the residents and the program. A portion of the work involved occurred outside of formal meeting time. |
| Remediation | |||
| Warburton et al, 201734 | Internal medicine residents’ assessment data | Single institution study of a remediation process (ie, Early Intervention Remediation Committee) | CCC referred learners to the remediation committee. All identified issues (eg, medical knowledge, professionalism, clinical reasoning, communication skills, and organization) were resolved. Required investment of ∼45 hours per learner. The process was well received by learners and faculty. |
| Blau et al, 202135 | Plastic surgery residents | Single institution case study on implementing their CCC and its various roles | CCC provided residents with feedback and created individualized learning plans—including making reading recommendations and monitoring residents’ progress. |
| Ketteler et al, 201483 | General surgery CCC members | Single institution study of Milestones and CCC implementation via the inclusion of “competency champions” with expertise in each of the core competencies | After one year, residents and faculty were more aware of the importance of the core competencies and the remediation. There was also an improved ability by faculty to evaluate the residents and provide feedback. The competency champions provided a monthly report to the CCC on the progress of residents requiring remediation. These competency champions also made recommendations as to whether such residents completed the remediation process. |
| Address curricular gaps | |||
| Klutts et al, 201584 | Pathology resident assessment data | Single institution study describing the process of developing their CCC | Authors describe in detail their CCC process including unique features such as faculty assessing all residents on only those Milestones which the faculty member has expertise in. They also discuss the impact of the CCC on their program eg, residents rated themselves lower on the systems-based practice Milestone that includes laboratory management, thus information on laboratory management was added to the curriculum and pathology residents were asked to attend hospital safety meetings (to address safety and quality issues with respect to laboratory management). |
| Unanticipated | |||
| Validation/calibration of self-assessments | |||
| Lyle et al, 201636 | General surgery residents | Single institution study to determine if there is a correlation between residents’ self-assessments and CCC Milestones ratings | Major differences were defined as >0.5 on a 4-point scale. 7% had a mean overall difference of >0.5, 6 had a difference of ≤0.5. Self-assessment scores were lower than those of the CCC (median difference of -0.06 (-0.25 to 0.16, P=.041). Negative self-assessments were more common for medical knowledge. Subgroup analysis by sex, postgraduate level, and time of academic year provide notable areas for further investigation. |
| Watson et al, 201737 | General surgery residents | Multi-institution prospective comparison of CCC ratings to residents’ self-assessments | Major differences were defined as >0.5 on a 4-point scale. 13% self-assessed >0.5 higher than the CCC and 13% self-assessed lower than the CCC. Difference for various subgroups (eg, level of training, sex, type of program) were reported. Overall found good concordance between resident self-assessment and CCC scores. |
| Tichter et al, 201638 | Emergency medicine residents, CCC members | Single institution retrospective study to determine agreement between resident self-assessments and CCC ratings | There was moderate agreement between residents’ self-assessments and their CCC ratings. No change between the mid and end-of-year reporting cycles. |
| Ross et al, 201639 | Anesthesiology residents’ self-assessments and CCC ratings | Single institution study to determine if there is a correlation between residents’ self-assessments and CCC ratings | Found high correlation between resident self-assessment and CCC scores in anesthesiology (with correlation coefficients for each Milestone score versus postgraduate year [PGY] level from 0.8-0.95). |
| Srikumaran et al, 201940 | Ophthalmology residents’ self-assessments and CCC ratings | Single institution comparison of CCC and resident self-assessment scores | No significant correlation between resident’s self-assessment and CCC ratings. For residents’ first assessment, CCC and self-assessment score were only strongly correlated for 4 of 24 Milestones—with less difference for medical knowledge and systems-based practice. Differences between self-assessment and CCC scores did not lower significantly with further assessments. |
| Meier et al, 201641 | General surgery residents’ self-assessments and CCC ratings | Single institution comparison of CCC and resident self-assessment scores | Compared residents’ self-assessment with those of both their faculty advisors and CCC. High correlation noted between the 3 groups of raters. Makes the argument for faculty and residents having more of a stake or ownership of the assessment process. |
| Chow et al, 201942 | Plastic surgery residents’ self-assessment and CCC ratings | Single institution comparison of CCC and resident self-assessment scores | Found concordance between residents’ self- assessments and CCC evaluations. The exception was second-year residents who self-assessed higher on medical knowledge and patient care. The authors concluded that residents are capable of accurate self-assessment. They also used areas of discordance to make curricular improvements. |
| Exploration of gender bias in assessment | |||
| Kwasny et al, 202143 | General surgery residents’ Milestones ratings | Single institution retrospective study to determine if there are gender differences in Milestones ratings (for both CCC ratings and self-assessments) | Examined found no significant difference in CCC scores of male and female residents and scores improved as residents advanced in training. |
| Santen et al, 202044 | Emergency medicine residents’ Milestones ratings | National study to compare male and female Milestones ratings in emergency medicine | Did not find major differences in EM male versus female residents’ Milestones scores across most competencies (small differences noted in patient care Milestones). Raises the issue of CCC possibly neutralizing biases in the assessments they receive. |
| Zuckerbraun et al, 202045 | Pediatric emergency medicine fellows’ Milestones ratings | National retrospective cohort study to determine if there are gender differences in Milestones ratings | There was no difference for male and female fellows in average Milestones scores across the 6 core competencies. By subcompetency females scored higher on PC3 (providing transitions in care) and ICS2 (understanding emotion and human response). This was significant, but not meaningful. |
| Hauer et al, 202146 | Internal medicine residents’ CCC ratings | National retrospective study to determine if there are gender differences in end-of-year Milestones ratings for patient care (PC) and medical knowledge (MK) competencies | For the MK Milestones there was no significant difference between scores for male and female residents at PGY-1. MK ratings were higher for PGY-2 and PGY-3 men compared to women. For PC, men and women had similar scores at all PGY levels. |
| Athy et al, 202147 | Pathology residents’ assessments | Single program retrospective study comparing CCC and resident self-assessment | No difference between male and female self-assessment scoring. PGY-1s overestimate their skills and PGY-4’s ratings were lower than the CCCs’. Self-assessment scores were lowest on systems-based practice and highest on professionalism Milestones. Increase in in-service examination scores correlated with increases in self- and CCC-scoring. |
| Predicting performance in other settings | |||
| Turner et al, 201648 | Anesthesiology residents’ assessments | Multi-institutional retrospective study to determine the effect of performance deficiencies on graduation and board certification rates | CCC scores and recommendations were used to predict residents who might have problems on their certifying examination and graduation. The graduation rate among those who had a performance deficiency was 93% (vs 99% in those without a deficiency). However, this was 55% if there was a deficiency in an “essential attribute” (eg, being honest, respectful, ethical, etc) 89% of those with a deficiency attained board certification compared to 99% of those without a deficiency. |
| Kimbrough et al, 201849 | General surgery residents’ Milestones ratings and American Board of Surgery In-Training (ABSITE) scores | Single institution study to determine if mid-year CCC ratings could predict ABSITE scores | “Minimal to small” positive correlations were observed between Milestones and ABSITE percentile scores (r=0.09-0.25) with a stronger correlation between Milestones scores and percent correct scores (r=0.65-0.76). The mid-year MK1 Milestone scores contributed significantly to this effect. |
| Rebel et al, 201921 | Anesthesiology residents | Single institution comparison of CCC Milestones ratings and those of an objective structured clinical examination (OSCE) |
CCC Milestones ratings were higher than those from the OSCEs—this effect was highest with junior residents. However, ratings improved over time for both CCC and OSCE ratings. There was also an expectation bias in the CCC ratings (primarily involving senior residents). |
| Cullen et al, 202220 | Multispecialty (16 residency, 1 fellowship) | Multi-institution to determine if situation judgement test (SJT) scores can identify professionalism issues and if they are correlated with CCC professionalism ratings | SJT scores correlated positively with CCC professionalism Milestones ratings (r=0.21 and 0.14 for mid-year and year-end, respectively). |
| Validity evidence for Milestones/EPAs | |||
| Bienstock et al, 202185 | OB/GYN residents (1184 residents from 240 programs) | National study to evaluate the correlation between CCC ratings and board scores | The strongest association was seen for all 7 subcompetencies of medical knowledge. Correlations were stronger with each subsequent Milestones rating period. |
| Mink et al, 202086 | Pediatric fellows’ assessment data | Multi-institution study to determine agreement between fellowship program directors and CCC entrustment decisions | For each of 6 EPAs, there was strong correlation between fellowship program director and CCC entrustments (P<.001)—although not as strong when the fellowship program director was not a CCC member. |
| Mink et al, 201887 | Over 200 pediatric fellowship programs | Multi-institutional study using a modified Delphi approach was used to develop scales for common EPAs and demonstrate validity evidence | Supervision scales were used by CCCs and fellowship program directors to assess fellows. Progressive entrustment was demonstrated for each EPA. |
| Evaluation of curricular elements | |||
| Frey-Vogel et al, 201650 | Pediatric interns | Single institution study to determine the correlation between summative simulation cases and CCC scores | There was a statistically significant relationship between summative and CCC scores. An increase of 1 point in summative scores was associated with a 0.2 increase in CCC ratings. Performance on formative and summative cases was not correlated (no P values were significant). |
| Mikhaeil-Demo et al, 202188 | Graduating neurology residents | Multi-institutional cohort study to assess status epilepticus management and correlation Milestones ratings | For simulation-based assessments regarding status epilepticus management (SE), simulation-based scores were compared with resident’s clinical experience (including CCC scores). Graduating residents scored poorly on simulation-based assessments (average SE checklist score was 60% correct); however, their end-of-rotation scores were higher (at level 4.3 of 5 for epilepsy Milestone and 4.4 of 5 for management and treatment). Concluded that end-of-rotation scores should not be used alone in judging Milestones performance for such skills. |
Note: All of the articles in the table were included in the study. There may be articles that overlap multiple categories; however, in order to list each article once, the authors associated each article with a particular theme.
We were aware of the posited secondary roles mentioned in the CCC guidebook,2 but we did not know if this would capture the breadth of what was published. Thus, in our taxonomy, we differentiated “anticipated” secondary roles (ie, those that had been mentioned as theoretically possible in the CCC guidebook2) versus those that we had not already seen described (“unanticipated”). We used the competency-based medical education (CBME) logic model developed by Van Melle et al to define effects of CCCs on GME training.89 A logic model is a visual representation of how components of a process interact to effect short- and long-term outcomes.89 Thus, changes in the assessment system (eg, new rotations, assessment tools) or actions and attitudes of stakeholders resulting from CCCs fulfilling primary and secondary roles were identified as effects or outcomes. We furthermore characterized outcomes as “short-term” (eg, changes that can be measured in 1 or 2 CCC cycles, for example, residents using CCC feedback to create individualized learning plans) versus “long-term” (eg, those changes that take a much longer time to identify and measure such as linking trainee learning to improved patient outcomes or determining whether CCC ratings influence revisions to the Milestones).
Reflexivity
We had ongoing discussions to determine how our a priori definitions and assumptions of CCC roles and experiential knowledge might influence our interpretation of the data. All authors have published under the broad heading of assessment. A.E. works for the Accreditation Council for Graduate Medical Education (ACGME) and has contributed to the CCC literature. E.H. worked for the ACGME at the time of the study and has contributed to the Milestones and CCC literature. M.G. has been involved in design of assessments in which decision-making relies on CCCs. She also has experiential knowledge of how CCCs work and how they can affect assessment quality. S.H. has been involved in the design and execution of programs of assessment and curricula of undergraduate medical education in which decision-making is done by a committee, and she has experiential knowledge on quality assurance of assessment programs.
Results
Table 2 presents an overview of the demographic characteristics of the studies included in our review. Evaluation of the quality of the studies included in our review identified most studies to be original research published in peer-reviewed journals, focusing on specific elements in the process of CCC implementation and decision-making. The majority were single-institution, single-specialty studies conducted in the United States involving CCC review of trainees at the residency level. See Table 1 for a brief description of each of the studies included in the review.
CCC Primary Role and Potential Associated Effect
The primary role of the CCC is to assess trainees and make recommendations to program directors regarding developmental progress.1 We found that most articles did not describe the primary role in its entirety (ie, from review of the assessment data to making a judgement and communicating it to programs and trainees). Rather, the focus was on various aspects such as trying to make sense of difficult data,5 the use of undocumented assessment data,6 role conceptualization,7 or implementation of new assessment tools.8 See data for themes and examples. Some articles focused on the process of Milestones ratings and patterns in performance scoring.9-16 An ethnography of CCCs furthermore revealed marked variability in the approaches CCCs use to conduct their work, such as the strategies used in decision-making and types of assessment data used.4 We also found little inquiry as to how decisions about trainee performance are actually made. The CCC process (before, during, and after meetings) as described by Al-Bualy and colleagues outlines multiple steps during CCC work.17 Our review reveals that still very little is known about how these steps are actually performed.
Although we did not find any studies that explicitly aimed to investigate the effect of CCC implementation on programs and assessment systems, there were multiple examples of “CCC-inspired changes” aimed to facilitate CCC work such as instituting new (1) curricula,18 (2) assessments,19-23 (3) ways of organizing the assessment data,24-29 and (4) assessors.30,31 Beyond these CCC-inspired changes, there were also some mentions of “short-term” outcomes involving the trainees, program, or CCC itself. For instance, at the level of the trainees, one program implemented a web-based direct observation assessment tool with 38% of residents subsequently reporting more immediate feedback on their operative skills.19 At the level of the program, Mamtani et al describe development and implementation of a curriculum and instructional strategies to support assessment of Patient Safety and Quality Improvement Milestones.18 Similarly, Pack and colleagues describe CCCs providing training programs with input for development of new learning experiences, based on what CCCs count as evidence for their decision-making and to fill gaps within the assessment system.32
Lastly, there were a number of instances of short-term outcomes on the CCC itself. In some cases, authors were able to report a resultant change to their process. For instance, the score cards used in one general surgery program to consolidate all relevant assessment data to a single page card reduced the length of the CCC meeting from 126 minutes to 106 minutes, yet the time available to actually discuss each resident increased from 1 minute prior to using the score cards to 5 minutes after implementation.28 Nabors and colleagues described reduced deliberation time during CCC meetings following a change in the assessment process (ie, CCC members assigning their ratings prior to the CCC meeting).33 Other studies reported the effect of including allied health professionals as assessors on the robustness of CCC decision-making. For example, Bedy and colleagues surveyed emergency medicine CCC members and found that they perceived pharmacists’ assessments as useful to their judgement of resident performance.31 In another study, nurses’ end-of-rotation assessments of emergency medicine residents did not correlate with final CCC proficiency levels; however, their descriptive comments were thought to be “invaluable” for identifying areas for improvement.30 Pack and colleagues found that through the CCCs struggle with problematic evidence, members unpacked their assumptions about the source of the data used and reflected on how they perceive certain types of data as useful evidence.5 All of these examples suggest the potential effect of CCCs at multiple levels in residency training and assessment, not only on trainees, curricula, and assessment programs but also on the CCC members themselves, for example through development of shared mental models around decision-making processes, including usefulness of assessment data.
CCC Secondary Roles and Potential Associated Effect
We found only one study that was specifically designed to explore CCC secondary roles.32 In this study by Pack and colleagues, CCCs deliberately contributed to ongoing evaluation of the curriculum and assessment in residency training and provided meaningful feedback to program directors about program limitations, thereby documenting an impact on CCC members, other faculty, and the program.32 As the vast majority of articles included in our review were not designed to explore CCC secondary roles, we extrapolated the secondary role of the CCC based on the narrative provided by the authors. For instance, some CCCs went beyond identifying struggling residents to assume other responsibilities in the remediation process. Warburton and colleagues describe their CCC not only identifying struggling learners but also referring them to the Early Intervention Remediation Committee.34 In another setting, CCC members helped create individualized learning plans and monitored residents’ progress.35 In these descriptions of secondary roles, the associated outcomes were not reported.
In addition to these “anticipated” secondary roles, we also found a myriad of secondary roles which had not been ascribed to CCCs previously and denoted these as “unanticipated” secondary roles (see Table 1 for themes and examples). The very presence of a CCC, and the data generated by these committees, allowed for “CCC data facilitated research” (ie, use of CCC data to investigate a variety of assessment-related topics). Several studies, for example, investigated the correlation between resident self-assessment and CCC scores.36-42 In various specialties and settings, CCC data was used to check for gender-based differences in Milestones ratings43-47 (ie, for evidence of gender bias in assessment). In those studies that did not find any significant gender-based differences,43-46 some authors hypothesized that CCCs may serve to neutralize biases in the assessments they receive.46 CCC data were also used to investigate correlations between Milestone ratings and performance in other settings, such as certifying or in-training examinations.48,49 Lastly, CCC data was used to support the evaluation of curricular activities or changes. For instance, Frey-Vogel and colleagues compared pediatric interns’ performance scores in a simulated setting with CCC performance ratings.50 In all of the examples mentioned above, CCC data was seen as an instrumental resource for investigating a variety of assessment-related questions that arise in GME.
Discussion
To our knowledge this is the first narrative review of the CCC literature focusing on CCC roles and the effect of these committees on residency training and assessment. Our work revealed 3 key findings: (1) most CCC studies address components of the primary role but do not explore the entire process; (2) CCCs fulfil secondary roles that we did not anticipate; and (3) despite the myriad of short-term “CCC-inspired changes,” there is a lack of documented long-term outcomes.
By revealing the lack of a comprehensive approach to studying their primary and secondary roles, our review illustrates the complex nature of the work of CCCs. Not only are we still trying to understand how CCCs accomplish their intended role, but our findings also suggest their actual role includes a broad range of secondary roles, some of which are unanticipated. Thus, our review illuminates the breadth of roles CCC play in GME assessment systems. With a clearer understanding of the roles these committees actually play, programs are in a better position to advocate for resources to support CCC efforts. A more systematic approach to investigating CCC processes may not only assist CCCs in developing their expertise, but also provide meaningful input into how assessment systems can be improved to ensure fair and transparent assessment of learning as well as enhance assessment for learning (feedback and learning opportunities for trainees), contributing to achievement of overarching goals in CBME.
CCC performance ratings have been used as a source of validity evidence in the context of self-assessments, exploration of bias in assessment decisions, and prediction of performance in other settings—an unanticipated role. This seems to suggest that CCC ratings are considered to be a “gold standard.” Although findings from these studies may help critically analyze and improve assessment practices in specific settings, more work is needed to justify this approach. Further research into both anticipated and unanticipated secondary CCC roles can help training programs leverage the data they are already collecting to evaluate their practices and make improvements to their education and assessment systems. Therefore, we want to emphasize that even though secondary CCC roles (both anticipated and unanticipated) were not the original intent, they should not be minimized. More work is needed to determine the full scope of these roles and their effects on assessment systems, and we call for a collaborative national research agenda centered on studying specific CCC roles and their associated short- and long-term outcomes.
We also need to determine how implementation of CCCs contributes to achievement of intended outcomes in CBME (ie, graduating trainees who can provide safe and effective care). Although we found multiple examples of “CCC-inspired changes” or “outputs” such as new assessment tools, learning experiences, and data organization platforms, we did not find any studies that focused on investigation of long-term outcomes (eg, a change in feedback culture). To better understand if and how implementation of CCCs contributes to CBME goals, we therefore recommend that future studies consider using the logic model to study CCC impact, both short- and long-term, and intended as well as unintended outcomes.
There are a number of limitations to this work. Our review is limited to English-language studies only. Since studying the roles and associated outcomes of CCCs was not the explicit aim of most of the articles we reviewed, it is possible that we made inferences without fully understanding the contexts described. Although our taxonomy can serve as a first step for conceptualizing CCC processes and effects, it will need to be revisited and revised as future work is performed. Most of the studies were conducted in the United States and it is possible that our findings and conclusions may not be generalizable to other contexts. Thus, future studies should investigate CCC roles and associated outcomes in other contexts.
Conclusions
Although this narrative review identified broader CCC roles than anticipated, there were significant gaps in the literature regarding descriptions of these roles and their associated effects. The lack of articles specifically focused on investigating the outcomes of implementing CCCs in GME is a key finding and a launching point for future work.
Flowchart of the Literature Search and Study Selection Process
Abbreviation: CCC, Clinical Competency Committee.
Author Notes
