Overview and Research Question

Scoping reviews are used to “examine the extent, range, and nature of research activity; … to summarize and disseminate research findings; and to identify research gaps in the existing literature.”^9,10 Given our aim to explore a broad topic, a scoping review was best suited to provide a sense of the range of educational research activity. As we wanted to capture work published contemporaneously to the 2009 revision, we reviewed studies published from 2008 onward. Therefore, our eligibility criteria were studies published between 2008 and 2017 in the setting of IM residency training programs in the United States and pertaining to ambulatory care training. We adhered to the methodologic framework laid out by Arksey and O'Malley.¹⁰

Data Sources and Search Strategy

A search strategy utilizing both index terms and keywords was designed by a medical librarian (R.P.) with expertise in search strategy development. A search query was designed to yield all results containing terms related to IM, residency, and ambulatory care. A second search query was designed to yield all results containing terms related to IM, graduate medical education, ambulatory care, and teaching/training/curriculum. PubMed (National Center for Biotechnology Information, Bethesda, MD), Education Resources Information Center (ERIC, US Department of Education, Washington, DC), and Scopus (Elsevier, Atlanta, GA) databases were searched. Results for both searches were combined, and duplicate results were removed, with the final searches run in May 2017. MedEdPortal (Association of American Medical Colleges, Washington, DC) was searched manually by the lead author (A.C.) using a variety of keyword combinations.

Screening

Based on a pilot review of the literature, articles were assigned the following themes: clinic redesign, curriculum development, evaluating resident practice/performance, teaching methods, program evaluation, and faculty development. Titles and abstracts were reviewed by one author (A.C.) to exclude irrelevant results and assign results to the appropriate themes. After initial screening, all articles in a theme were read by a research team member who completed full-text review to confirm relevance, perform data extraction, and complete a quality assessment. While some studies could potentially have been included under multiple themes, for this review, all studies were assigned to a single theme. In the case of a disagreement, the entire research group would discuss the assignment and come to a consensus. Seventeen articles were transitioned from one theme to another after team member discussions.

Quality Appraisal

Quality assessment was conducted using the Medical Education Research Study Quality Instrument (MERSQI) for all studies when complete data were available.¹¹ To ensure consistency, 5 articles were successively reviewed and scored by all authors. Authors then discussed any inconsistencies in ratings and finalized characterizations of the tool. At the conclusion of the pilot process, each article received a score by all reviewers within 1 point. Using the scores of the final 3 articles from the pilot process, the calculated Krippendorff alpha was 0.884, indicating high interrater reliability.¹²

Curriculum Development

Thirty-four articles (19%) and 24 MedEdPortal submissions (13%) whose primary focus was curricular innovation were identified. Of those 58 articles and submissions, the most common topics addressed were geriatrics and palliative care (n = 10, 17%), chronic pain and substance abuse (n = 8, 14%), quality improvement (n = 6, 10%), and transitions of care (n = 6, 10%), among others.

A number of studies evaluated the effectiveness of geriatrics, palliative care, and advance care planning curricula. Two studies showed improvements in knowledge and self-assessed skills after didactic content, and they both combined traditional didactics with online simulation or role playing.^13,14 Multiple studies explored the benefits of mandated rotations in concert with didactic content,^15–17 and all found high learner satisfaction and improved knowledge on posttesting, but no changes in attitudes or behaviors.

Numerous curricular interventions focusing on chronic pain and substance abuse were published in the last several years. Low levels of trainee comfort with substance abuse management were seen across needs assessments.¹⁸ Most successful interventions combined didactic exposure with either simulation or clinic-based quality improvement interventions. One intervention explored resident training in the screening, brief intervention, and referral to treatment (SBIRT) approach at 4 different sites combined with clinic-wide SBIRT implementation, finding improved identification and documentation.¹⁹ Another study explored the impact of 10 hours of substance abuse instruction combined with preceptor SBIRT training, showing improved knowledge and skills with 6-month retention.²⁰

O'Sullivan and colleagues²¹ examined the impact of resident education on smoking cessation and rotation through a dedicated smoking cessation clinic, demonstrating improvements in identifying active smokers, an increase in counseling and prescription of therapies, and improved resident confidence. Two studies looked at the impact of introductory “boot camps” for new interns, with a multiday intervention that included didactics and clinic orientation/tours showing improvements in knowledge and clinical confidence.^22,23

The relatively limited number of published curricular interventions may be due to the large number of residency programs using extramural curricula (developed at outside institutions for broader use)²⁴; however, few groups published examinations of the implementation or impact of these modules in their individual residency programs. Additionally, very few curricula were published in MedEdPortal, limiting the ability to share curricular resources. Upon review of MERSQI subscores, validity evidence for studies was infrequently cited. Most studies used pretest and posttest methods, with no studies demonstrating patient-level outcomes and only 8% (n = 15) showing changes in behavior; knowledge-based outcomes were most frequent, often with demonstration of only short-term retention.

Clinic Redesign

Forty-nine articles (27%) whose primary focus was on developing, implementing, and/or evaluating efforts to redesign ambulatory clinic practice were identified. Of those 49 articles, the most common topics addressed were improving quality of care (n = 14, 29%); X + Y scheduling (models that alternate blocks of inpatient rotations with dedicated ambulatory blocks; n = 12 [24%]); end-of-year handoffs (n = 5, 10%); and transitions of care from inpatient to outpatient (n = 6, 12%), among others.

The highest-rated articles on improving quality of care through clinic redesign described interventions such as medication reconciliation training,²⁵ depression screening protocols,²⁶ creation of an interdisciplinary chronic illness management practice,²⁷ and development of an electronic health record (EHR) decision tool to improve geriatric screening.²⁸ Many of these studies were able to show changes in behavior using EHR data.

Many programs have moved to X + Y scheduling with the aim of improving resident-patient continuity and minimizing inpatient-outpatient conflicts. Three articles focused on continuity with the introduction of an X + Y schedule with mixed results.^29–31 Several studies focused on resident satisfaction with an overall trend toward improvement in reducing inpatient-outpatient conflict^30,32–35 and resident satisfaction with ambulatory education,^30,33,36 but no studies found an increase in the desire to pursue primary care careers.^29,32,36 Only 2 studies evaluated improvement in quality of care in X + Y schedules with mixed results.^31,37 Shalaby and colleagues³⁸ provided a practical review of crafting X + Y schedules. No included studies compared different X + Y permutations, making it unclear which option would be optimal. Further investigation is needed regarding optimal X + Y structures and their long-term effects on trainees and patients.

A survey³⁹ found that only 34% of programs had a year-end handoff process to transition patients from outgoing postgraduate year 3 (PGY-3) residents to incoming PGY-1 residents. Of the 6 articles focusing on ambulatory handoffs, only 3 implemented standardized documentation processes,^40–42 and no changes in patient-level outcomes were noted. Doctoroff et al⁴³ found that residents' patients have less-robust outpatient follow-up after hospitalization as compared with patients with faculty primary care physicians (PCPs). Two of the 49 studies described hospital follow-up visit programs in residency clinics.^44,45 Two other studies reported on resident-driven quality improvement projects that addressed communication during the inpatient-outpatient transition.^46,47 None of these studies showed significant reductions in 30-day readmission rates; however, improvements in resident skills, knowledge, and confidence were noted, and the interventions were well-received.

Although opiate use for chronic pain and substance abuse have become major issues in the ambulatory setting, our search found only 3 of the 49 articles (6%) that described how residency clinics are addressing this issue with a clinic redesign-based approach. Holt and colleagues⁴⁸ described an addiction clinic embedded in the resident practice, staffed by residents, a chief resident, a clinical psychologist, and board-certified addiction medicine specialists, which was associated with high patient and trainee satisfaction.

Evaluating Resident Practice and Performance

Forty-one articles (23%) were identified whose primary focus was evaluating resident practice and performance. The most common topics addressed were use of entrustable professional activities (EPAs) and the assessment of transitions of care (both n = 7, 17% each). A wide variety of tools were used in studies to evaluate outcomes. For example, Kessler and colleagues⁴⁹ used self-developed questionnaires to assess medical knowledge, Lundberg⁵⁰ used a previously validated tool to assess communication skills in PGY-2s, and Stark et al⁵¹ demonstrated the feasibility of evaluating professionalism using a 360° National Board of Medical Examiners tool. However, many studies identified in this theme were for single institutions and used institution-specific instruments.

Multiple studies evaluated the use of EPAs, such as hospital transitions of care, demonstrating the potential for the development and incorporation of EPAs to assess specific competencies.^52,53 Meade and colleagues⁵⁴ described a practical method for identifying curricular milestones that can be used to assess whether a resident can be entrusted with increased responsibility, demonstrating across multiple programs the potential application of milestones. A perspective⁵⁵ went further and suggested modifying the primary care exception rule to mandate the evaluation of residents' competence using the existing milestones framework. Legault and colleagues⁵⁶ noted the high frequency of inadequate or inaccurate documentation in resident discharge summaries. They suggested potential solutions, such as clinic and curricular innovations, and even an EPA.⁵²

Multiple studies identified disparities in the care of patients with resident PCPs (versus faculty PCPs), in areas such as diabetes care, cancer screening, discharge follow-up, and readmission rates.^43,57–62 Given those disparities, further research is needed in assessing resident competence via EPAs and in determining what factors predict worse outcomes in patients seen by residents to identify areas for intervention and find the balance between autonomy and supervision.

Nearly all studies were for single institutions and relied on resident self-assessment or evaluation of EHR documentation, with few studies using direct clinical observation. However, 3 of the 41 studies (7%) describe the evaluation of resident practice through direct observation, demonstrating logistical feasibility and trainee acceptance.^50,63,64 Enhancing opportunities for direct observation may enable better assessment of competency and improved supervision.

Teaching Methods

Nineteen articles (10%) focused on evaluating and comparing teaching methods used in ambulatory education. The most common teaching modalities evaluated included web-based (n = 10, 53%) and clinical practice as education (n = 4, 21%).

Multiple studies evaluated the use of web-based instruction as an adjunct to other teaching methods or as a stand-alone teaching method, showing successful improvements in knowledge-based outcomes. Sullivan and colleagues⁶⁵ completed a randomized educational trial that compared access to interactive web-based training on opiate prescribing for chronic pain with access to practice guidelines among residents in 5 different residency programs. They found that the web training group had greater improvements in knowledge and self-rated competence.⁶⁵ Cook et al⁶⁶ developed case-based web modules and, using non–case-based modules as a control, completed a randomized crossover trial in 2 academic residency programs, demonstrating that while learners preferred case-based teaching methods, they did not result in superior knowledge retention.

Relatively few studies directly explored changes in clinical practice as a teaching method. Petersen and colleagues⁶⁷ described a trial that randomized faculty-trainee dyads to either conference room presentations or examination room presentations after the trainee completed patient assessments, finding increased time spent with patients and greater patient, trainee, and faculty satisfaction in the examination room presentation model.

A wide variety of teaching methods were used in studies included in this review, although relatively few directly compared outcomes from different teaching methods. The ease of applying web-based training makes it a potentially appealing method. However, little research has been done looking at the benefits of combining clinical practice (and practice transformation) with online training modalities, which may enable higher-level educational outcomes. The articles in the teaching method theme, on average, achieved the highest MERSQI scores, with especially high marks on study design (many studies compared 2 separate groups or even randomized participants). Given the lack of evidence on optimal instructional methods and the methodologic flexibility enabled by a lack of direct impact on patients, further intrainstitutional and interinstitutional, nonrandomized and randomized, controlled trials should be pursued to further clarify optimal didactic methods for postgraduate trainees.

Program Evaluation

Fourteen articles (8%) related primarily to comprehensive program evaluation. The most common topics addressed related to understanding the current state of national ambulatory care education (n = 4, 29%), evaluating clinic structure and the effect on resident experience (n = 4, 29%), and evaluating of ambulatory education (n = 3, 21%), among others. These studies tended to be national surveys with good response rates; however, most relied on program self-assessment at the trainee satisfaction level.

A few studies used surveys to understand the current state of ambulatory care education or how new regulations affected ambulatory care education.^7,68,69 The majority of residency clinics are hospital based and provide care to patients with lower socioeconomic status. A majority had firm systems with teams caring for patients, and clinic schedules were often secondary to inpatient scheduling needs. The overall experience in clinic was highly variable and perceived levels of trainee stress were high.⁷ Trainee stress was due to competing demands in continuity clinic and poor knowledge about how to manage ambulatory issues. Multiple perspective articles highlighted the need to focus on residents' schedules, emphasizing the importance of patient continuity, teams, and mentors.^70–72 The greatest challenges were in implementing the ACGME requirements, recruiting core faculty, and performing competency-based assessment.^68,69

Kisiel and colleagues⁷³ observed residents' perspectives on effective outpatient education using focus groups, finding the teacher-learner relationship to be of paramount importance. Creating a second continuity clinic experience at a free clinic site, a private practice clinic site, and a patient-centered medical home clinic were all associated with improved resident satisfaction.^74,75 However, one study that evaluated components of the continuity clinic experience found that while most residents were satisfied with individual components of their ambulatory experience, no single feature was predictive of a career choice in primary care.⁷⁶

Larger-scale program evaluation studies noted several similarities to single institution studies previously discussed, with high levels of trainee dissatisfaction and significant clinical and curricular deficiencies. Structural changes, such as X + Y models and team-based care and interventions, to increase the educational quality of the clinic and didactic experience have the potential to address many of these concerns.