Assessing the Impact of Innovative Training of Family Physicians for the Patient-Centered Medical Home
Abstract
Background
New approaches to enhance access in primary care necessitate change in the model for residency education.
Purpose
To describe instrument design, development and testing, and data collection strategies for residency programs, continuity clinics, residents, and program graduates participating in the Preparing the Personal Physician for Practice (P4) project.
Methods
We developed and pilot-tested surveys to assess demographic characteristics of residents, clinical and operational features of the continuity clinics and educational programs, and attitudes about and implementation status of Patient Centered Medical Home (PCMH) characteristics. Surveys were administered annually to P4 residency programs since the project started in 2007. Descriptive statistics were used to profile data from the P4 baseline year.
Results
Most P4 residents were non-Hispanic white women (60.7%), married or partnered, attended medical school in the United States and were the first physicians in their families to attend medical school. Nearly 85% of residency continuity clinics were family health centers, and about 8% were federally qualified health centers. The most likely PCMH features in continuity clinics were having an electronic health record and having fully secure remote access available; both of which were found in more than 50% of continuity clinics. Approximately one-half of continuity clinics used the electronic health record for safety projects, and nearly 60% used it for quality-improvement projects.
Conclusions
We created a collaborative evaluation model in all 14 P4 residencies. Successful implementation of new surveys revealed important baseline features of residencies and residents that are pertinent to studying the effects of new training models for the PCMH.
Introduction
Primary care is undergoing transformative and rapid change in the United States, including new models of delivery such as the Patient Centered Medical Home (PCMH), and new standards for health information technology.1–8 These innovations are being implemented in the context of busy, complex, primary care health systems.6 In a time of transformation of primary care practices and societal demands for increased accountability, this suggests a need for educational research to study the outcomes of efforts to redesign primary care physician training.
The American Boards of Family Medicine, Internal Medicine, and Pediatrics are interested in directing attention and resources to inspire innovation in residency education.9–12 Studies have shown that evaluation of educational interventions often lack the rigor needed to draw meaningful conclusions.13–18 Studies with a longitudinal database from multiple educational sites with a larger sample of learners have the advantage of being able to detect outcomes over time and to provide greater generalizability than single-site studies.19 The extent to which residents are exposed to features of PCMH and whether their appreciation of those features is related to working with them during residency has not, to our knowledge, been studied to date. We describe the development of standardized core measures that capture longitudinal data at 14 Preparing the Personal Physician for Practice (P4) sites,20 and report baseline findings derived from those measures.
Methods
The P4 project (2007–2012) is a comparative case study of 14 residencies that are experimenting with changes in residency education. It seeks to produce physicians capable of effectively practicing in the PCMH model of care and to lead the transformation process to implement this new model. Detailed descriptions of individual programs' innovations, hypotheses, and site-specific measures are reported elsewhere.20–22
Core Measurement Development and Testing
Each program evaluated site-specific hypotheses, using locally selected measures and implementation schedules,22 and participated in a mixed-methods evaluation study using standardized core measures and an online diary system. To identify core measures, members of the P4 Executive Committee and Evaluation Team, located at Oregon Health & Science University, developed a list of process and outcome variables relevant for P4 evaluation, regardless of the innovations being tested. To ease the burden of data collection, the study used existing data sources, such as the Accreditation Council for Graduate Medical Education's Accreditation Data System and the National Resident Match Program. In addition, board scores and in-training exam scores will be supplied by the American Board of Family Medicine to examine changes in clinical knowledge over time.
Survey Development
The assessment of the P4 consists of 4 surveys. A resident survey is completed annually in November and assesses demographic information, the influence of P4 on residency ranking, satisfaction with rotations and faculty participation, how residents provide input into program revision, satisfaction with the quality of the program, and attitudes about the importance of PCMH features. A program survey, completed by program directors in the fall of each year, collects faculty and program characteristics, including financial data and residency match data, and asks the program directors to rate the health of the program. A continuity clinic survey, completed by clinic medical directors, collects information on patient demographics, practice characteristics, and aspects of the PCMH that have or have not been implemented during the project period. Finally, a graduate survey, completed by graduates of P4 programs 18 months after residency graduation, assesses practice characteristics, scope of practice, adequacy of residency training, and presence of PCMH features in graduates' clinical practices.23 All surveys were pilot-tested using cognitive interviewing techniques24 with program directors, medical directors at continuity clinics, and family medicine residents at both the University of Washington and Oregon Health & Science University (neither of which are involved in the P4 program).
The surveys were designed for historical cohort analyses, to allow comparisons of programs, continuity clinics, and residents participating in their respective innovations. The graduate survey will also create cohorts of pre-P4 and post-P4 resident graduates to facilitate the assessment of changes over time. Lastly, the P4 evaluation team launched an online diary system based at the Agency for Health Care Research and Quality that is designed to electronically prompt P4 program faculty, residents, and staff approximately every 6 months, allowing for collection of qualitative data on aspects of, and experiences with, implementing change. Online diary questions from the first 3 years will be reported elsewhere. Descriptive statistics were used to profile data from P4's baseline year, and Spearman correlation coefficient was used to assess baseline relationships between attitudes toward, and exposure to, PCMH features.
Results
The response rates for the core surveys in the baseline year (2007) were 100% (14/14) for the program survey, 100% (24/24) for the continuity clinic survey, and 94% (310/330) for the resident survey. The baseline graduate survey was obtained in 2008 for those who graduated from residency in 2006 with a response rate of 89% (88/99). The survey data from program, resident, and continuity clinic are reported in tables 1, through 3, respectively. About 70% of visits done by residents are routine visits (chronic care and health maintenance), whereas about 25% are acute visits. Faculty clinicians, as a group at each continuity clinic, performed on average just more than 9000 visits per year, whereas residents performed nearly 8000 visits per year. In 2009, approximately one-half of the continuity clinics were using the electronic health record (EHR) for safety projects, and nearly 60% (14/24) were using it for clinical quality improvement projects. figures 1 and 2 illustrate the status of implementing electronic and nonelectronic features of the PCMH at the continuity clinics, indicating that the electronic features most likely to be mature (in 50% [12/24] of continuity clinics) were having an EHR and having fully secure remote-access available. The mean number of years an EHR was in place was 3.65 years with a range of 1 to 12 years. Only 42% (10/24) of clinics were using teams to manage patient care. In contrast, integrated behavioral health and integrated case management were present or mature in 71% (17/24) and 54% (13/24) of continuity clinics, respectively.
Citation: Journal of Graduate Medical Education 4, 1; 10.4300/JGME-D-11-00035.1
Citation: Journal of Graduate Medical Education 4, 1; 10.4300/JGME-D-11-00035.1
Residents exposed to the P4 innovations in the baseline period were primarily women with similar racial and ethnic diversity as has been reported to exist among current, practicing primary care physicians in the United States.25 Nearly one-third of P4 residents attended medical school outside the United States, and many were born outside the United States.
The predominant type of continuity clinic for P4 residents at baseline was a family health center “model” office; few were in federally qualified health centers. Most patients in all settings were assigned to a personal physician at baseline, which likely indicates the high commitment these programs had for patient-centered health care at the start of the project. Nearly half of the clinics were using an EHR for safety projects, and most were using this technology for clinical quality improvement projects. These high rates at baseline likely reflect the high commitment to PCMH features in these early adopters of innovation. Although a recent study comparing hospitals at various stages in the adoption of computerized health records found little difference in quality of care,26 continued study of how best to implement EHR systems to get the greatest gains in care is needed.
Discussion
Evaluating innovation in the P4 residencies requires establishing baseline measures about essential features and individuals in the programs and prospectively studying how these key characteristics change over time. The P4 core evaluation instruments have been successfully designed and are currently being used in all 14 P4 residency programs, and we have attained nearly 100% response rates. Using a central evaluation core allowed for development and implementation standards for data exchange and the formation of a successful, residency-redesign collaborative around a common research agenda.
Program directors' global ratings of satisfaction with their programs were lower, relative to their ratings of faculty development programs and the morale of both the faculty and residents. This lower relative level of satisfaction may have motivated the program directors to apply to be a P4 program as an impetus for change. Overall, however, the means for all these ratings indicated there is room for improvement in faculty development and in the morale of faculty and residents. It will be important to see how these measures change over time.
We found that the continuity clinics for residency training varied in implemented features of the PCMH, with less than 50% having a mature EHR and fully secure remote access available, although other electronic features were in place in less than one-half the continuity clinics at baseline. Most of the continuity clinics were using integrated behavioral health and integrated social services to deliver care to their patients, and many were beginning to use a team-based care approach. Managing a multidisciplinary team is a skill physicians need if primary care is to be successful in achieving high quality outcomes.27
Ultimately, evaluation of the P4 approach may be effectively performed by Centers of Excellence in Educational Research. Such a model has been proposed previously19 because it is unlikely that every residency program and undergraduate medical school can afford to support experts in educational design, faculty development, measurement, and analytic/statistical expertise at the local level. Forming regional or national exchanges with a comprehensive longitudinal database from multiple collaborating programs strengthens our ability to answer important medical education research questions.28 As a result of the P4 initiative, we have registered a practice-based research network—PCERN (Primary Care Educational Research Network)—through AHRQ and are hopeful that our efforts will serve as a model for other schools and programs that could provide important, much appreciated expertise.
Conclusions
Fourteen family medicine programs selected to be a part of the P4 project have successfully implemented annual core study measures and achieved high response rates and continue to participate in P4 although they received no funding directly from P4 for their work. Our careful pilot-testing of core instruments resulted in robust interpretable data, as indicated by responses that were well characterized and usable in complex analyses, which they were in our assessment of the correlation between exposure to PCMH features and the residents' ratings of their importance. Most of our core measures are survey based and primarily quantitative. Although an online diary system was implemented to collect supplemental, qualitative data, entries tended to be brief because of respondents' busy schedules, and assessing change processes may be more challenging than we expected.
A collaborative evaluation model was created in all P4 residencies. Successful implementation of new surveys discerned important baseline features of residencies and residents that are pertinent to studying the effects of revised training for the PCMH.
Status (%) of Electronic Features of the Patient-Centered Medical Home (n = 24)
Status (%) of Nonelectronic Features of the Patient-Centered Medical Home (n = 24)
Author Notes
Patricia A. Carney, PhD, is Professor of Family Medicine and of Public Health and Preventive Medicine at Oregon Health & Science University; M. Patrice Eiff, MD, is Professor and Vice Chair of Family Medicine at Oregon Health & Science University; John W. Saultz, MD, is Professor and Chair of Family Medicine at Oregon Health & Science University; Erik Lindbloom, MD, is Associate Professor of Family and Community Medicine at University of Missouri; Elaine Waller, BA, is Research Associate of Family Medicine at Oregon Health & Science University; Samuel Jones, MD, is Program Director of the Fairfax Residency Program at Virginia Commonwealth University and Chair-Elect of the American Board of Family Medicine; Jamie Osborn, MD, is Assistant Professor of Clinical Medicine at Loma Linda University School of Medicine; and Larry Green, MD, is Professor of Family Medicine at University of Colorado.
Funding: This work was supported by the Preparing the Personal Physician for Practice (P4) Project, which is jointly sponsored by the American Board of Family Medicine Foundation and the Association of Family Medicine Residency Directors, and the Family Medicine Research Program at Oregon Health & Science University.
