Program Evaluation for Graduate Medical Education: Practical Approaches From the Reimagining Residency Evaluation Community of Practice
Introduction
Program evaluation is a crucial component of graduate medical education (GME), as it systematically assesses educational interventions through standardized methods, feeding data back to programs for evidence-based improvements to advance training in high-quality, patient-centered care. Without this rigorous analytic approach, we cannot know how, when, and why interventions affect learning, care quality, and patient health outcomes. Program evaluation requires specialized skills and training1 that educators and administrators may not possess, and for which institutional resources may be lacking. This gap hinders evidence-driven change in GME.
Educational innovations require iterative evaluation cycles for continuous quality improvement. Perspectives of those affected should also be considered based on evidence provided, to increase the applied utility.2 Outcomes that are products of a deliberately iterative improvement process are an important engine for transforming the medical education system.3 Program evaluation provides evidence for an intervention’s feasibility, applicability, and effectiveness for learning at the micro (individual learner), meso (institution) and macro (system) levels, including whether the innovation contributes to the medical profession’s social contract. This can be determined by connecting patient outcomes–focused research to medical training as evidence for competence, accountability, and the public good.4
In this Perspective, American Medical Association (AMA) Reimagining Residency Initiative (RR) program evaluators provide practical considerations for GME educators as they plan evaluation in the complex educational environment of residency and fellowship. We relied on program evaluation resources5 applied in GME settings and built unique approaches to support our innovations. The information provided aims to help other educators evaluate their programs.
Reimagining Residency Program Evaluation Community of Practice
In June 2019, the AMA launched RR with a vision to “transform residency training to best address the workplace needs of our current and future health care system.”6 The initiative funded 11 teams to develop, pilot, and evaluate innovations to improve workforce training in the AMA’s GME priority areas: competency-based medical education, improving the transition from undergraduate medical education to GME, enhancing the clinical learning environment, advancing health systems science competencies (eg, systems thinking, health care delivery, health system improvement), and developing new models of culturally competent care to address social drivers of health and health equity (eg, access to nutritious food, quality housing, clean water, transportation). Each team identified and included an evaluation expert who assessed progress relative to stated goals and outcomes.
To foster collaboration and deliberate learning across these 11 teams, the AMA’s research and evaluation leadership scheduled monthly meetings to promote the sharing of expertise, provide feedback on research and evaluation approaches, work through challenges, carve a space for professional development (leading medical education experts joined some meetings), promote scholarship, and determine the extent to which results realized the desired change(s). In medical education, only some institutions have this type of expertise on staff, underscoring the need to build community. The AMA organized sessions to provide value and develop expertise further. This collective commitment to learning has been previously described as a community of practice (CoP).7 Over time, the group focused on advanced thinking and practices to benefit the field of medical education and patient-centered care. The group also gained perspective from discussions of program evaluation methods across institutions while managing tailored approaches for specific projects. Each of the 11 teams benefited greatly from this collective platform supporting unique designs and growth. Fostering this type of peer community for deliberate learning and advancement is highly recommended for multi-institutional efforts.
Program evaluation requires rigor but also adaptability in cycles of data collection, harmonization or calibration, analysis, and incorporation of change. In the online supplementary data, we include the title of each grant, website address for a description of the project and progress, exemplar evaluation approaches, rationale, and resources for interested GME educators. We recommend program directors create a similar evaluation culture that strategically weaves assessments across the program, rather than creating assessments later and risking misalignment with implementation.
Key Considerations for Program Evaluation in GME Innovations
As the RR CoP developed and revised evaluation plans for innovations, key questions shaped approaches. These considerations, along with traditional program evaluation standards, such as accuracy, utility, integrity, and feasibility8 are some of the first steps taken in designing a program evaluation effort. We include questions here to assist other GME educators. When answers are not immediately evident, educators should develop their evaluation approach further until sufficient information is available.
Types of Change
Innovations can be improvement-oriented with little expectation that organizational cultures will shift. This type of innovation focuses on incremental change9 within the current education system and framework. Innovations can also be transformative, shifting individual beliefs and frameworks leading to cultural change in education systems. A local innovation may include both types of change in different facets of the program. It is important to identify the change desired, theory behind it, and what is needed for change to occur. To determine the type of change, questions include: Why and how will change occur in a particular context (theory of change)? What are the observable behaviors of residents and faculty today, and what are the behaviors expected after the innovation? How different are these behaviors based on measures and metrics? What policies and resources are in place to facilitate learning the innovation employed versus staying with the status quo?
Systems Perspective
Understanding a system’s beliefs and behaviors will determine how GME program leaders design and evaluate effective innovations, frame outcomes, and promote change. It is also important to consider other influential systems (eg, health care) within which residents practice, as well as accreditation and professional organizations’ influence on graduate education and clinical practice. National policies also influence the environment within which the innovation occurs. To achieve an effective systems perspective, questions include: What practices and processes of your educational system are working and not working? What are the shared mental models of your organizational culture? What individual mental models exist that could be barriers or facilitators to your innovation? What other systems influence the prospect of change, and how do they do this?
Program Evaluation Designs
Program evaluation should aim to assess both the adoption of the initiative (process) as well as its effectiveness at creating and sustaining change (outcome measures). To provide framing for a tailored program evaluation approach, questions include: What is the explicit theoretical framework (eg, developmental, behavioral, social constructionist) guiding the innovation and its alignment with all program components? Has a culture of evaluation been created so all invested parties are committed to what will be assessed and changed as a result? Is there a process in place for rapid cycles of evaluation3 aimed at desired change with results fed back into the system in an iterative fashion? Are there processes in place for collaboration where all have an opportunity for meaningful involvement?
Social Return on Investment
Social return on investment (SROI) is a conceptual approach for calculating the social, health, environmental, and economic value of program interventions for invested parties.10 This calculation is useful when communicating the return realized from education innovations in terms more familiar to those interested in financial and cost-benefit outcomes. Steps include: (1) establishing scope and identifying key program beneficiaries11; (2) mapping outcomes to beneficiaries using identified theory of change or outcomes framework; (3) identifying evidence for outcomes and assign value; (4) establishing what would have occurred without the intervention and what happened due to the intervention; (5) calculating SROI with partners by assigning financial values for nonfinancial changes; and (6) completing cost-benefit calculation, including sensitivity analysis.
Summary Considerations
The diverse evaluation methods used by the AMA RR Evaluation CoP (online supplementary data) reflect the importance of carefully selecting evaluation methods that are specific to the goals and objectives of the innovation.12 Employing a variety of methods can ensure that real-life intricacies, unexpected changes, and local contexts are considered in real-time. By using evidence-informed flexibility in program evaluation, it is possible to adapt to changing environments and circumstances, which increases the potential for others to use effective innovations in different contexts.
The CoP environment provided a longitudinal opportunity to build relationships and expertise focused on the dynamic nature of GME program implementation. This powerful community experience rises above while also enhancing the technical methods learned and shared in the supplemental materials. We strongly recommend program directors, GME educators, and evaluators look for ways to create, facilitate, maintain a collaborative, iterative, continuous learning process when designing their own residency or fellowship programs.
Author Notes
