Introduction

Graduate medical education (GME) programs in the United States are substantially funded by Medicare, Medicaid, and the Department of Veteran Affairs.¹ The Balanced Budget Act of 1997 capped the number of GME positions used to calculate reimbursement for sponsoring institutions,² yet residency positions have increased from 20 209 positions in 1997 to a record high of 30 232 positions in 2018.³ Hospitals have employed alternative funding strategies, such as seeking state funding^4,5 and absorbing GME program costs into institutional operating budgets, to expand their GME footprint.⁶ There are many reasons a health care system would invest in non–federally subsidized GME positions. Residents provide more workforce hours, including less desirable night and weekend shifts, for a relatively low professional salary.⁷ Residents are critical members of many hospital safety net programs^8,9 and teach junior residents and medical students, thereby relieving clinically productive faculty from some teaching duties.¹⁰ In addition, GME programs provide a pipeline for faculty recruitment, which is particularly important in rural states.^11–14 Teaching hospitals may provide higher quality of care in some settings.¹⁵

In 2017, the Maine Medical Center (MMC) Graduate Medical Education Committee (GMEC) received an unprecedented 18 requests to create new GME programs or to expand existing programs. Although our GMEC has a vetting process for individual requests, there was no established strategic process to differentiate worthy program requests to guide the overall expansion of the GME footprint. A literature search revealed 2 articles that described factors influencing strategic GME growth,^16,17 but none that could be applied in our setting.

Using systems engineering principles to address issues in health care is increasingly popular.¹⁸ Tradespace exploration is a systems engineering methodology to facilitate decision making regarding complex systems. In this methodology, alternative system designs are assessed based on a chosen set of attributes, sometimes called ilities (eg, accessibility, reliability, profitability, reproducibility). Each ility is assigned a scoring algorithm based on available quantitative or qualitative data. Finally, each alternative system design is collectively scored and compared based on each of its ilities.^19,20 While tradespace exploration is traditionally used to explore alternative technical system designs, it has also been identified as a useful tool for evaluating sociotechnical enterprises, such as health care systems.²¹ Based on the ability of this method to account for difficult to quantify factors, it was identified as applicable to our project.

We describe an innovative process framework for developing institution-specific metrics to strategically assess and prioritize GME program expansion requests to yield the greatest benefit for patients, learners, and the GME community as well as to meet local and state priorities.

Methods

The MMC is an independent academic medical center in Portland, Maine, and the anchor institution for MaineHealth, a unified group of hospitals across the state. The MMC sponsors 272 residency and fellowship training positions in 22 programs. Maine is the second-most rural state in the country, with 59.6% of the population living in federally defined rural areas.²² Many rural hospitals and practices struggle to recruit a robust physician workforce (MaineHealth Office of Recruitment, unpublished data, August 6, 2018).

In 2017, a steering committee was created that included the chief academic officer, vice-president of medical education, designated institutional official (DIO), vice-president of quality and safety, senior financial analyst, and director of the Center for Performance Improvement. This group defined the desired outcome of this project: to develop a reproducible, fair, valid, and reliable process to assess all requests for growth in GME. The 6-step process is summarized in figure 1.

View Figure

• Download as Powerpoint
• Download Figure

Step 6: Feedback and Analysis

All applicants or existing programs received recommendations for their programs, suggestions for strengthening a future application, and a summary spreadsheet of all programs and scores (with program names removed except their own). Programs were invited to give feedback at 2 focus groups and were informed that they could appeal their program's recommendation. Correlation analysis was performed in Microsoft Excel 2016 (Microsoft Corp, Redmond, WA) to ascertain if any categories dominated others when assigning final scores and if some categories did not affect the final score. The score a program received in each category was graphed relative to its final score, and an R² correlation coefficient was calculated. While this analysis did not necessarily show causation, it did expose variables that offered little predictive value and therefore would be less useful in future iterations of this process. Table 1 summarizes the estimated time all stakeholders spent for process development, scoring, and feedback sessions.

table 1 Estimated Time Spent on Each Step of the Process Development by Participants^a

View Fullscreen

Results

Eighteen programs submitted applications requesting 72 new positions in 1 new residency program, 10 new fellowship programs, expansion for 6 existing residency programs, and 1 new APP residency program (table 2). The GME residency and fellowship programs and APP programs were considered together for funding recommendations. Scoring and final recommendations for all programs are summarized in the online supplemental material.

table 2 Summary of Requested Residency and Fellowship Positions and Final Recommendations

View Fullscreen

Metrics group survey respondents (48%, 14 of 29) determined the weight of each category and agreed or strongly agreed that the process had incorporated input from all stakeholders and was unbiased by any group. Focus group feedback revealed 2 main themes: communication and rigorous model deliberate process. Improvement in communication around the process was recommended, despite considerable effort to provide oral and written communication during the process. Consistent feedback suggested general acceptance of the process as rigorous and deliberate. Existing programs questioned the need to be evaluated in this process if they were not seeking to expand. No program appealed their recommendation. Institutional leaders who scored programs suggested adding a session next year for applicants to present their request to scorers with an opportunity for a question-and-answer session.

Final scores were fairly evenly spread across a wide range, suggesting that the measurement process offered enough detail to differentiate programs from one another. The correlation analysis between each program's total score and score in each category was calculated (provided as online supplemental material). The program quality category scored low (R² = 0.011), suggesting that this category did not have a significant impact on final scores. Individual R² scores for the remaining categories ranged from 0.413 to 0.662, which are relatively high considering the qualitative nature of the measures and the social complexity of assessing a GME enterprise.

Discussion

An innovative, reproducible 6-step strategy for guiding GME growth at a sponsoring institution met our main objective of developing a rigorous process to allow differentiation among program requests such that an objective peer group could make fair recommendations for funding considerations.

Variables in 4 of the 5 categories had very high R² correlation values with the final scores for each program, suggesting a relationship between that variable and the final outcome. The combined scores in these 4 categories also had a very high correlation of R² values with the final scores for each program (figure 2). It was unexpected that the only variable not correlating well with final scores was program quality. One possible reason is that the rubric utilized for this category had more subcategories than others and may have been too refined to substantively impact total score. This category also was the only one related to program performance rather than some measure of program importance, suggesting that it fundamentally scored programs differently. Based on this, the process was modified for next year so the program quality metric will be scored first to ensure a minimum threshold for any new requests before further scoring and assessment.

View Figure

• Download as Powerpoint
• Download Figure

A significant effort was made to maximize the rigor, transparency, and objectivity of this process, given the certainty that all 72 requested positions would not be funded in a single budget cycle. Engagement of a systems engineer and use of a method with prior evidence directly applicable to our project reassured stakeholders that the process was evidence-based.

Little has been published on strategies to develop reproducible metrics to guide GME program expansion within sponsoring institutions. Chen et al²³ measured workforce outcomes of GME programs within sponsoring institutions and teaching sites, similar to the MaineHealth metric, but recruitment factors alone were not comprehensive enough to guide global GME expansion. Greenwood-Clark and Boll presented an innovative approach to conducting a GME value analysis to determine financial value of GME programs.¹⁶ They offered a method to analyze existing GME programs to maximize value to the institution, considering such important variables as teaching revenues, clinical coverage, and program costs. This strategy, while comprehensive, is more applicable to existing program analysis, where financial data already exist. The time and effort required to collect relevant data for many programs would be significant, and this process was not vetted and created by GME and institutional stakeholders, which we believe was critical to ultimate acceptability of any process that allocates limited resources.

The time commitment for this process in subsequent years will be significantly less; process and metric development time is nonrecurring, existing programs not seeking expansion will not be scored, and the application will be modified to have applicants, rather than GME staff, provide more relevant data. We estimate that the process will require approximately 8 hours of DIO and GME staff time, 2 hours of senior leadership time, and 2 hours of GMEC time. The decision to score all existing programs not looking to expand was unnecessary and added time to the overall process. All programs are already assessed by our annual program evaluation process, and the MMC, MaineHealth, and financial metrics were not relevant to current programs' value to the health care system. We included program contraction recommendations to explore potential reallocation of underutilized resources to new requests. However, decisions about contraction were made solely on review of recruitment data and program fill rates—not on metric scoring—which was a source of confusion and dissatisfaction for some programs.

One of the unforeseen benefits of this process was that broad communication of the program quality scoring rubric helped define local standards for program quality and clearly articulated desired characteristics for programs most favored to receive funding for expansion, such as high achievement in resident and faculty scholarly activity. This may allow for some measure of program alignment in the future.

Our study has limitations. Although the process to guide GME program prioritization may be utilized by any sponsoring institution, the developed metrics are unique to MMC and cannot simply be applied elsewhere. Sponsoring institutions interested in adapting this process would need to spend a considerable amount of upfront time and stakeholder engagement to develop local metrics. Additionally, the intentional separation of different groups to minimize carryover of biases from one step to the next added considerable time and effort to the project.

The longitudinal success of this process to prioritize GME expansion to meet specific local and state goals will not be assessable for several years. This process will be repeated annually, which will allow for tracking of any impact this GME expansion prioritization process may have on population care and recruitment in Maine.

Conclusion

Applying a systems engineering approach to develop institution-specific metrics for assessing GME expansion requests provided a reproducible framework, allowing consideration of institutional, health care system, and regional societal needs, as well as program quality and funding considerations in a process that participants perceived to be fair.