Introduction

There is worldwide interest in quantifying the benefits of accreditation systems to patient care, trainee learning, and well-being, and the quality of the learning environment. Accreditation systems, concentrating on specific metrics and outcomes, have the potential to impact health locally and globally.^1–3 Recent examples include findings of a reduction in operating room times⁴ and improvement in nurses' perceptions of the quality of care.⁵

However, assessing the impact of accreditation may be challenging in nations such as the United States with well-established accreditation systems, as these benefits may have accrued over time and may be difficult to detect in the current cohort of accredited programs. This may explain the few high-quality studies and lack of positive relationships in a 2016 Cochrane review of the effectiveness of external inspection systems in health care.⁶ It may also be that the impact of accreditation is most powerful when accrediting bodies specify and emphasize quality improvement processes, such as data collection, oversight, and monitoring of outcomes.^7–13

In the United States, the Accreditation Council for Graduate Medical Education (ACGME) emphasizes quality improvement for residency programs in its accreditation process across several areas: curricula, educational goals and objectives, teaching and supervision, a robust evaluation system, increased opportunities for learning, an enhanced focus on quality and patient safety, and opportunities to work in interprofessional teams.

The ACGME and its Review Committees have been accrediting US residency programs since the 1950s. In Singapore, the Accreditation Council for Graduate Medical Education International (ACGME-I) began accrediting programs in 2010 at the request of the Singaporean Ministry of Health. The “newness” of accreditation in Singapore provides an excellent opportunity to assess the impact of accreditation, and in this article we seek to determine whether introduction of and adherence to accreditation standards in Singapore improved key process measures relevant to program quality over the 7-year period from 2011 through 2018. Our hypothesis is that these measures will show gradual improvement across the years of accreditation for all programs, with the greatest impact in the initial years of accreditation, and for programs with the lowest early scores.

Methods

The ACGME-I began accrediting core specialty programs in 2010, with the first year of data collection in 2011. While ACGME would eventually accredit 41 core programs in 3 health systems (National Healthcare Group, National University Health System, and Singapore Health Services), the initial cohort consisted of 34 programs (table 1). We followed this cohort from 2011 through 2018. As part of the accreditation process, ACGME-I requires annual reporting once a program has reached the first accreditation stage for new programs, termed Initial Accreditation. Data in these annual reports include program-level information about the clinical learning environment, adherence to the accreditation standards, and information regarding participating sites, residents on duty, and teaching faculty. Additionally, after January 2012, residents and faculty in accredited programs participated in annual surveys that collected data on their perception of their clinical learning environment. These surveys were anonymous and confidential, and the annual reporting window began in January and lasted through May. ACGME-I expects a 70% response rate to be reached each year. Annual results are provided to programs having at least 4 out of 5 respondents. Programs not meeting this criterion receive aggregate survey data once achieving a minimum of 4 out of 5 respondents over a 3-year period.

table 1 Sample of Programs and Residents (2011–2017)

View Fullscreen

Measures

We selected 7 questions from the larger survey that measured adherence to and could be affected most directly by specific standards and processes. We also selected the survey's global rating (respondents' overall perception of the program). The measures in our analysis are shown in table 2.

table 2 Content Areas, Changes in Resident Ratings (Means) by Year, and Significance Across Time

View Fullscreen

All items were rated on a 1 to 5 scale; 6 of the 8 items had scale anchors of 1, not at all/never, to 5, extremely. The remaining 2 items (“80 hours per week” and “education compromised by service obligations”) were reverse scored so that 1, extremely, and 5, not at all/never. We limited our analyses to resident responses because of the larger sample size and because we anticipated that program accreditation activities, including ongoing monitoring, would have the most powerful impact on residents and will be most prominently reflected in their evaluations.

Results

Table 1 shows the number of programs and residents by specialty type and health care system for the 34 programs and 465 residents used in the analyses.

While there were no effects for sponsor or specialty type, repeated measures analysis for all items combined showed significant improvement across the 7 years (F(6,23) = 3.51, P = .013).

To further explore this significant effect, additional repeated measure analyses were conducted for each question and content area of the survey, assessing whether there were any significant effects across time. Six of these 8 individual items attained a significance level of P ≤ .05 for improvement across time, while 2 (“work in teams” and “overall evaluation of the program”) did not reach significance.

In addition, for the duty hour item, there was a significant interaction between time and specialty group (F(12,46) = 2.90, P < .005), suggesting that some specialty groups improved at a greater rate than others. Examination of the data show that the medical specialties showed greater improvement across the time frame. Similarly, for the “environment of inquiry” item, there was a significant interaction between time and sponsor group (F(12,46) = 2.63, P = .009). Table 2 shows the year-to-year increase overall and for each survey item.

Improvement effects overall were similar for the 3 specialty groupings and the 3 sponsoring institutions. These data showed variability in the rates of change, with greater change in the earlier years, as well as differences in the starting points for the different content areas. For example, while resident perceptions of “service obligations” improved, this was at a lower rate than change in the other areas. The figure shows these data in graphical form.

View Figure

• Download as Powerpoint
• Download Figure

To test the hypothesis that accreditation results in greater impact in initially lower-performing programs, we analyzed the 11 programs in the lowest third of the distribution for “all items combined” in the first year of accreditation. The results show a trend for these 11 programs: their increases were larger across time than those for programs in the second or third quartile (F(6,22) = 2.38, P = .06). However, if we restrict this analysis to the first 3 years of accreditation (2011–2013), the results do reach significance (F(2,27) = 5.30, P < .011).

While there were few differences among specialties, we found an interaction between specialty type and duty hour compliance, with greater improvement in programs in medical specialties, compared with surgical and hospital-based specialties. We also found a significant interaction for “environment of inquiry” by sponsoring institution and time, with 1 sponsor (Singapore Health Services) showing greater improvement.

Discussion

The application of accreditation standards in 2010–2011 affected Singaporean residents' evaluations of their programs, suggesting that accreditation influenced key process measures that related to program function and performance. These effects were observed across all specialty groupings types and the 3 sponsors, with the greatest improvement for programs initially in the lowest third of the distribution. These programs saw the largest improvement in the accreditation process.

The duty hour interaction noted (specialty type and time) suggest that the impact of accreditation, while showing improvement across all specialty types, was greater for medical specialties. This may reflect the more challenging nature of reducing duty hours in surgical and hospital-based specialties such as orthopaedic surgery and emergency medicine. The significant interaction for “environment of inquiry” and time for 1 sponsoring institution suggests greater relative ease with which this institution was able to attain improvements on this measure. This may have been due to the sponsoring institution's medical school affiliation.

Our findings are similar to the results of research on the effect of accreditation in undergraduate medical education, which showed that accreditation influenced data collection and analysis, monitoring, creation and revision of policies and procedures, quality improvement, curriculum reform, and faculty engagement, among other measures.⁷ The ultimate impact of accreditation may depend on alignment of organization goals with accreditation standards. This includes organizational commitment to accreditation standards, educational capabilities and infrastructure that is strengthened through accreditation, and sufficient time to work toward those shared goals.¹¹ That some specialties experience greater challenges in compliance with duty hour standards echoes findings from the United States, suggesting that surgical specialties in particular find it more difficult to reduce duty hours for their trainees.¹⁴ A limitation of this analysis is the relatively small sample sizes of programs. However, this sample does adequately capture the population of residents in Singapore. Another limitation is the fairly narrow range of responses on the survey. We attempted to address this by using repeated measures to more powerfully test the possible changes across time, thus ensuring that each program serve as their own control in the analyses.

Conclusion

The implementation of ACGME-I accreditation in Singapore provided for added educational and clinical learning environment infrastructure that was not in place prior to the 2010 accreditation launch. The existence of these more deliberate education and evaluation systems is reflected in residents' perceptions of the learning environment in their programs, including prioritizing learning over service obligations and creating an environment for inquiry. Future assessments of the effects of ACGME-I accreditation, both in Singapore and other international settings, might expand these results by using stakeholder interviews to further describe the efficacy, value, and impact of the accreditation system.