Does Early Involvement of Interns in a Quality Improvement Project Enhance Their Quality Improvement Skills?
ABSTRACT
Background Teaching quality improvement (QI) is required, and earlier exposure might improve QI skills.
Objective To examine whether interns assigned to an existing QI project early in training will have increased QI knowledge and confidence at the end of internship and require less supervision to conduct QI activities at the conclusion of residency compared with those who design their own project or join one later.
Methods In 2019, all pediatric interns were randomized to joining an ongoing QI project in the second quarter of internship (EARLY; n=16) or joining or creating one by the end of internship (LATER; n=15). QI skills were assessed at the beginning and end of internship using the AQIKS (Assessment of Quality Improvement Knowledge and Skills) and a 9-item QI self-assessment survey. At the end of residency, faculty mentors rated the amount of supervision needed to conduct a QI project using a 5-level scale.
Results At baseline, the groups were similar in AQIKS score, self-assessment, and previous QI experience. AQIKS scores increased over the internship year in EARLY (mean improvement 4.57; 95% CI, 1.85-7.30), but not in LATER (1.83; 95% CI, -0.64-4.31). EARLY interns improved in all 9 self-assessment items, whereas those in LATER improved in only 3. At the end of residency, LATER residents generally required less supervision to conduct QI projects compared with EARLY.
Conclusions In the short term, assigning interns to a QI project early in internship improved their QI knowledge and confidence but not their ability to conduct QI studies without some supervision at the conclusion of residency.
Introduction
Teaching quality improvement (QI) is a key component of graduate medical education, but the optimal method is unknown.1,2 Experiential learning is very effective in teaching QI, and efforts that promote this method would be expected to increase QI knowledge.3-5
Our hypothesis was that interns assigned to an existing QI project early in training would have increased QI knowledge and confidence at the end of internship and would require less supervision to conduct QI activities at the end of residency compared with those who design their own project or join one later.
Methods
We conducted the study during the 2019-2020 academic year at a large children’s hospital. We have a repeating 18-month QI curriculum that consists of a monthly didactic session and participation in a longitudinal QI project. During the investigation, there were no changes to the curriculum or QI mentor expectations.
Participants
We randomized all 33 pediatric interns into 2 groups using a random generator. Interns in one group were assigned to an existing project (EARLY), and those in the second designed their own projects or joined one of their choosing (LATER). We created a list of existing projects and asked the EARLY group to select one (online supplementary data Table 1). We made the assignments for the EARLY interns early in the second quarter of intern year, while those in the LATER group had a deadline to identify or create a project by the end of internship.
Outcome Assessments
To measure QI knowledge, we used the Assessment of Quality Improvement Knowledge and Skills (AQIKS),6,7 a tool developed for pediatrics similar to the Quality Improvement Knowledge Application Tool.8 The instrument includes 6 cases, each with 9 questions and a maximum score of 28. It was administered at the beginning and end of internship. Three faculty proficient in QI scored each case, and their ratings were averaged. Raters received no specific training and were blinded to group designation.
At the beginning and end of internship, interns completed a 9-item self-assessment of their QI proficiency. This tool utilizes a 5-point Likert scale ranging from not proficient to very proficient.9
At study onset, we asked interns if they had previously participated in a QI project, whether they had QI training, and if they intended to pursue QI in their career. At year end, we asked about their intention to pursue QI and preference for allowing future interns to create or be assigned a QI project.
At completion of residency, we asked each resident’s QI mentor to rate the amount of supervision the resident would need to execute a pediatric entrustable professional activity (EPA) previously entitled Apply public health principles and QI methods to improve population health on a 5-level supervision scale10,11 (online supplementary data Table 2).
We estimated we would need 15 subjects in each group to show a 30% increase in the AQIKS score at 90% power (α=0.05).6,7 We used a paired t test, Fisher’s exact test, or Mann-Whitney test, as appropriate. For the self-assessment, we converted the categorical choices to numerical data ranging from 1 to 5 and used the Mann-Whitney test.
The Washington University School of Medicine Institutional Review Board approved the study.
Results
All 33 interns completed the first set of assessments, 17 in EARLY and 16 in LATER. One intern in each group did not complete the post-intervention assessments and are not included in the analyses leaving 16 in EARLY and 15 in LATER.
At baseline, there was no difference (P>.05) between the 2 groups in previous QI experience, AQIKS scores, or the QI self-assessment survey (Table 1).
AQIKS scores improved in the EARLY group from the beginning to the end of internship (mean difference 4.57; 95% CI, 1.85-7.30; P=.003), but not in LATER (1.83; 95% CI, -0.64-4.31; P=.13). There was improvement in the QI self-assessment survey from the beginning to the end of internship in both groups. However, EARLY interns improved in all 9 items while those in LATER improved in 3 items (Table 2). Compared with LATER, EARLY interns showed greater improvement in 3 areas: (1) using cycles of change to improve care; (2) formulating a data plan; and (3) creating an interdisciplinary team. Over all 9 questions, the EARLY group improved by 0.91 points versus 0.45 in LATER (mean difference 0.46; 95% CI, 0.01-0.92; P=.048) on the 5-point Likert scale.
At the end of residency, LATER residents were judged to require less supervision to perform the QI EPA (P=.01). The median level of supervision required for LATER was 3 (IQR 3-4; online supplementary data Table 2) while that for EARLY was 2 (2-3).
Interns’ intent to pursue QI work after residency did not differ significantly between groups (P=.18). Interns had mixed feelings about when the QI project should be assigned, 7 of 16 (44%) interns in EARLY and 7 of 15 (47%) in LATER thought that in the future, interns should be assigned a QI project early in internship.
Discussion
In this single-site study, we found that EARLY interns showed improvement in AQIKS and self-assessment scores at the end of internship, while at the end of residency, those in LATER were deemed to require less supervision to conduct QI activities. Furthermore, nearly half of the interns were in favor of assigning a project early in internship.
Greenlaw et al used simulation to teach pediatric residents QI and found that their post-pretest AQIKS scores increased by an average of 2.6 points.12 This increase is higher than the 1.8 points we observed in the LATER interns, but less than the 4.6 points in the EARLY group, supporting a beneficial effect of beginning the project earlier.
We were surprised that LATER residents were deemed to require less supervision to conduct QI activities at the end of residency. This might relate to the variability of resident ability to contribute to the project and their engagement. The projects chosen by the EARLY interns included those that faculty indicated would benefit from an intern’s involvement. Some allowed for more participation, while others were in a sustainability phase. Alternatively, the entrustment decision involves more than capability, and perhaps those who created a project better recognized their limits and asked for assistance when necessary.13 The EPA results also may be affected by which phase of the project a resident joined, the resident role in the project, and the extent of the mentoring relationship.
There are several limitations. We did not repeat the AQIKS or the self-assessment survey at the end of residency due to the COVID-19 pandemic. Perhaps if we were able to, the LATER residents would have shown a greater improvement in QI knowledge. Other limitations include faculty understanding of EPAs and insufficient information in which to assign an EPA level of supervision such as the extent of trainee engagement and role in project. However, since the groups were randomized, this should have affected both equally. Nonetheless, these are opportunities for further research. All interns are now beginning QI projects earlier. This provides an opportunity for further research on factors leading to the ability to conduct QI activities without supervision.
Conclusions
Early involvement of interns in their QI activity improves their QI knowledge and confidence at the end of internship but not their ability to conduct QI studies without supervision at completion of residency.
Author Notes
