Comparing the Effects of Design Thinking and A3 Problem-Solving on Resident Attitudes Toward Systems Change
ABSTRACT
Background
Quality improvement (QI) is a required component of graduate medical education. Many medical educators struggle to foster an improvement mindset within residents.
Objective
We conducted a mixed-methods study to compare a Design Thinking (DT) approach to QI education with a Lean, A3 problem-solving approach. We hypothesized that a DT approach would better promote a mentality of continuous improvement, measured by residents' resistance to change.
Methods
Thirty-eight postgraduate year 2 internal medicine residents were divided into 4 cohorts during the 2017–2018 academic year. One cohort participated in an experimental QI curriculum utilizing DT while 3 control cohorts participated in the existing curriculum based on Lean principles. Participants voluntarily completed a quantitative Resistance to Change (RTC) scale pre- and post-curriculum. To inform our understanding of these results, we also conducted semistructured interviews for qualitative thematic analysis.
Results
The effect size on the overall RTC score (response rate 92%) was trivial in both groups. Three major themes emerged from the qualitative data: factors influencing the QI learning experience, factors influencing creativity, and general attitudes toward QI. Each contained several subthemes with minimal qualitative differences between groups.
Conclusions
This study found similar results in terms of their effect on attitudes toward systems change, ability to promote creative change agency, and educational experience. Despite positive educational experiences, many residents still did not view systems-based problem-solving as part of their professional identity.
Introduction
Quality improvement (QI) is a required component of graduate medical education (GME).1 In 2017, the Accreditation Council for Graduate Medical Education (ACGME) formalized the expectations for QI education in the Common Program Requirements, prompting even more curricula in this area.2
Teaching QI in GME often involves guiding residents through a project designed around a local gap in health care quality, using A3 problem-solving.3,4 These curricula are based on principles from Lean5–7 and the Model for Improvement8,9 which use the plan-do-check-act (ie, Deming) cycle.10 However, many medical educators still struggle with how to adapt them to health care and meaningfully engage residents in ways that promote a mentality of continuous improvement.11–13
Innovation frameworks such as Design Thinking (DT) are now being taught in some medical schools.14 Several major health care systems have leveraged these frameworks to improve patient outcomes while remaining economically viable in the volatile health care market.15,16 DT emphasizes observation, empathic interviewing, and immersing oneself in a problem from another person's perspective. The insights gained inspire inexpensive, low-effort prototypes that can be rapidly tested through small-scale iterative experiments to methodically test evolving hypotheses (Table 1).17,18
We hypothesized that a DT approach to QI would more strongly promote a mindset of continuous improvement in residents, compared with traditional QI curricular approaches because of the similarities between DT and clinical medicine, namely their explicit focus on empathic problem-solving. Since all QI work requires change, we drew from Rogers' Diffusion of Innovations Theory, which posits that an individual's willingness to adopt change falls on a bell-shaped curve.19 We then hypothesized that learning a structured approach to problem-solving in a familiar health care context with conceptually accessible QI tools could positively affect one's own attitude toward systems change, regardless of where one might naturally fall along Rogers' curve. We utilized a quantitative measure of residents' attitudes toward change as a curriculum evaluation measure for our research, then performed qualitative interviews to inform our understanding of this measure and which elements of the curriculum had the most influence on resident attitudes.
Methods
Setting and Participants
The QI curriculum for categorical internal medicine residents at our institution is delivered during the postgraduate year (PGY)-2 in the form of an experiential longitudinal project, facilitated by a faculty member with QI training and experience. The curriculum includes 16 in-person contact hours distributed over 10 months in the form of 1- or 3-hour teaching sessions every 8 weeks. During our study period, there were 38 categorical residents in the PGY-2 class. Residents are randomly grouped into 4 cohorts of 8 to 12 residents. Each cohort works on a different project that is facilitated by 1 of 4 faculty members. These faculty members, referred to as core QI faculty in our residency program, are proficient in A3 problem-solving and had at least 3 consecutive years of experience teaching QI to residents prior to the study. During this study, one core QI faculty member (R.B.) had both Lean A3 training and DT training. This study took place during the 2017–2018 academic year.
While our institution utilizes a blended QI framework that incorporates elements from both the Model for Improvement and Lean, the internal medicine residency program utilizes the Lean A3 problem-solving approach as the scaffolding for its curriculum.
Intervention
During the study period, we had one experimental group of residents (cohort D) who learned and applied the DT framework to approach a local QI problem. The other 3 groups of residents (cohorts A–C) served as control groups and learned to apply the A3 problem-solving framework to a local QI problem. The curriculum for the experimental DT cohort (provided as online supplementary data) was developed using tools and resources available online to the public through IDEO and The Hasso Plattner Institute of Design at Stanford17,18 as well as educational resources recommended by the chief innovation officer at our institution.20–22 The curriculum for the 3 control cohorts was developed and refined by our faculty over many years and is grounded in the A3 problem-solving approach to QI.3 An outline of this curriculum can be found in the online supplementary data. The characteristics of each cohort's QI project can be found in Table 2. QI projects were selected based on resident interest (cohorts A and B) or departmental and residency program strategic priorities for QI that involved residents (cohorts C and D). During the study period, residency program leadership requested that one cohort work on the problem of handoff interruptions. The QI faculty member for cohort D (R.B.) addressed this problem with his cohort.
Outcomes
We used the Resistance to Change (RTC) scale (provided as online supplementary data) as a quantitative measure of residents' attitudes toward systems change before and after the curriculum.23 Semistructured interviews were analyzed using thematic analysis24 as the qualitative evaluation to inform our understanding of resident attitudes toward health care systems change, how the curriculum may have impacted their views of change, other aspects of the curriculum, and QI as a discipline.
Quantitative Outcomes
The RTC scale is a 17-item, 6-point survey instrument that has validity evidence through studies in adult populations and was “designed to measure an individual's dispositional inclination to resist changes.”23 It was administered on paper before and after participation in the QI curriculum. Each response form was deidentified, but pre-post linkage was maintained using unique identifier codes.
RTC scale responses were tabulated and analyzed for effect size using Cohen's d.25 Effect size was chosen as the statistical measure for this study because of the small size of our experimental group. Using Cohen's standards, an effect size with an absolute value < 0.2 was considered trivial, ≥ 0.2 to < 0.5 was considered small, ≥ 0.5 to < 0.8 was considered medium, and ≥ 0.8 was considered a large effect. A negative effect size indicated a decreased resistance to change and was considered the desirable outcome.
Qualitative Outcomes
An interview guide (provided as online supplementary data) was developed through an iterative process by the research team, comprised of 2 medical educators with QI expertise (R.B. and J.M.), a qualitative researcher (J.S.), an MD/MPH candidate (A.S.), and the chief innovation officer of our institution (R.R.). All residents were invited to participate in a semistructured interview via email. No additional incentives were provided. Interviews were conducted in person or over the phone from May to July 2018 by a trained interviewer (A.S.) who had no association with the development of the QI curriculum or the leadership of the internal medicine residency program. All interviews were audio-recorded, transcribed, deidentified, and loaded into NVivo 12 (QSR International Inc, Burlington, MA) for analysis. Interview transcriptions were reviewed for accuracy against the audio recordings.
The research team developed a codebook for analysis through an iterative process. Interviews were analyzed by 2 investigators (R.B. and A.S.) who met routinely to review and refine coding. Four interviews were selected for duplicate coding with interrater reliability reaching an initial median kappa of 0.57 (-0.01–0.95). Coding differences were discussed until agreement was reached, changes were made to the codebook, and a second round of duplicate coding with 4 more interviews was conducted. After this round, the combined interrater reliability reached a median kappa of 0.66 (0.22–0.89) for all 8 interviews. Again, differences were discussed until agreement was reached with a focus on those codes with lower kappa scores. The codebook was adjusted, and remaining transcriptions were divided and coded by 1 of the 2 investigators. Once coding was complete, the research team used thematic analysis to identify emergent themes from the data.26–28
The study was reviewed and considered exempt by the University of Pennsylvania Institutional Review Board. While resident participation in the QI curriculum was a mandatory residency component, completion of surveys and interviews was voluntary, and verbal informed consent was obtained.
Results
The experimental group included 11 residents (11 of 38, 29%), while the other 27 residents (27 of 38, 71%) were divided among 3 control groups. Twenty residents (20 of 38, 53%) volunteered to participate in interviews, and we reached saturation of themes with these interviews. Eight of these residents (40%) were from the experimental group and 12 of 20 (60%) were from the control groups. All 38 residents completed the pre-curriculum RTC questionnaire, while 35 of 38 (92%) completed the post-curriculum questionnaire. The post-curriculum response rates in the experimental and control groups were 11 of 11 (100%) and 24 of 27 (89%), respectively.
Quantitative Results
At baseline, the average overall RTC score for the entire study population was 2.97. During validation studies, Oreg found means ranging 3.00–3.36.23 At baseline, the average overall RTC score was higher, indicating more resistance to change, in the experimental group (3.26 of 6) compared to the control group (2.84 of 6). Postintervention, the effect size on the overall RTC score within each group was trivial, measuring 0.03 and 0.16 for the experimental and control groups, respectively.
Qualitative Results
Three major themes emerged from the qualitative data: factors influencing the QI learning experience, factors influencing creativity, and general attitudes toward QI. Within each major theme, there were several subthemes. Table 3 contains a full list of these subthemes with sample quotations. We will focus on the most prominent themes and note any differences between the groups.
Factors Influencing the QI Learning Experience
The 2 most prominent factors influencing the QI learning experience were peer engagement and learning a systematic methodology. Residents from both groups cited examples of how their peers' level of interest or excitement about the project directly impacted their overall experience. These comments carried both positive and negative connotations in both groups. Both groups viewed learning a systematic approach, either DT or A3 problem-solving, as a positive contributor to their experience. Facilitator factors such as enthusiasm, organizational skills, and delegation were noted by several residents as was the importance of having a personal connection to the problem they were trying to solve. Residents reported that working on a project that was meaningful to them and/or addressed a problem that they encountered in their work led to a more positive experience. This sentiment was expressed by residents in the cohorts who self-selected their QI project and the cohorts who worked on leadership-selected projects with relevance to residents.
Factors Influencing Creativity
Creative agency, or the recognition that an individual or team was able to creatively affect their environment, was noted by residents in both groups. Residents described that the curriculum helped them feel empowered to impact the health care system in a creative way. Most residents, regardless of group, believed in creative plasticity, or the thought that creativity could be learned to some degree. However, the experimental group more frequently displayed creative confidence or a positive self-image about their own creative skillset. Furthermore, the experimental group more frequently identified specific curricular activities as promoting creativity, describing many creative tools within the methodology. Residents from both groups described seeing medicine as the antithesis of creativity, citing treatment algorithms and clinical pathways as evidence. While this was not a widespread sentiment, some felt strongly that these skills were unfamiliar and at times even unnecessary in clinical practice.
Attitudes Toward QI
General attitudes toward QI surfaced when questions related to prior QI experiences and future career plans were explored. There were no qualitative differences between groups under this theme. There was a mix of positive and negative attitudes with many residents simultaneously offering both views. One resident reported that their QI curricular experience helped them discover a new career interest in QI. Many residents identified applications in their future career for lessons learned during the curriculum. Some who identified specific career interests in basic science research or medical education saw a potential translation of QI methodology, such as creative problem-solving skills, to their specific career goals. Others saw benefits of QI knowledge and soft skills that they developed through their QI project, such as communicating with stakeholders and empathic interviewing. Still, while many residents saw opportunities to apply these lessons learned, several specifically mentioned that QI work was not part of their career plans.
Discussion
This study suggests equipoise between DT and A3 problem-solving as frameworks for QI curricula in their effect on internal medicine residents' attitudes toward systems change. While there were minimal differences between groups, our qualitative findings can help inform QI curriculum development in several respects. The most surprising results were that residents did not necessarily view creative problem-solving as a useful skill for their careers as physicians, and that despite being able to see other applications of QI skills, many residents did not view QI as part of their clinical work or professional identity as a physician.
Regardless of the curricular framework utilized, residents found value in learning a logical, systematic approach to QI. Perhaps this can be attributed in part to similarities between these frameworks. While the terminology is different, they are both rooted in the scientific method. Furthermore, residents are familiar with applying a structured approach to history-taking, differential diagnosis generation, and other aspects of clinical medicine, so this was not surprising. Similarly, it was expected that the level of peer engagement directly affected the learning experience. Since interpersonal dynamics are a key factor to the success of any team, and all QI work involves a team, this should be a deliberate consideration in QI curriculum design.
Other key factors to consider when developing a QI curriculum, regardless of framework, include project selection and faculty development. Our residents wanted to feel a personal connection to the problem they were solving. This sentiment did not seem to be directly related to the residents' control over project selection. However, residents who can select their own project are likely to choose something meaningful to them. This presents a challenge to QI educators who are trying to balance resident engagement with the desire to engage them in interprofessional projects that are aligned with local clinical quality goals.12,29 Indeed, finding a QI problem that is meaningful to the residents, measurable, actionable, and institutionally aligned is the elusive holy grail of QI education. Since residents remarked on several facilitator factors such as enthusiasm for the subject, investing in professional development for QI faculty is likely to pay dividends both for assistance with project selection and for residents' learning experiences.
While curricula can be modified to influence the QI learning experience for residents, it is much more challenging to foster a mindset of continuous improvement. Residents in both the DT and Lean groups of our study recognized that QI concepts and skills could be applied to aspects of their future careers, but also did not view QI as part of their future work. QI education remains unpopular with many residents and medical students30,31; thus, the applicability of QI principles to other aspects of a physicians' career may be a critical “hook” for QI educators. It remains concerning that many of the residents in this study did not view improving health care as part of their future work as physicians. Some also viewed clinical practice as algorithmic, precluding creativity. Solving problems for individual patients and solving problems for the local health care system appeared to be conceptually different to our residents.
For resident physicians, the feeling of being powerless to effect change can loom large. While it was encouraging that residents across both groups reported feeling empowered to have a creative impact on their environment, we did not find meaningful quantitative shifts in either groups' overall RTC score. However, the qualitative differences in factors influencing creativity suggest that DT may have more effective applications for specific learners or specific problems. It is important to note that these frameworks contain tools, not formulas, and we believe that other QI educators could benefit from the expanded curricular toolbox that DT provides.
This study is limited in that only internal medicine residents in one cohort at a single institution were included, which limits generalizability. Comparison across groups is confounded by faculty differences. While all our faculty were proficient in QI methods, teaching abilities and level of enthusiasm for the subject among the faculty may have varied, which in turn could impact the residents' satisfaction with their education. Similarly, the selection of QI projects (resident selected vs leadership selected) varied among cohorts, which could have impacted resident attitudes. However, our qualitative results emphasized that the relevance of the problem to residents' work, rather than whether they selected the problem, was the more important factor. As we modify available DT resources to meet our curricular needs, other applications might produce different resident perceptions. The RTC scale has some validity evidence but was not designed to measure creative tendencies and has not been used with physicians, thus it may not have effectively captured resident attitudes toward change. The study was not designed to examine other outcomes such as QI knowledge or skills, so we are unable to determine if our approaches influenced these outcomes. Participants did not provide feedback on our qualitative analysis findings, and the interview guide was not piloted with residents prior to use.
Future research steps may include exploring factors that influence resident professional identity formation related to QI and interventions or curricula that promote openness and creativity related to systems-based problem-solving.
Conclusions
This study, the first to compare different problem-solving methodologies for use in QI education, revealed qualitative and quantitative equipoise between DT and Lean A3 frameworks with both curricula fostering creative agency yet producing trivial effect size on residents' RTC. While our residents identified creative positive changes that they were able to effect on the clinical environment, as well as the broad applications of the skills learned, many still did not see systems-based problem-solving as part of their future professional identity or practice.
Author Notes
Editor's Note: The online version of this article contains the syllabus for an experimental quality improvement curriculum using design thinking in graduate medical education, a description of the curriculum used in the study, the Resistance to Change scale, and a structured interview guide.
Funding: The authors report no external funding source for this study.
Conflict of interest: The authors declare they have no competing interests.
This work was previously presented at the NEGEA Annual Conference, Philadelphia, Pennsylvania, April 4–6, 2019.
