Association of a Gamified Journal Club on Internal Medicine Residents’ Engagement and Critical Appraisal Skills
Background Prior literature demonstrates internal medicine residents have suboptimal competence in critical appraisal. Journal clubs are a common intervention to address this skill, but engagement and critical appraisal skill improvement are variable.
Objective We evaluated journal club engagement and critical appraisal skills after implementation of a gamified format.
Methods This was a single-arm study, conducted from July 1, 2020 to June 30, 2021, involving internal medicine residents at 2 US programs. Residents participated in a 12-month gamified journal club that sorted residents into 2 teams. Residents attended an orientation followed by 6 to 10 monthly, hour-long competitions. In each competition, a subset of the resident teams competed to answer a clinical prompt by critically appraising an original article of their choice. A chief medical resident or faculty member moderated each session and chose the winning team, which received a nominal prize of candy. The primary outcome was engagement, measured by a 7-question survey developed de novo by the authors with Likert scale responses at baseline and 12 months. The secondary outcome was critical appraisal skills assessed by the Berlin Questionnaire.
Results Sixty-one of 72 eligible residents (84.7%) completed both engagement surveys. Residents reported statistically significant improvements in most dimensions of engagement, including a higher likelihood of reading articles before sessions (posttest minus pretest score -1.08; 95% CI -1.34 to -0.82; P<.001) and valuing time spent (posttest minus pretest score -0.33; 95% CI -0.55 to -0.11; P=.004). Critical appraisal skills marginally improved at 12 months (posttest minus pretest score -0.84; 95% CI -1.54 to -0.14; P=.02).
Conclusions Our study demonstrates a gamified journal club was associated with improvements in engagement and minimal change in critical appraisal skills.
Introduction
Teaching evidence-based medicine and critical appraisal of the literature is a longstanding tradition of residency education. Sir William Osler is credited as the originator of journal club beginning in 1889 at Johns Hopkins University.1,2 This honored tradition has evolved over the last century. Journal club was initially conceived as a mechanism to disseminate limited and difficult-to-obtain publications. Due to the exponential increase in the number and accessibility of journal articles, journal club has transformed to identify rigorous articles from a voluminous literature and to endow residents with the skills to interpret and apply findings to individual patients.
Unfortunately, multiple observational trials have shown that medical trainees struggle with interpreting statistical data and applying the information to clinical practice.3-5 Graduate medical educators have the unique opportunity to cultivate this skillset during residency training. Despite critical appraisal being included in the Accreditation Council for Graduate Medical Education Milestones for residency education, little is known about the effectiveness of specific curricula. Anecdotally, programs frequently implement a journal club to imbue these skills, but effectiveness and resident engagement are variable.6,7
Gamification, the application of game design elements such as competition to nongame contexts, is a common technique implemented to increase engagement and information retention in medical education.8 One adaptation of gamification principles to a journal club curriculum demonstrated that a majority of residents preferred the format, with a significant increase in the number of articles read.9
In this study, we assessed journal club engagement and resident critical appraisal skills after implementing a gamified journal club format.
KEY POINTS
Methods
Setting and Participants
This was a single-arm, exploratory study of a medical education intervention conducted between July 1, 2020 and June 30, 2021. We implemented our curricular innovation at 2 US internal medicine residency programs. Northside Hospital Gwinnett is a community-based training program of 45 residents in Atlanta, Georgia. The University of Alabama at Birmingham-Huntsville is a hybrid academic-community program of 27 residents in Huntsville, Alabama.
Intervention
The gamified journal club format was adapted from a protocol described by Rodriguez and Hawley-Molloy.9 Authors S.A., A.B., B.H., and R.K. integrated the protocol into their respective sites using their background as graduate medical educators in evidence-based practice. The curricular format simulated the practice of translating clinical questions into clinical decision-making through independent literature search, study selection, critical appraisal, and application to individual patients. The curriculum consisted of 1 to 2 orientation sessions and 6 to 10 monthly competition sessions during the 12-month intervention. Orientation sessions introduced the format to residents, detailed how to conduct a literature search, and reviewed the essential elements of critical appraisal.
At the beginning of the intervention, residents were divided into 2 competing teams. Before each competition session, 2 resident groups of 2 to 4 people each were drawn from the larger teams to compete. A case vignette with a clinically relevant question developed by a chief medical resident or faculty member was disseminated to the competing groups by email at least 2 weeks before each session. Each resident group performed its own literature search, selected a randomized controlled trial of its choice from the primary literature to answer the clinical question, and prepared a critical appraisal. The same clinical prompt was repeated in multiple sessions until all resident cohorts within the program’s rotation cycle participated.
Each 1-hour competition session was moderated by a chief medical resident or faculty member involved in the curriculum and familiar with the format. During each session, both resident groups presented a critical appraisal of their chosen study and discussed its applicability to the patient scenario. A faculty expert discussant facilitated critical review and gave a brief didactic on a biostatistical topic relevant to the studies.
Before ending, the faculty discussant and chief medical resident deliberated briefly and selected the winning group. Although a formal scoring rubric was not used, 3 domains were considered: (1) the quality of the selected article; (2) the quality of the residents’ critical appraisal; and (3) the quality of the residents’ translation of the study findings to the unique patient in the clinical vignette. These domains were highlighted for residents in orientation sessions and repeated in correspondence with residents before each competition session.
The winning group each session received candy as their award. A cumulative score was tallied after each session for the larger competing teams. At the end of the academic year, residents from the winning team received a meal catered by the program as well as an inscription on a rotating novelty trophy.
Outcomes
The primary outcome was subjective engagement in the gamified journal club format compared to the previous format. Engagement was measured by a 7-question survey administered to residents immediately before and after the 12-month curricular intervention (provided as online supplementary data). We developed the survey de novo to assess engagement, as we could not find an existing survey in the literature that queried the domains deemed most relevant to the curricular format. Authors R.K., S.A., J.Z., B.H., and A.B. identified the relevant domains, including consideration and review of journal articles prior to sessions, engagement during sessions, and self-assessed transformation of engagement into practicable skills for interpreting and interacting with scientific literature in the future. The survey assessed unique dimensions of engagement with Likert scale responses. Four additional questions were included in the post-intervention survey that queried direct comparisons between the gamified journal club format and the program’s previous format. Survey questions were piloted to a small convenience sample of residents who indicated appropriate interpretation of question wording and response scales. Paired observations were collected from residents in postgraduate year (PGY)-1 or higher.
The secondary outcome was objective critical appraisal skills as assessed by the Berlin Questionnaire, a measure for assessing critical appraisal skills with validity evidence in postgraduate medical education.10 This questionnaire was administered at baseline and at 12 months.
Statistical Analysis Plan
A paired samples t test was conducted to compare baseline and post-intervention responses for statistical analysis of the primary outcome. For the secondary outcome, a paired samples t test was conducted comparing the absolute number of correct questions at baseline and post-intervention. A 2-sided test was considered significant at an alpha level of 0.05. No adjustments were made for multiple comparisons, as this was an exploratory study.
This study was deemed exempt by the Institutional Review Board at both participating sites.
Results
Paired Likert scale data were available for 61 internal medicine residents who completed pre- and post-surveys. The survey response rate was 84.7% (61 of 72) for residents participating in the journal club, with relatively equal proportions from each PGY (32.7% PGY-1; 36.1% PGY-2; and 31.1% PGY-3).
Subjective Engagement
Residents reported a statistically significant improvement in 6 of the 7 dimensions of engagement surveyed (Table). Notably, means improved among residents reporting they liked the articles chosen for journal club (posttest minus pretest score -0.57; 95% CI -0.80 to -0.35; P<.001), and they were more likely to read the articles before attending journal club (posttest minus pretest score -1.08; 95% CI -1.34 to -0.82; P<.001). Despite these behavioral changes, residents were not more likely to look forward to journal club (posttest minus pretest score -0.16; 95% CI -0.48 to 0.15; P=.30).
Sixty-nine residents (95.8%) completed the post-intervention survey, which was sent to all residents independent of their completion of the pre-intervention survey. It included 4 questions designed to compare the gamified journal club format and the program’s previous format. A frequency distribution details the percentage of respondents choosing each option within the Likert scale (Figure). More than 75% (52 of 69) indicated they were “probably” or “definitely” more involved in journal club and more likely to engage during journal club sessions with the new format. The most commonly selected response for all direct comparator questions was “definitely.”
Citation: Journal of Graduate Medical Education 15, 4; 10.4300/JGME-D-22-00812.1
Objective Critical Appraisal Skills
Residents correctly answered 34.3% (5.145 of 15) questions before the intervention and 39.9% (5.985 of 15) questions after the intervention on the Berlin Questionnaire (posttest minus pretest score -0.84; 95% CI -1.54 to -0.14; P=.02).
Discussion
Our study demonstrates that a gamified journal club format was associated with improvements in residents’ self-reported engagement at 12 months. The behavioral domain of engagement—the proportion of residents who actively read journal articles before each session—had the largest magnitude of improvement. Prior systematic reviews have consistently demonstrated a similar positive effect of gamification on behavioral engagement in various learner-driven tasks.8 For example, a gamified intervention to increase first-year surgical residents’ voluntary usage of a laparoscopic simulator yielded an approximate doubling in usage time.11
With respect to objective critical appraisal skills, our baseline results are consistent with prior literature demonstrating suboptimal skills in this domain. A previous trial examining a dedicated 4-week evidence-based medicine rotation for internal medicine residents and medical students reported a baseline score on the Berlin Questionnaire of 46%, similar to our study’s 34.3%.12 This study demonstrated an improvement in attitudinal measures but no improvement in Berlin Questionnaire scores. Although the present study yielded a statistically significant improvement in the Berlin Questionnaire, the small magnitude of the improvement is of questionable educational significance. The Berlin Questionnaire content primarily emphasizes quantitative biostatistical skills, whereas our didactics emphasized qualitative microskills related to study appraisal and interpretation.
The gamified journal club format required additional preparation time and resources compared to a traditional, faculty-led format. It also required an initial time investment by curriculum leadership to develop clinical vignettes that accomplished several goals: addressed a relevant clinical question, optimized for specific journal article selections, and selected a topic with 2 or more high-quality, relevant articles available for resident selection. Each clinical vignette required approximately one hour of allocated time to craft a prompt satisfying these goals. This task was shared by the curriculum leadership at both programs to reduce its impact on each site. Nominal prizes required a small budgetary allocation of approximately $250 per program per year. The residents selected to compete in each session spent additional time selecting an article and preparing their critical appraisals. However, despite the additional prompt development and resident time required, the gamified format has continued at both study sites because of the increase in journal club engagement.
There are several limitations to this study. The nonrandomized, single-arm nature precludes causal inference. It is possible that the residents progressing through another curriculum would demonstrate similar improvements in self-reported critical appraisal skills. The primary and secondary outcomes assessed domains relevant to resident education but did not capture the impact on clinical outcomes, such as greater adherence to evidence-based treatment decisions.
Further study of a gamified journal club format could better delineate the possible benefits of gamification on teaching evidence-based medicine to residents. A randomized controlled trial of 2 or more journal club formats, including a gamified and a traditional journal club format, would isolate the effect of format on engagement and critical appraisal skills with externally generalizable results.
Conclusions
The results of this study suggest a gamified journal club format implemented for internal medicine residency programs is associated with increased resident engagement in journal club.
Participant Responses to Direct Comparator Questions Between Gamified Journal Format and Previous Formats
Author Notes
