Longitudinal Experience With a Transparent Weighted Lottery System to Incentivize Resident Scholarship
ABSTRACT
Background
Promoting resident scholarship is important to programs. Positive Peer-Pressured Productivity (P-QUAD) is a dual incentive model that combines increased transparency through awareness of peers' engagement in scholarship, with a weighted cash lottery where tickets are earned for various dimensions of academic success (ie, 1 point/ticket for an abstract submission up to 6 for manuscript acceptance).
Objective
We explored whether a weighted lottery system contributes to sustained increases in academic productivity in a residency program.
Methods
We implemented P-QUAD in 1 pediatrics residency program in July 2015. Residents reported their scholarship submissions/acceptances for the prior year, establishing a program baseline. During the 2-year intervention, residents logged their academic submissions/acceptances on a web interface where they could view real-time scores and the work of their peers. At the end of each academic year, we compared P-QUAD points for each category to baseline.
Results
During the intervention, 31% of residents (68 of 218) reported engaging in scholarship. Using P-QUAD was acceptable to most residents. Engagement in scholarship across the program, as measured by total P-QUAD score, increased 53% from baseline (329 versus 504 points per year). Mean submission and acceptance rates for individual residents reporting research through P-QUAD increased across all categories, ranging from 19% for abstract submissions (1.62 to 1.93 per year) to 275% (0.24 to 0.90 per year) for accepted manuscripts.
Conclusions
The residency program sustained gains in academic productivity at the program-wide and participating resident level in the 2 years since implementing P-QUAD.
Introduction
Building skills and habits in scholarship during residency is a critical foundation for physicians, including those who do not pursue careers as researchers.1 Although many programs have infrastructure in place to promote scholarly activity, creating a flourishing culture of resident scholarship can be challenging.2,3
Programs have aimed to increase scholarly productivity with various interventions, including protecting residents' time, creating dedicated research tracks or curricula, or assigning research mentors.3–8 These interventions often require substantial time commitments and resources. We implemented a novel approach to promote resident scholarship that would not require a new curriculum or changes in resident schedules.
We describe the implementation and outcomes of a novel dual incentive program called Positive Peer-Pressured Productivity (P-QUAD). The intervention sought to leverage positive peer pressure associated with increased awareness of colleagues' scholarly activity and used a lottery-based incentive where more productivity leads to more chances to win cash prizes.9
Methods
Intervention
We previously developed P-QUAD for faculty as a peer-mentoring model with financial incentives, where points are earned based on degrees of academic productivity. Point values include 1 point for an abstract submission, 2 points for a poster acceptance or rejection or manuscript resubmission, 3 points for a platform or workshop acceptance, 4 points for a manuscript submission, and 6 points for a manuscript acceptance. These points translate into tickets for a semiannual raffle, with a greater chance of winning based on an individual's degree of productivity. During the faculty pilot, we saw a statistically significant doubling of abstract submissions.9
As we brought P-QUAD to residents, we wanted them to be able to see a leaderboard of submissions in real time, both to promote their work and to leverage the gamification aspect where the positive peer pressure of friendly competition can encourage outcomes.10–13 We created a web-based platform (www.pquad.org), where residents could voluntarily log their submissions, see their P-QUAD score, and search through peers' submissions. We used a free web-based survey software (Google Forms, Google Inc, Mountain View, CA) for residents to log their P-QUAD points and ultimately contracted with a web designer to create a more polished web interface for P-QUAD logging and leaderboard display. We promoted its use at residency meetings and via e-mail (figure 1). We held cash drawings every 6 months for $750 in prizes. The only direct cost associated with P-QUAD was the cash prizes, which came from philanthropic funds already earmarked for supporting resident research.
Citation: Journal of Graduate Medical Education 10, 4; 10.4300/JGME-D-18-00036.1
At the completion of the intervention, we sent residents an anonymous electronic survey for program evaluation. The survey was developed by the authors and did not undergo validity testing. We used Likert scale questions to determine the degree to which residents found the various aspects of the program to be motivating and/or stressful as well as multiple-choice questions to ask residents if they anticipated logging future scholarship achievements via P-QUAD or if they had connected with peers about scholarship because of what they had seen.
The study received Institutional Review Board approval from the University of Minnesota.
Analysis
We implemented P-QUAD in our pediatrics residency program on July 1, 2015, and collected data through June 30, 2017. Residents' scholarship submissions and acceptances for the prior academic year provided the baseline data. At the end of each academic year, we compared mean P-QUAD points reported for each category, comparing groups of residents at the same level of training as well as the overall residency program.
Results
Engagement in scholarship across the program, as measured by the total P-QUAD score for each year, increased 53% from baseline, from 329 points across the program preimplementation to 504 after the second year, with sustained increases in all categories except abstracts submitted.
In the preintervention year, 34% of residents (37 of 109) reported engaging in scholarly activity as measured by submissions and/or acceptances of academic work. During the collective 2-year intervention period, 31% of residents (68 of 218) reported engaging in scholarship. There were increases in mean scores from baseline across all categories, ranging from a 19% increase (1.62 to 1.93 per year) in abstract submissions per resident engaged in research to a 275% increase (0.24 to 0.90 per year) in accepted manuscripts per resident (figure 2).
Citation: Journal of Graduate Medical Education 10, 4; 10.4300/JGME-D-18-00036.1
Fifty-four percent (59 of 110) of residents provided feedback via the survey. Most (78%, 46 of 59) agreed that P-QUAD increased their awareness of peers' scholarly work, with some (12%, 7 of 59) reporting that they reached out to a resident to collaborate specifically because of what they had seen logged; and 41% (24 of 59) agreed that P-QUAD motivated them to engage in scholarship, with (34%, 20 of 59) reporting that the increased transparency was motivating and 28% (16 of 59) indicating that the opportunity to win cash was a motivation for engagement. Some residents (25%, 15 of 59) indicated that P-QUAD produced stress due to the competition; more than twice that number disagreed with this sentiment (54%, 32 of 59). No respondents indicated that they were opposed to submitting their future work via P-QUAD.
Discussion
Implementing a transparent, engaging platform that highlights resident scholarship was associated with increased scholarly productivity in our program. Although we did not see an overall increase in the number of residents who reported scholarly work, we did notice increased productivity for the residents who engaged with the lottery system, and for the program as a whole, suggesting that this approach may be most effective in motivating residents already inclined to participate in research. As previous work has shown a strong association between productivity during residency and plans to pursue research in a future career,14 furthering the productivity of these engaged residents during their training may be beneficial.
An important finding is the increase in residents submitting manuscripts. Fostering an environment that leads to residents climbing the scholarship ladder from abstracts to manuscripts is an important goal for all training programs. Previous work has demonstrated that requiring research within residency leads to an increase in manuscript publications.15 Our study demonstrates, however, that similar increases can be attained via a voluntary incentive-based system. Interestingly, abstract submissions were relatively stagnant. It is possible the increased emphasis on scholarship motivated some residents to advance their prior work to a full-fledged publication.
While many residents indicated that P-QUAD motivated them to engage in scholarship, more study is needed to determine if the explicit emphasis (via increased point values in the lottery) on manuscript submission was a motivator for residents or if engaging more frequently with scholarship naturally led them to increased productivity.
Our study has limitations. Our program served as its own control, and it is possible that scholarship among residents may be increasing across all programs. We recognize that some residents who engaged in scholarship may have chosen not to log their submissions. Future study will include tracking scholarly productivity over time for graduates who participated in P-QUAD compared with graduates prior to the intervention.
Conclusion
Implementation of a transparent weighted lottery system that reported and incentivized scholarly work was associated with increased scholarly productivity in our residency program. The greatest increase was seen in the category of submitted manuscripts.
Sample of Explanatory Flyer to Introduce and Promote Use of P-QUAD
Comparison of Average Annual Scholarship Contributions per Resident
Author Notes
Funding: Research reported in this article was supported by the National Center for Advancing Translational Sciences of the National Institutes of Health Award Number UL1TR000114.
Conflict of interest: The authors declare they have no competing interests.
This work was presented as a poster at the Pediatric Academic Societies Meeting, San Francisco, California, May 6–9, 2017.
The authors would like to thank Abe Jacob, Ron Furnival, Ann Marie Weber-Main, and Nancy Raymond for their roles in the development and initial pilot of P-QUAD and Brian Krohn for his assistance in creating the web-based platform.
The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
