Introduction

Unverifiable research citations have been shown to occur at an alarming rate among applicants to various medical fellowships and specialty programs.^1–4 In 2008, we found that 11% of applications to the Maine Medical Center primary care sports medicine fellowship contained an unverified publication.² In 2009, the American Medical Society for Sports Medicine changed the primary care sports medicine application by requiring that written proof of claimed publications and presentations be submitted, including copies of the publication cover page or the conference presentation program listing.

While many programs have documented the high prevalence of unverifiable citations among applicants,^1–8 to our knowledge, this change in sports medicine application requirements marks the first action taken by an academic medical specialty to confirm scholarly citations. The current study was designed to evaluate the rate of unverifiable citations after the sports medicine fellowship application began requiring proof of publications and presentations.

Methods

We retrospectively examined all applications to 5 primary care sports medicine (PCSM) fellowship programs for 3 academic years (2010–2013), beginning after the application started requiring proof of citations in 2009. The total number of programs ranged from 108 (2010–2011) to 131 (2012–2013). The participating fellowship programs were located in New England, in the Midwest, and in the West. Our verification methodology was more extensive than that of other studies, and included directly contacting the journal publisher or conference sponsor if we could not otherwise verify a citation.^5–9 One trained research investigator traveled to each site, reviewed applications, and abstracted applicants' primary medical specialty, claimed publications and presentations, and any substantiation of their citations.

We defined publications as articles in peer-reviewed journals and presentations as talks presented at regional, national, or international meetings. Other types of publications or local presentations were not included in the study. Citations were considered verified if the cover page of the publication or if the meeting program listing was included with the application. For publications, if a cover page was not included, we performed PubMed and Google Scholar searches by author name. If this was unsuccessful, a medical research librarian performed an additional search. If the publication was still unconfirmed, we directly contacted the publisher of the claimed journal to confirm validity. Presentations were confirmed by auditing the program of the cited meeting or by contacting the sponsoring organization for validity. Any citation not identified by this process was considered unverified.

The Institutional Review Boards at Maine Medical Center and the University of Minnesota approved the research protocol.

Results

During the study period, 311 individuals submitted a total of 468 applications to 5 PCSM fellowship programs. Of these, 157 applicants applied to more than 1 of the participating programs (table 1). Our sample represented 39% (311 of 806) of all applicants to PCSM programs during this time period.¹⁰ Approximately half (54%, 168 of 311) of the applicants trained in family medicine, 16% (51 of 311) in pediatrics, and 11% (33 of 311) in internal medicine.

table 1 Characteristics of Applicants to Sports Medicine Fellowships (N = 311)

View Fullscreen

Publications

Of the 311 applicants, 150 (48%) included claimed publications, totaling 405 publications. Among the 405 publications claimed, 75% (303) included a cover sheet, 19% (76) were verified using PubMed and/or Google Scholar, 1.2% (5) by a medical librarian, and 0.7% (3) through publisher inquiry, with 4.4% (18) unverifiable.

Overall, 15 applicants claimed at least 1 unverifiable publication. Three applications contained more than 1 unverifiable publication. The total unverifiable rate for all applicants was 4.8% (15 of 311). For applicants with claimed publications, the unverifiable rate was 10% (15 of 150). The rate increased with number of claimed publications: 5.6% (4 of 71) for 1, 8.6% (3 of 35) for 2, and 18% (8 of 44) for those claiming 3 or more publications (table 2).

table 2 Misrepresentation Rates Among Applicants (N = 311) to Sports Medicine Fellowships

View Fullscreen

Discussion

For 3 years of sports medicine fellowship applications, 75% of claimed publications and 43% of claimed presentations included proof via the required cover sheet. After a thorough verification process, 15 applicants had at least 1 unverifiable publication, which was 4.8% of all applicants and 10% of those who reported a publication. Similarly, 9 applicants had at least 1 unverifiable presentation, which was 2.9% of applicants and 11% of those reporting a presentation.

Unverified citations are a persistent problem among medical graduate training programs, and published rates vary widely among specialties, for example, 14% in plastic surgery,⁵ 18% in pathology,⁷ and 21% in orthopaedic¹¹ residency applicants. Lower rates have been observed in urology (4.5%)⁶ and ophthalmology (1.9%).⁸ Of these, ours is the only study that included data from multiple institutions. A recent meta-analysis of 13 studies of graduate medical training programs found that the “misrepresentation” rate was 4.9% overall, similar to the 4.8% rate we observed, and 15.9% among those reporting any published articles, somewhat higher than the current study's rate of 10%.⁹ Recent studies^5–8 have shown that rates have not changed dramatically from those published a decade earlier. In fact, a recent study of orthopaedic residency applicants noted an increase in unverifiable citations from 18% in 1999 to 21% in 2007.¹¹ While our results are not directly comparable to our 2008 analysis of a single institution before the change in the application requirement, it is worthwhile to note that the current unverifiable rate is lower than we previously observed (11% previously versus 4.8% in current study for publications).

Our study took place after the sports medicine application process began requiring proof of all publications and presentations. This change has been suggested by previous scholars,^6,9 although no standardization for monitoring claimed research citations currently exists. While the Electronic Residency Application Service now requires providing PubMed ID numbers, not all medical specialty programs participate in this system. We found that a substantial number of applicants failed to submit proof of publication or presentation even when it was required.

Although our study only included 5 of 144 sports medicine programs, we attempted to survey a representative portion of the total fellowship applicants by reviewing sites across several geographic regions, and we captured 39% of total applicants to PCSM programs during this time period.¹⁰ A potential limitation of our methodology is the possibility that we may have mislabeled a true citation as “unverifiable,” yet this is unlikely given the thorough search methods, including contacting publishers and sponsoring institutions before labeling a citation unverifiable.

It is important to note that it is unknown how many unverifiable citations were intentional misrepresentations. Some applicants may truly falsify their curriculum vitae, but others may honestly misidentify or mislabel their work. Application errors may be lessened by national guidelines, such as those adopted in 2009 by the American Medical Society for Sports Medicine, or electronic systems designed to easily verify citations. Further research should specifically examine unverifiable rates before and after such systems are implemented.

Conclusion

Following the requirement to provide proof of publications and presentations, this study of 5 PCSM fellowship programs, which included 39% of applicants to all PCSM programs, found that 4.8% of all applicants and 10% of those who reported a publication had at least 1 unverifiable publication. Similarly, 2.9% of applicants and 11% of those of those reported a presentation had at least 1 unverifiable presentation.