ABSTRACT

Background

The application of crowdsourcing to surgical education is a recent phenomenon and adds to increasing demands on surgical residency training. The efficacy, range, and scope of this technology for surgical education remains incompletely defined.

Objective

A systematic review was performed using the PubMed database of English-language literature on crowdsourced evaluation of surgical technical tasks up to April 2017.

Methods

Articles were reviewed, abstracted, and analyzed, and were assessed for quality using the Medical Education Research Study Quality Instrument (MERSQI). Articles were evaluated with eligibility criteria for inclusion. Study information, performance task, subjects, evaluative standards, crowdworker compensation, time to response, and correlation between crowd and expert or standard evaluations were abstracted and analyzed.

Results

Of 63 unique publications initially identified, 13 with MERSQI scores ranging from 10 to 13 (mean = 11.85) were included in the review. Overall, crowd and expert evaluations demonstrated good to excellent correlation across a wide range of tasks (Pearson's coefficient 0.59–0.95, Cronbach's alpha 0.32–0.92), with 1 exception being a study involving medical students. There was a wide range of reported interrater variability among experts. Nonexpert evaluation was consistently quicker than expert evaluation (ranging from 4.8 to 150.9 times faster), and was more cost effective.

Conclusions

Crowdsourced feedback appears to be comparable to expert feedback and is cost effective and efficient. Further work is needed to increase consistency in expert evaluations, to explore sources of discrepant assessments between surgeons and crowds, and to identify optimal populations and novel applications for this technology.

ABSTRACT

Background

Advances in information technology have increased remote access to the electronic health record (EHR). Concurrently, standards defining appropriate resident supervision have evolved. How often and under what circumstances inpatient attending physicians remotely access the EHR for resident supervision is unknown.

Objective

We described a model of attending remote EHR use for resident supervision, and quantified the frequency and magnitude of use.

Methods

Using a mixed methods approach, general medicine inpatient attendings were surveyed and interviewed about their remote EHR use. Frequency of use and supervisory actions were quantitatively examined via survey. Transcripts from semistructured interviews were analyzed using grounded theory to identify codes and themes.

Results

A total of 83% (59 of 71) of attendings participated. Fifty-seven (97%) reported using the EHR remotely, with 54 (92%) reporting they discovered new clinical information not relayed by residents via remote EHR use. A majority (93%, 55 of 59) reported that this resulted in management changes, and 54% (32 of 59) reported making immediate changes by contacting cross-covering teams. Six major factors around remote EHR use emerged: resident, clinical, educational, personal, technical, and administrative. Attendings described resident and clinical factors as facilitating “backstage” supervision via remote EHR use.

Conclusions

In our study to assess attending remote EHR use for resident supervision, attendings reported frequent remote use with resulting supervisory actions, describing a previously uncharacterized form of “backstage” oversight supervision. Future work should explore best practices in remote EHR use to provide effective supervision and ultimately improve patient safety.

ABSTRACT

Background

In 2013, milestone ratings became a reporting requirement for emergency medicine (EM) residency programs. Programs rate each resident in the fall and spring on 23 milestone subcompetencies.

Objective

This study examined the incidence of straight line scoring (SLS) for EM Milestone ratings, defined as a resident being assessed the same score across the milestone subcompetencies.

Methods

This descriptive analysis measured the frequencies of SLS for all Accreditation Council for Graduate Medical Education (ACGME)–accredited EM programs during the 2015–2016 academic year. Outcomes were the frequency of SLS in the fall and spring milestone assessments, changes in the number of SLS reports, and reporting trends. Chi-square analysis compared nominal variables.

Results

There were 6257 residents in the fall and 6588 in the spring. Milestone scores were reported for 6173 EM residents in the fall (99% of 6257) and spring (94% of 6588). In the fall, 93% (5753 residents) did not receive SLS ratings and 420 (7%) did, with no significant difference compared with the spring (5776 [94%] versus 397 [6%]). Subgroup analysis showed higher SLS results for residents' first ratings (183 of 2136 versus 237 of 4220, P < .0001) and for their final ratings (200 of 2019 versus 197 of 4354, P < .0001). Twenty percent of programs submitted 10% or more SLS ratings, and a small percentage submitted more than 50% of ratings as SLS.

Conclusions

Most programs did not submit SLS ratings. Because of the statistical improbability of SLS, any SLS ratings reduce the validity assertions of the milestone assessments.

In 1982, John H. Stone III, MD, a cardiologist-poet, addressed the graduates of Emory University School of Medicine with these words:

“For the heart will lead

For the head will explain

but the final common pathway is the heart

whatever kingdom may come

For what matters finally is how the human spirit is spent

For this is the day of joy

For this is the morning to rejoice

For this is the beginning

Therefore, let us rejoice”

—Gaudeamus Igitur ¹

The title of this address, Gaudeamus Igitur, means in Latin, “Therefore, let us rejoice.” It is the