Introduction

Scheduling interviews during the residency selection season can be challenging and time consuming for applicants and programs. Applicants have to coordinate interviews at multiple programs, and offers may not be extended in a way conducive to easy scheduling. Additionally, applicants are often involved in active patient care during business hours, which may prevent them from calling or reaching program coordinators in a timely fashion to secure their most desired interview date.

Several scheduling systems are available to allow applicants access to online scheduling. A prior study of a single site online scheduling system showed a 77% applicant preference for scheduling online.¹ Similarly, our pilot study in 2010–2011 reported that 87.5% of emergency medicine (EM) applicants preferred an online program.²

Our primary aim was to measure EM applicant satisfaction and perceived time savings using a commercially available Internet scheduling program compared to traditional manual scheduling.

Methods

This multicenter study surveyed applicants who applied to a sample of EM residency programs during the 2014–2015 application cycle. We formed a research study group consisting of 6 EM residency programs from a broad geographic distribution (Pacific, Pacific Northwest, Midwest, South, Northeast, and mid-Atlantic). Three programs use a commercial online scheduling platform (Interview Broker [The Tenth Nerve LLC, Lewes, DE]; www.interviewbroker.com), and 3 did not use any online scheduling program but rather relied on e-mails and phone calls between applicants and programs. No participating residency program had a financial interest in the online scheduler. Each program sent an e-mail with a survey link to all applicants who applied to its residency. We used the US applicants to the 6 representative programs as a purposive sample of the national EM applicant pool. No personal applicant information was shared between programs to ensure confidentiality. While applicants may have received more than 1 link to complete the survey, the survey program prohibits more than 1 response from an applicant based on IP address. A list of unique Electronic Residency Application Service identification numbers was sent to the coordinating site to generate the list of possible respondents and eliminate redundancy. The survey was sent after Match lists were submitted but before the National Resident Matching Program rank lists were released.

The survey was previously developed for use during a pilot study² and then modified by the primary department faculty with residency leadership experience to improve content validity; such modifications involved change in format only. It was further reviewed by 6 experienced program directors from the participating sites. The revised survey was then reviewed with representative chief residents at the primary institution to optimize response process validity.³ Finally, the survey was piloted with all EM residents within the primary institution's program to further support its validity.⁴ Applicants were asked to estimate the time it took to schedule with the online system compared to other methods, and to rate their satisfaction with scheduling on a 5-point anchored scale.

The study protocol was granted expedited approved by the Institutional Review Board at the primary institution and approved at each participating institution.

Data analysis followed the assumption that questions regarding subjects' experiences with and without the software should be treated as nonindependent observations, and that the modified Likert scale should not be treated as an interval variable. We dichotomized the scale to “satisfied” and “less than satisfied,” which we analyzed with McNemar's test for nonindependent observations. The choice of these statistical tests limits the hypothesis testing in many cases to only P values.

Results

Of the 1720 unique US applicants to the 6 training programs, 856 applicants (49.8%) completed the survey (table). The sample represented approximately half of all US graduates applying to EM programs. Applicants' self-reported median time to schedule interviews using the online system was 5 minutes (interquartile range [IQR] 3–10) compared to a median of 60 minutes (IQR 15–240) for those programs not using the online system (P < .0001). In addition, 783 applicants (91.5%) who used the online platform were “satisfied” with the ease of scheduling interviews compared to 94 respondents (11.0%) who were “satisfied” with the typical system (P < .0001; figure 1).

table   Applicant Demographics

View Fullscreen

View Figure

• Download as Powerpoint
• Download Figure

Among applicants who had to change interview dates, 792 of 851 respondents (93.1%) were “satisfied” or “somewhat satisfied” in their ability to change their date online. For programs not using the online system, 243 of 847 respondents (28.7%) were “satisfied” or “somewhat satisfied” with the process, whereas the majority (604, 71.3%) described themselves as “neither satisfied or unsatisfied,” “somewhat unsatisfied,” or “unsatisfied” with the process of rescheduling (P < .0001), coded dichotomously as “less than satisfied.”

Overall, 801 of the 856 respondents (93.6%) reported they preferred the online system (figure 2). Only 12 (1.4%) preferred the nononline system, and 43 (5.0%) stated they had no preference.

View Figure

• Download as Powerpoint
• Download Figure

Discussion

The results of this multicenter study suggest that residency applicants prefer the online platform for scheduling interviews, consistent with prior smaller studies.² We hypothesize that applicants prefer online scheduling because it may conform to their schedules, and is consistent with their access to and comfort with technology. An online system allows scheduling or changing interviews at any hour. Similarly, the online system may allow applicants to rapidly change interview dates to coordinate interviews at other programs. Interestingly, respondents noted that the time to schedule an interview with the online system starting from receipt of the invitation was less than 5 minutes for half of all applicants.

An online system may substantially alter the standard mechanism for scheduling interviews. We view the high level of satisfaction as indirect evidence of the fidelity of data communication in the online system.

While we did not produce a cost benefit analysis due to the high variability of coordinator salaries across the country, it is possible that the increased satisfaction and the relatively low cost of the product (approximately $2 per interview) would result in a substantial return on investment.

During the 2014–2015 interview season, approximately 400 programs, mostly in EM and internal medicine, used Interview Broker, which initially became available in the 2010–2011 application cycle (e-mail conversation with Interview Broker Accounts Team, August 2014). It is not clear how many programs are currently using online interview tools as the number of commercially available products has increased.

Further research might include vigorous cost benefit analyses of online scheduling programs and the study of program director and program coordinator satisfaction with such programs.

Limitations of this study include a relatively low response rate (49.8%). However, the overwhelming preference by participants for the online system (93.6%) suggests it is the preferred approach for applicants. The data identifying the time spent scheduling interviews were self-reported, and may be subject to recall bias.

In addition, the coordinators at programs not using the online system were not standardized in terms of customer service training or scripting. Respondents were asked to reply based on their experiences at all programs they applied to, not just the ones participating in the study. Furthermore, we could not ensure that each respondent had both types of experiences, which could influence a respondent's ability to compare.

Conclusion

The results of this multicenter study suggest that EM residency applicants prefer an online scheduling system when compared to a nononline system. They found it beneficial when initially scheduling an interview and changing interview dates, and recalled that it saved a significant amount of time.