Introduction

Surgical residency programs have long sought objective measures of determining applicants' long-term success, given the limited training positions and significant time and money expended in their training. Current data to evaluate and rank applicants focus on academic and standardized test performance, letters of recommendation, honor society membership, and research experience. Spatial and manual skills currently are not assessed as part of the application process.

We hypothesized that dexterity and visual spatial testing of applicants for general surgery and otolaryngology residency provides information that is not assessed through the current process, and that these assessments would not correlate with the variables traditionally used to rank applicants. Additionally, we wanted to assess whether these tests could be completed during a single scheduled interview day without significant disruption to the interview structure.

Methods

Medical student applicants to our institution's general surgery and otolaryngology residency programs were included in the study. Applicants interviewed were given the option to participate in the study or to decline but still undergo testing to blind faculty to an individual's participation status.

Following collection of demographic data and information on surgical experience, applicants underwent a battery of 5 tests shown in published studies to be related to surgical resident applicants' dexterity or to other measures of their success. We added the series of 5 tests to the interview day, using the time before and between interviews for testing. This required 3 additional staff members during a 4-hour interview period to conduct testing on as many as 8 applicants per session. Evaluation of video-recorded microscope testing required an additional 20 minutes per student. Each test is described separately below.

The Purdue Pegboard Test (Lafayette Instrument) is a commercially available validated evaluation of gross and fine finger dexterity.¹ Participants used their dominant, nondominant, and both hands to place pegs into a board within 30 seconds. Scores were collected for total number of pegs placed.

The Rey-Osterrieth Complex Figure drawing is a widely used test of visual spatial construction and executive planning functions.^1,2 Applicants first copy the complex figure, then immediately reproduce the figure from memory, followed by another reproduction 20 minutes later. Scores are based on position and accuracy of 18 drawing subunits.

Applicants were given 2 endoscopic skills modules (3 and 6) from the Lap Mentor (Simbionix) trainer in Practice Hall mode.^1,3 The modules were abstract task representations selected for their subjective ease of instruction and their 2-handed requirement. The first module timed participants touching spheres with the matching color right- or left-hand instrument. The second test consisted of the concurrent use of 2 Maryland graspers to push a tactically stimulating “blob” of tissue off a colored sphere in order to grasp the sphere with the other hand. Each module was completed 3 times, with efficiency of movement scores (as determined by the computer program) and time to complete the modules averaged over the trials.

The laparoscopic peg transfer task of the Fundamentals of Laparoscopic Surgery program was selected for its standardized implementation,⁴ and for the extensive testing ensuring its validity for use in teaching and assessment. Applicants were given 3 attempts with a maximum time of 4 minutes each. Their average time was used for comparison.

The final test was a microvascular knot-tying station based on the protocol described by Carlson et al⁵ as a “go, no-go” evaluation used at their institution. Our modified setup used video-recording equipment and 2 independent reviewers blinded to participants' identities. A Penrose drain with a midline slit was anchored to a stable base under a microscope. After a standard orientation and familiarization period, applicants tied as many knots as possible in 10 minutes. They were video-recorded in 2 views: (1) a microscope working view, and (2) a distant view behind the participant's back, observing body position. Grading was done by 2 independent reviewers (a senior surgeon and a senior resident with microvascular training) using a standard form for grading effective microscope use, tissue handling, suture technique, body position, and evidence of concentration or frustration. The total number of successful knots was recorded.

During the US Military Graduate Medical Education Selection Board, an objective, numerical score is assigned to all applications by nonaffiliated reviewers. For postgraduate year 1 selections, applicants' Electronic Residency Application Service composite scores are based on standardized board scores (United States Medical Licensing Examination or Comprehensive Osteopathic Medical Licensing Examination of the United States), medical school grade point average, interview scores, research experience, prior military and medical experience, letters of recommendation, and program director's assessment.

The study was approved by the Naval Medical Center Portsmouth Institutional Review Board.

Program directors were blinded to the test results and to the applicant's status as a study participant. Following the board's selection of applicants to the graduate medical education program, study data were analyzed using the Pearson product moment correlation coefficient (Pearson r) and regression analysis with the enter method to determine if there was a correlation between US Military Graduate Medical Education Selection Board scoring and information obtained in the testing. The Pearson r is a standard means to assess association between 2 interval variables.

Results

Data were collected from August to October 2012. A total of 64 applicants participated in the testing battery. Participants who withdrew their applications were not included in the study. This resulted in 58 applicants who completed the testing battery and were included in the study.

Regression analysis using the enter method was performed for each test and composite score. The table details the P value for each test method as well as its Pearson r coefficient. None of the values were significant as defined by P ≤ .05. Neither the Pearson r nor the data scatterplots showed a linear correlation between the participants' scores using the current academic factors and the scores from any of the surgical dexterity tests. Because there was no significant association between any of the tests and the current assessment score, we concluded that the addition of this testing battery provides information to the selection board not currently available in the application process.

TABLE Linear Regression and Pearson r Coefficient for Test Conditions Versus Current Objective Scoring Methods^a

View Fullscreen

Discussion

In our study of applicants for 2 military surgical residency programs, visual spatial testing scores and psychomotor aptitude did not correlate with variables traditionally considered in the applicant selection process. In addition, with appropriate coordination, we show that surgical dexterity testing can be completed during a standard interview day.

Including objective assessments of applicants' baseline dexterity in the selection process for surgical specialties has long been a challenge. Dirschl et al³ described an attempt to use the existing application data to determine dexterity. They identified potential predictors of manual skills based on students' lists of hobbies and interests, then tested students' psychomotor skills, but they did not find a correlation. Other investigators also have sought to prospectively and retrospectively identify—with varying degrees of success—additional surrogate indicators of surgical adeptness during the application process. For example, Carlson and colleagues⁵ demonstrated a significantly poor correlation between otolaryngology applicants' United States Medical Licensing Examination scores and dexterity scores on a microvascular suturing module.

The adoption of a surgical skills test as an Accreditation Council for Graduate Medical Education requirement for graduates of general surgery residencies and the renewed focus on evaluating core competencies in an objective manner make the acquisition of surgical skills during the residency period an imperative. Implicit in this is the role of innate ability on the final skill level attained during training. A systematic review of 27 articles by Maan and colleagues⁶ concluded that intermediate- and high-level visual spatial perception could be correlated with operative ability by the end of training. They also concluded that collectively assessed psychomotor aptitude correlates with the rate of skills acquisition.⁶ In an era of duty hour restrictions and increasing complexity of surgical techniques, baseline dexterity testing may become a more important factor in admission to surgical specialties. More recently, Buckley et al⁷ established that applicants with lower baseline manual dexterity consume a larger period of finite training time to achieve surgical proficiency, and that nearly half of all surgical residents may not possess the necessary comfort in their skill set by the conclusion of their training.

We assured applicants during the consent process that the results of the testing would not have an impact on their residency application score. This made for a low-stakes environment, which does not reflect the intended implementation of this protocol. Arora et al⁸ have shown that stress has a detrimental impact on novice participants in a surgical simulator.

There are limitations to our study. It was not designed to answer the larger question about whether obtaining this additional information would lead to improved performance and surgical skills in residents. Future research should include longitudinal results of these assessments.

Conclusion

The addition of 1 or more evaluations of visual spatial perceptive skills and psychomotor aptitude may provide an additional component of evaluation for medical student applicants that is not considered in the current application process. Such additional testing can be conducted within the constraints of a standard residency interview day.