Association Between Performance on COMLEX-USA and the American College of Osteopathic Family Physicians In-Service Examination
ABSTRACT
Background
The primary goal of residency programs is to select and educate qualified candidates to become competent physicians. Program directors often use performance on licensure examinations to evaluate the ability of candidates during the resident application process. The American College of Osteopathic Family Physicians (ACOFP) administers an in-service examination (ISE) to residents annually. There are few prior studies of the relationship between the Comprehensive Osteopathic Medical Licensing Examination of the United States of America (COMLEX-USA) series and formative assessments of residents in training.
Objective
We explored the relationship between performance on COMLEX-USA and the ACOFP in-service examination to offer support on the use of licensing examinations in resident selection.
Methods
In 2016, performance data from the COMLEX-USA and the ISE were matched for 3 resident cohorts (2011–2013, inclusive; N = 1384). Correlations were calculated to examine the relationship between COMLEX-USA and ISE scores. Multiple linear regression models were used to determine if performance on COMLEX-USA significantly predicted third-year ISE (ISE-3) scores.
Results
Findings indicated that correlations among performance on COMLEX-USA and ISE were statistically significant (all P < .001), and there was strong intercorrelation between COMLEX-USA Level 3 and ISE-1 performance (r = 0.57, P < .001). Performance on the COMLEX-USA Levels 1 and 2–Cognitive Examination significantly predicted performance on the ISE-3 (F(2,1381) = 228.8, P < .001).
Conclusions
The results support using COMLEX-USA as a part of resident selection in family medicine. Additionally, program directors may use performance on COMLEX-USA to predict success on the ISE-3.
Introduction
The Comprehensive Osteopathic Medical Licensing Examination of the United States of America (COMLEX-USA) is the national series of licensure assessments for osteopathic physicians. The COMLEX-USA series assesses osteopathic medical knowledge and clinical skills for the practice of osteopathic medicine.1 It consists of 4 examinations in the COMLEX-USA series: Level 1, Level 2–Cognitive Evaluation (CE), Level 2–Performance Evaluation (PE), and Level 3. Levels 1, 2-CE, and 3 are predominantly multiple-choice examinations, and Level 2-PE is a standardized patient-based examination.1 All levels follow the same blueprint: they are designed to measure competency across the same 9 domains of patient presentations and 6 domains of physician tasks.1 Levels 1, 2-CE, and 2-PE scores are widely used by osteopathic medical schools for graduation purposes, and by programs for resident selection, as scores provide reliable evidence of residents' medical knowledge and application. Numerical scores on Level 2-PE are not reported; therefore, the predictive validity of these scores is not applicable to the current study.
Consistent with the mission of the American College of Osteopathic Family Physicians (ACOFP) “to promote excellence in osteopathic family medicine through quality education, visionary leadership, and responsible advocacy,”2 it partners with the NBOME to offer an annual in-service examination (ISE) to evaluate the progress of each resident and provide feedback for continuous professional development. The NBOME develops, processes, and scores the ISE as a service to ACOFP. Scores on the ISE are used by residency program faculty to determine areas of academic strengths and deficiencies for individual residents and to plot a curriculum to ensure each resident meets the minimum osteopathic medical knowledge benchmarks and milestones in the program.
In evaluating the relationship between performance on COMLEX-USA and ISE, the timing of both examinations is a factor. Medical students take levels 1, 2-CE, and 2-PE during their second, third, and fourth years of osteopathic medical school. COMLEX-USA Level 3 is typically administered at the end of the first osteopathic residency year (OGME-1) or the beginning of the second year (OGME-2). It is required for progression to the third year of residency (OGME-3). The figure displays the sequence of COMLEX-USA and ISE administration.
Citation: Journal of Graduate Medical Education 10, 5; 10.4300/JGME-D-17-00997.1
Following best practices in assessment, it is important to evaluate and provide empirical evidence supporting the relationship between performance on examinations that intend to measure similar constructs.3 A meta-analysis conducted on the relationship between performance on the United States Medical Licensing Examination (USMLE) and ISE showed there was a strong positive correlation.4 Findings from research on the association between performance on the USMLE and the internal medicine ISE also have shown significant, positive correlations.5 Yet, few studies have investigated the relationship between performance on the COMLEX-USA and specialty ISEs. One study on the relationship in performance on the COMLEX-USA, American Board of Family Medicine (ABFM) ISE, and ABFM certification examination showed positive, significant associations between performance on COMLEX-USA levels 1, 2-CE, and 3; ABFP ISE; and ABFP certification examination.6 Our study builds on existing research by examining the association between COMLEX-USA scores and the ISEs administered in family medicine osteopathic residency programs. We sought to evaluate the predictive validity of COMLEX-USA scores on performance on ACOFP ISE, focusing on the following research questions:
-
What is the relationship between performance on COMLEX-USA levels 1, 2-CE, and 3 and performance on the ISE-1, -2, and -3?
-
How well does performance on COMLEX-USA levels 1 and 2-CE predict ISE-3 performance?
Methods
Participants
In 2016, we compiled scores from the ACOFP ISE for the 2011, 2012, and 2013 entering resident cohorts. The completion years of these cohorts were 2014, 2015, and 2016, respectively. We were able to obtain ACOFP data for 1720 residents who were administered the ISE each program year (OGME-1, OGME-2, and OGME-3) between 2011 and 2015.
Levels 1, 2-CE, and 3 COMLEX-USA data between 1990 and 2015 were used to create a longitudinal review of resident performance. COMLEX-USA data were matched to ACOFP data by residents' first and last names. Residents were then matched based on their first year in an ACOFP-affiliated residency program and their osteopathic medical school graduation year. Residents with 1 or more gap years between graduation and residency were excluded from this study due to the matching criteria.
Outcomes
COMLEX-USA levels 1, 2-CE, and 3 include 350 to 400 multiple-choice questions. Cronbach's alpha measure for reliability (internal consistency) is approximately 0.90 for each of the 3 cognitive examinations in the COMLEX-USA series.7 Although there are small differences in the number of scored items across years, the ACOFP ISE contains approximately 200 scored multiple-choice questions. The reliability over the indicated time frame ranged from 0.81 to 0.86.8–10 The standard scores for all examinations were used for analyses.
The study was declared exempt by the Institutional Review Board of the Center for the Advancement of Healthcare Education and Delivery.
Analyses
Descriptive statistics, including frequencies, means, and standard deviations, were used to describe the sample with respect to performance on COMLEX-USA and ISE. One-way analysis of variance (ANOVA) was used to identify statistically significant differences among the means of the 3 cohorts' COMLEX and ISE scores. The Pearson product-moment correlation was used to measure the strength of the linear relationship among performance on levels 1, 2-CE, 3, and ISE-1, -2, and -3. We used Cohen's effect size to interpret effect sizes to measure strength of associations (eg, 0.10 for small effect, 0.3 for medium effect, and 0.5 for large effect),11 and applied the Bonferroni correction for multiple comparisons.12 Predictive validity evidence was evaluated using a multiple linear regression model of scores on levels 1 and 2-CE onto ISE-3. Levels 1 and 2-CE are often used in residency selection.13 Therefore, we used residents' ISE-3 scores as a proxy for successful completion of the residency program. All analyses were done in R version 2.1 (R Foundation for Statistical Computing, Vienna, Austria). We employed a significance level of .05, unless otherwise specified.
Results
Of the original 1720 residents in the sample, 1384 (80%) residents were successfully matched with COMLEX-USA performance across the 3 cohorts. Six residents (0.3%) were excluded from the analysis due to duplicated records of name and graduation year. One hundred thirty-nine residents who did not begin residency immediately following graduation were excluded from the analysis. Results from independent sample t tests indicated these residents scored significantly different on the COMLEX-USA and ISE (P < .001; see online supplemental material).
The match rate by cohort ranged from 76% to 84%. Table 1 displays the mean and standard deviations of COMLEX-USA standard scores and ISE standard scores by cohort. One-way ANOVA results indicated no significant differences among COMLEX-USA or ISE scores across the 3 cohorts (P > .05; see table 2), which provide rationale for analyzing all cohorts together.
Association of Performance on COMLEX-USA and ISE
The results for our examination of the relationship among COMLEX-USA scores and ISE scores indicated a significant correlation among performance on COMLEX-USA levels 1, 2-CE, and 3 and performance on all levels of the ISE (table 3).
The strongest correlations were found within each examination group. For example, the strongest associations are between performance on the ISE-2 and ISE-3 (r = 0.70, P < .001) and ISE-1 and ISE-2 (r = 0.65, P < .001). Similarly, there was a significant strong correlation between COMLEX-USA Level 1 and Level 2-CE (r = 0.64, P < .001). Correlations greater than 0.5 indicate a large effect.10
Across examinations, the strongest intercorrelation was found between performance on ISE-1 and COMLEX-USA Level 3 (r = 0.57, P < .001). This is likely due in part to the similar timing of these 2 examinations, as more than 95% (1326 of 1384) of residents were administered COMLEX-USA Level 3 after the ISE-1. The strong intercorrelations may also be related to the similarity in test blueprints for the 2 examinations.
The intercorrelations indicated that performance on levels 1, 2-CE, and 3 accounted for 18%, 22%, and 27% of the variance on ISE-3 scores, respectively. In addition, first- and second-year performance on ISE accounted for 35% and 49% of the variance in ISE-3 scores, respectively. While COMLEX-USA Level 1 and ISE-3 have the weakest relationship, that relationship is significant (r = 0.42, P < .001).
Predictive Validity of COMLEX-USA on ISE-3 Scores
Results from a multiple linear regression indicated that performance on COMLEX-USA levels 1 and 2-CE significantly predicted ISE-3 scores (F(2,1381) = 228.8, P < .001). Performance on levels 1 and 2-CE accounted for 25% of the variance in ISE-3 scores. Table 4 displays the results of the multiple regression analysis, including the coefficient (weight) of each variable, the standard error, and the significance associated with the t-statistic.
Discussion
Our findings support the utility of COMLEX-USA scores as a part of the process in selecting qualified residents for matriculation into postgraduate residency programs. The use of ISE-3 as a proxy for successful completion of residency programs was based on prior work showing that residents who earn high scores on ISE-3 are considered to have the requisite knowledge and skills to be osteopathic family physicians. Examining thresholds of resident knowledge is a plausible topic for future research.
The significant correlations among ISE scores were expected due to the consistent test blueprint. Additionally, results showed a significant strong correlation between ISE-1 and Level 3 scores, which may partially be explained by closeness in timing between the examinations. The significant correlations provide evidence of the congruent validity of the COMLEX-USA series and the ISE. It is important to review associations between examinations that purport to measure similar constructs as this evidence supports the congruent validity of both assessment programs.
Limitations of our study include that it was restricted to family medicine residents who were administered the examination for 3 consecutive years from 2011 through 2015. Research with different groups of residents, date ranges, and specialties is warranted to validate the consistency of results. Study of residents who did not begin their residency program immediately after graduation and residents who did not take the ISE for 3 consecutive years also is of interest. Finally, evaluating the relationship between performance on COMLEX-USA, ISE, and board certification and recertification examinations as well as quality indicators in practice is also warranted.
The results from the multiple linear regression support the predictive validity of COMLEX-USA on successful completion of ISE-3 (measured by ISE-3 performance). If program directors view scores on ISE-3 as a proxy for successful completion in the residency program, the findings of this research support the utility of COMLEX-USA levels 1 and 2-CE scores as a key parameter for selecting qualified residents for osteopathic family medicine residency programs. Despite the evidence to support the secondary use of scores on licensure examinations for predicting successful performance in residency programs, the National Board of Osteopathic Medical Examiners advocates for holistic review of residency applicant qualifications.
Conclusion
The evidence presented in this study supports the concurrent and predictive validity of performance on COMLEX-USA and performance on the ACOFP ISE. Performance on these 2 examination series is significantly correlated, and can be used to glean information about applicants to osteopathic residency programs. This study contributes to the evidence that supports the use of COMLEX-USA as part of holistic application review for residents entering into osteopathic family medicine residency programs.
Sequence of Examination Administration of COMLEX-USA Series and ACOFP In-Service Examinationa
a M1 denotes the first year in osteopathic medical school, and OGME-1 denotes the first year in an ACOFP-affiliated residency program. ISE-1 represents the examination administered during the first year of residency.
Abbreviations: COMLEX-USA, Comprehensive Osteopathic Medical Licensing Examination of the United States of America; ACOFP, American College of Osteopathic Family Physicians; ISE, in-service examination.
Author Notes
Funding: The authors report no external funding for this study.
Conflict of interest: The authors declare they have no competing interests.
The authors would like to thank NBOME colleagues, Edward Tsai, PhD, and Sandra Waters, MEM, and American College of Osteopathic Family Physicians colleagues, Bina Mehta and Judy Schwellenbach, for their support in this research study.
