Introduction

When the Accreditation Council for Graduate Medical Education (ACGME) identified 6 core competencies to evaluate postgraduate trainees,¹ teaching institutions incorporated various methods to assess these competencies, including objective structured clinical examinations (OSCEs). In the Next Accreditation System (NAS), milestones were developed for each specialty to define learning progression from beginner to expert within the core competencies.² Few institutions have reported on how to use current assessment tools to evaluate trainees on milestones. Some individual specialties have discussed the incorporation of milestones,^3–6 but to date no reports have described developments at the institutional curriculum level.

The Henry Ford Hospital in Detroit, Michigan, developed an institutional curriculum in 2008 that included OSCEs to assess interpersonal and communication skills (ICS). The ICS-OSCEs are offered to the institution's 18 ACGME-accredited residency programs, with most postgraduate year (PGY) 1 residents participating annually. OSCEs are a recommended assessment method by the ACGME and accepted as part of the multisource assessment used for postgraduate trainees.^7,8

OSCE evaluations may be used by clinical competency committees to determine milestone progression for residents. We explored whether ICS-OSCEs could be linked to multiple specialties' ICS milestones. Our primary aim was to assess how to use existing institutional curriculum OSCEs for milestones without additional time or expenditure. This quantitative study assessed 2 questions: Do multiple specialties have ICS milestones in common that relate to our ICS-OSCEs? Do trainees' ICS-OSCE scores relate to their ICS milestone scores?

Methods

Program directors (or their appointee) from the institution's 18 ACGME-accredited residency programs were invited to form a work group to determine whether observable behaviors from the ICS-OSCEs could be mapped to ICS milestones across specialties. No fellowships were included, as the ICS-OSCEs assess beginner-level skills. Twelve programs participated: emergency medicine (EM), family medicine, general surgery, internal medicine (IM), neurological surgery (NS), obstetrics-gynecology, ophthalmology, orthopedic surgery (OS), otolaryngology, pathology, psychiatry, and the transitional year. PGY-1 residents of these 12 programs were considered eligible for the study's analyses.

Results

The table shows the varied ICS subcompetencies and beginner-level milestones determined to relate best to the physician-patient communication focus of our ICS-OSCEs. Pathology has no ICS milestones related to physician-patient communication.

table Specialty Milestones for Interpersonal and Communication Skills (ICS) Related to Physician-Patient Communication

View Fullscreen

The ICS milestone descriptors identified as similar to behaviors outlined in the ICS-OSCE scoring tool included “establishes rapport,” “engages patient,” “develops positive relationship with patient,” and “seeks and understands patient's perspective” (table). Although neurological surgery's ICS beginner-level milestones did not specify engagement of the patient in the “ICS–Relational” subcompetency, residents must be able to “describe methods to compassionately break bad news,” which the work group considered relatable to our ICS-OSCE on sharing bad news.

Of the 63 PGY-1 residents identified for the second part of the study, 18 were from EM (3 of whom were EM-IM), 37 IM, 2 NS, and 6 OS. Three residents (2 EM, 1 EM-IM) were excluded for lack of complete data. Of the 60 residents included in the quantitative comparison, all received at least a beginner-level ICS subcompetency score and ICS-OSCE scores of ≥ 3.

Discussion

Our findings suggest that ICS-OSCE scores for PGY-1 residents appear to relate to beginner-level milestones for specific ICS subcompetencies across multiple specialties. The PGY-1 residents scored at least at the beginner level for physician-patient ICS milestones, as submitted by programs, and also scored at least at a satisfactory level on physician-patient ICS-OSCEs.

A literature search did not show any institutional-level studies that incorporated the milestones into assessments.^3–6 In an overview of the milestones, Sullivan et al¹⁵ described how interpersonal and communication skills, as well as some of the other competencies, appear to be generic enough to invite multiple program interaction.

Our study of multiple specialties had an unexpected finding of variation in the number and detail of ICS subcompetencies across specialties. The ICS subcompetencies range from 1 to 4, with a wide variation in focus (eg, communication with patients and families versus health professionals versus medical records/technology communications, operating room communication, and team communication). For example, otolaryngology has 1 ICS subcompetency defined as “Interpersonal and Communication Skills,” whereas the transitional year has 4 ICS subcompetencies, each of which has a separate focus (medical records, teams, health professionals, and patients/families/public).

A limitation of our study was the small sample size of the NAS Phase 1 group. In addition, while we used standardized patient OSCE scores for comparison because they may be more accurate than faculty or resident self-assessments, these scores did not distinguish any residents who did not meet the beginner level. This finding may have surfaced because our PGY-1 residents legitimately acquired and demonstrated beginner communication skills, or the assessment may not be able to detect variations in communication skills with more specificity. Future studies may target other milestones, which might lend themselves more to assessment variability and ultimately identify residents who need additional time to meet particular milestones.

Use of the centralized graduate medical education office helped to provide structure and bring together program directors across specialties for scholarly activity. However, results from 1 institution may not be generalizable, as ICS-OSCEs vary across institutions. We tested the feasibility of mapping observable behaviors to beginner-level milestones and cannot comment on the adaptability of a similar mapping process for higher-level milestones across specialties, as expected achievement levels vary among specialties.

Conclusion

The ICS-OSCE scores for PGY-1 residents appear related to beginner-level milestones for specific physician-patient related ICS subcompetencies across multiple specialties. Our findings show that existing ICS-OSCEs can be used to assess residents' ICS performance at the institutional level without additional expense or time required for implementation.