Introduction

The objective structured clinical examination (OSCE) is widely used for assessing communication and interpersonal skills in undergraduate and graduate medical education.^1,2 OSCEs are used to provide feedback to residents and medical students following standardized patient (SP) encounters.^3,4 Resident self-ratings with the Kalamazoo Essential Elements Communication Checklist–Adapted (KEECC-A)⁵ are used at Wayne State University School of Medicine, Detroit, Michigan, as part of the OSCE to promote resident self-reflection.

Self-ratings are important because they require self-reflection and self-monitoring, which are essential for lifelong learning and improvement. Yet, few studies have included resident self-ratings,^6,7 and only 1 has incorporated resident self-ratings using the KEECC-A. Joyce et al⁵ compared faculty, SP, and resident self-ratings, and reported that only the correlation between faculty and SP ratings was statistically significant (r = 0.31, P < .001); resident self-ratings were not significantly correlated with faculty ratings (r = 0.09, P > .05) or SP ratings (r = 0.12, P > .05). No KEECC-A training of residents was offered prior to the OSCE.

The goal of this study was to determine whether residents with a modicum of training with the KEECC-A would accurately self-rate their own communication skills during an OSCE. We compared the scores from the self-ratings with ratings by trained objective raters (ie, individuals not involved in the clinical encounter itself). Given the extensive KEECC-A training that SPs received, we expected a relationship between the SPs' scores and those of the trained observers, and a lower correlation between resident self-ratings and the ratings provided by the observers and the SPs.

Methods

Results

Intercorrelations of the KEECC-A total scores between the 3 raters are reported in the table. The internal consistency for observers (coefficient α) with the KEECC-A was 0.93 and interrater agreement (intraclass correlation coefficient) with Spearman-Brown correction for double coding ranged from 0.63 to 0.82, with a good overall reliability of 0.74.⁹

TABLE KEECC-A Total Score Means, SDs, Ranges, and Correlations of Communication and Interpersonal Skills by Resident, Standardized Patient, and Observer

View Fullscreen

The magnitude of the relationship between observers and SPs exceeded a medium effect size, while the magnitude of the relationship between residents and the more objective raters reached a small and small-to-medium effect size, respectively. The results suggest that the relationship between the more objective raters was stronger than either rating with resident ratings. Results of Fisher (1-tail) r-to-z comparisons failed to show statistically significant differences between any of the correlations: observer/SP versus resident/SP (z = 1.19, P = .12) or observer/SP versus resident/observer (z = 0.88; P = .19). Resident self-ratings significantly correlated with observer (P = .004) and SP (P = .02) ratings, suggesting that residents can accurately rate their own communication skills.

Discussion

With a modest amount of training, residents were able to provide ratings of their communication skills that were consistent with those from SPs and trained observers. This is the first project to include independent observers as raters using the KEECC-A with an OSCE. Observers provided a unique opportunity to evaluate the communication scoring of resident self-ratings and SPs, as they are naturally less biased. In contrast to the faculty observers in the study by Joyce et al,⁵ the observers in this study received extensive training and achieved a good level of interrater reliability. Nonfaculty observers also may have more time available to learn the coding systems and participate in the process, as they may have lower clinical and educational demands than physician faculty.

Although ratings in the current study and in the study by Joyce et al⁵ between SPs and observers/faculty both exceeded a medium effect size, differences were found between resident self-ratings and ratings by both observers and SPs. In the current study, both correlations were statistically significant and within a small-to-medium effect size, while both correlations in the study by Joyce et al⁵ were nonsignificant and of a small effect size. These results indicate that the self-ratings in the current study were more robust than those in previous research when residents did not receive prior training.

This study also adds to the literature on the assessment of communication skills by using observer ratings as a criterion variable. Although the observer ratings should not be considered the “gold standard” for ratings of interpersonal and communication skills, they provide a more objective rating than ratings by SPs or clinical faculty raters. Thus, the findings also lend support for the reliability and validity of the KEECC-A rating scale for use in applied OSCE settings.

The study has several limitations. It was conducted at a single institution, reducing the generalizability of the findings. Another limitation is the lack of a control group. Future research should examine how varying levels of training affect the accuracy of resident self-ratings of interpersonal communication.

Conclusion

We demonstrated that residents with a modicum of training using the KEECC-A can accurately self-rate their own communication and interpersonal skills during an OSCE.