Introduction

Direct clinical observation is an essential component of assessment in residency education, codified in the United States in the Accreditation Council for Graduate Medical Education (ACGME) Milestones.^1,2 Available data suggested that episodes of direct observation can contribute to clinical development and improve the accuracy of trainee evaluation, yet direct observation occurs less frequently than recommended or desired due, in large part, to faculty time constraints and a lack of structure to consistently accommodate it.^3–5

The clinical coaching paradigm, a training strategy based on direct observation with attention to discrete skills development, is an increasingly popular model in medical education. Coaching places an emphasis on reflection. Learners are encouraged to think about what they did well and how they might care for their patients more effectively in the future.^6,7 An important goal of coaching is to help learners achieve deliberate practice, usually through rigorous self-assessment.⁸ While clinical coaching generally involves a longitudinal relationship with a learner and serial observations over time, it can also occur in isolated episodes.^9,10 In contrast to teaching, where competence is the goal, the objective for coaching is excellence.

Procedural specialties have embraced the coaching model in recent years, demonstrating acceptability to learners and an impact in improving surgical techniques.¹¹ Information on clinical coaching in other areas of medicine is sparse. Ambulatory practice is a crucial aspect of internal medicine residency training in which direct observation is often limited because of encounters occurring behind closed doors with several residents supervised by 1 attending physician. Data suggested that structured episodes of direct observation in ambulatory settings might be valuable for educational assessment and resident skills development,^12,13 although no published studies, to our knowledge, have examined the benefits of adding structured coaching in this clinical setting.

We hypothesized that direct observation and coaching by experienced preceptors would identify important deficiencies in clinical practice unrecognized by residents and that coaching would help residents create discrete performance goals. We sought to evaluate the differences between resident self-assessment and faculty observation and to assess the impact of coaching feedback on goal setting and achievement in an academic general internal medicine (GIM) practice.

Methods

Coaching Intervention

Residents were introduced to the intervention in advance and instructed that the coaching intervention was designed as a quality improvement initiative to optimize their growth in the ambulatory setting. Learners understood that the assessment was intended to be formative.

Two GIM faculty preceptors, with 19 years of combined experience, conducted full-length direct observations of clinical encounters once with each resident in the program. Both coaches participated in the intervention design and completed a 9-month teaching skills course prior to the intervention.¹⁵ Coaching faculty had longitudinal relationships with many residents.

Coaching sessions were structured to improve on existing strategies for direct observation in several ways.¹⁶ The workflow for the coaching episode is shown in figure 1. Faculty evaluated trainees using a 30-item checklist of behaviors based on the mini-CEX and published tenets of clinical excellence, which had previously been developed for coaching of outpatient attending physicians.¹⁷ The checklist was developed with consideration of 6 domains of clinical excellence relevant to the ambulatory setting: (1) professionalism and humanism; (2) communication and interpersonal skills; (3) use of the electronic health record (EHR); (4) diagnostic acumen; (5) skillful negotiation of the health care system; and (6) medical knowledge. Pilot testing was conducted in which coaches watched several faculty members caring for patients to optimize ease of use, enhance clarity, and ensure that agreement was acceptable.

View Figure

• Download as Powerpoint
• Download Figure

Immediately after the encounter, residents were given a copy of the checklist to complete as a self-assessment. Next, a 15- to 30-minute focused coaching feedback session was led by the faculty coach. Together, the pair reviewed the 2 copies of the completed checklist to identify areas of disagreement in assessment. The coaching feedback discussion was informed by specific, concrete examples of resident behavior and language during the encounter. Particular emphasis was given to items in which resident self-assessment differed from faculty objective assessment—blind spots. Blind spots were presented to residents as (1) “surprising good news,” when coaches observed residents execute checklist behaviors the residents self-reported as not performed; and (2) “surprising bad news,” when coaches did not observe behaviors the resident reported they had completed.

The study was approved by the Johns Hopkins School of Medicine Institutional Review Board.

Results

Forty-six of 48 residents (96%) were directly observed during the study period (54% [n = 25] were men, 28% [n = 13] were in the primary care residency track, 35% [n = 16] were interns, 33% [n = 15] were postgraduate year 2 [PGY-2], and 33% [n = 15] were PGY-3).

Behaviors: Observed and Self-Assessed

The most common items in which directly observed behaviors by coaches differed from resident self-assessment (blind spots) are shown in box 1. Among the 208 blind spots identified, “surprising good news” (n = 110, 53%) was slightly more common than “surprising bad news” (n = 98, 47%). More detailed results are shown in figure 2.

View Figure

• Download as Powerpoint
• Download Figure

The proportions of checklist item behaviors completed (as noted in the direct observation) and accurately self-assessed by the residents are shown in the table. The least commonly performed behaviors entailed acknowledging the role of the computer (34%, 15 of 44), collaborating with patients when using the EHR (48%, 21 of 44), hand washing (54%, 25 of 46), assessing understanding (teach back; 51%, 22 of 43), and using appropriate physical examination techniques (64%, 28 of 44).

Table Individual Behavior Completion Rates and Accuracy of Self-Report

View Fullscreen

There were no differences by sex for behaviors (women, 87% [479 of 549 items completed]; men, 85% [562 of 662]) or for blind spots in self-assessment (women, 24% [133 of 549]; men, 23% [154 of 662]).

Discussion

Our direct observation of internal medicine residents in the outpatient setting by experienced coaches using a behavioral checklist found differences between resident-reported and faculty-observed behaviors. This prompted formative coaching by faculty and goal setting by residents, with some residents reporting continued focus on those goals 1 month later. The coaching sessions were acceptable to residents.

The most robust literature for clinical coaching exists for surgery, where incorporating direct observation and focused feedback on procedural skills were shown to improve technical skills.¹¹ Previous studies examining direct observation in the outpatient setting have shown acceptability among faculty and changes in practice.^12,13 A longitudinal coaching program in pediatrics was associated with an increase in integration of resident self-assessment into feedback sessions and improved resident confidence in faculty feedback,¹⁸ and other models of direct observation, such as videotaping, have shown promise in improving the quality of feedback.¹⁹

We found differences between resident self-evaluation and faculty-observed performance, consistent with previous literature showing poor rates of physician self-assessment of clinical skills.^20–23 Integration of the computer and/or EHR into the visit was a commonly omitted and inaccurately self-assessed behavior. This was not surprising given the documented concerns about the effect of EHRs on patient-physician interactions and the lack of standards for resident training and evaluation in using the EHR during patient visits.^24,25

Resident-identified goals for improvement following coaching interventions lined up well with the least frequently performed behaviors and blind spots identified in resident self-assessments, suggesting that incorporating resident self-assessment into coaching sessions helped inform the goal-setting process. This may be important, as goal setting and self-assessment of clinical performance are components of professional development.^26,27

This study has several limitations. The behavioral checklist was developed based on previously identified tenets of clinical excellence and used by faculty members involved in its development. Interrater agreement when used by other faculty may not be as high. Residents were not trained to use the checklist and may not have interpreted items similarly to faculty. We relied on self-reporting of changes in practice, which may be subject to social desirability bias.²⁸ Our prospective cohort study lacked a control group, and we did not assess goal-setting in residents who did not receive a coaching intervention.

Additional study is needed to further elucidate the utility and broader applicability of direct observation with coaching. Next steps could include introducing serial resident-coaching observations by the same coach to evaluate the effect of longitudinal coaching on clinical performance and expanding checklist use to other faculty preceptors to examine interrater agreement and to determine the need for training in tool use to guide coaching interactions.

Conclusion

Our study demonstrated that an outpatient direct observation and coaching intervention illuminated significant blind spots in resident clinical self-assessment and was well received by residents.