Editor's Note: The online version of this article contains the surveys used in the study.

Introduction

The Accreditation Council for Graduate Medical Education defines milestones as “competency-based developmental outcomes … that can be demonstrated progressively by residents and fellows from the beginning of their education through graduation to the unsupervised practice of their specialties.”¹ Direct observation of trainees during clinical encounters is essential to assess their development toward competence. Research supports the feasibility of direct observation and its role in the provision of more formative, substantive feedback to trainees.^2,3

Medical educators use direct clinical observation to ensure that trainees learn to provide high-quality and safe care, yet there is a dearth of published materials regarding the patient perspective of direct observation,⁴ and nothing has been written from the perspective of the adolescent patient. Patient centeredness, defined as “respectful of and responsive to individual patient preferences, needs, and values,” is a core element of high-quality health care.⁵ As educators in adolescent medicine, we are sensitive to the confidentiality concerns and emerging independence needs of adolescents and young adults, and we worried that direct observation might be viewed negatively by this population. Additionally, given the comfort and facility that today's adolescents and young adults have with technology, we considered that a preference may exist for remote observation using technologies such as video, Skype, or FaceTime. We conducted a quality improvement initiative to ensure that our practice of direct clinical observation of medical trainees was patient centered.

Methods

Our quality improvement project was conducted from September 2013 through June 2014 in the Adolescent/Young Adult Medicine Clinic at Boston Children's Hospital, an outpatient clinic providing primary and specialty care to a diverse patient population, from ages 12 to 26 years.

Following the existing direct clinical observation protocol, the fellowship director observed the 7 Adolescent Medicine fellows during all patient encounters in a given clinical session. Interns in the Boston Combined Residency Program rotating through the Adolescent/Young Adult Medicine Clinic were directly observed for 1 patient encounter by the faculty rotation director or a senior medical fellow. For each encounter, the trainee introduced the observer to the patient/family before joining them during the clinical encounter. The trainee and clinical observer then stepped out of the room to discuss the case and plan, before returning to review the diagnosis and care plan. The observer provided feedback to the trainee in the form of a “debrief” following the observation.

We asked patients and their family members, if present, to complete a survey regarding their experience with direct observation. Questions included degree of comfort with the observer in the room, and whether or not they felt clinical care was affected by the presence of the observer in the room (surveys provided as online supplemental material). For the survey of trainees/observers, the questions pertained to the feedback that was received/provided following the observation compared to a precepting arrangement. Additionally, trainees were surveyed as to their sense of clinical autonomy and their interaction with the patient/family during the observation. Observers were surveyed as to the utility of direct observation for milestone assessments. All respondents were asked whether or not they would prefer the observation to be conducted through FaceTime (a nonrecorded viewing technology using an iPad) rather than using direct observation for future encounters (surveys provided as online supplemental material).

The anonymous surveys were developed by the authors and completed in a private setting immediately after the observation. Surveys were left in an envelope at the clinic's front desk and collected monthly. We did not track how many patients, family members, trainees, and observers were asked to complete the survey, and midyear, we added questions about the patients' age and sex.

Results

Overall, patient and family respondents (n = 31) reported feeling comfortable with the observer's presence in the room during the clinical encounter. The quality of care they received was deemed to be equal to or better than the care received during unobserved, precepted visits. Respondents unanimously preferred direct observation to the idea of remote observation. Regarding the proposed use of remote observation, adolescent/young adult patients wrote: “In person seems more human,” “It would be creepy [to be observed remotely],” “In person is more comfortable,” and “I felt very comfortable with another doctor in the room. I didn't even notice.”

Responses from trainees (n = 14) and observers (n = 6) showed that feedback was perceived as more constructive when provided after direct observation, compared to prior unobserved, precepted outpatient encounters. Trainees reported they were better able to reflect on their clinical practice after feedback following direct observation (score of 4.6 on a 5-point scale). Observers thought direct observation was essential to assessing trainees from a milestone perspective (score of 5 on a 5-point scale). While 36% (5 of 14) of trainees expressed feeling nervous when observed directly, 93% (13 of 14) reported that direct observation maintained a feeling of autonomous medical decision making. The majority of trainees and observers, 79% (11 of 14) and 83% (5 of 6), respectively, expressed a preference for direct observation over remote observation. Trainees wrote: “I think using technology may be more uncomfortable for the patient, and having the [observer] in the room was not distracting at all,” “Very impersonal to have an iPad,” and “I find it helpful to have the observer in the room; better to watch body language, subtleties, etc.”

Discussion

Direct observation of clinical practice is an essential assessment tool in medical education, yet we had concerns that it could negatively impact the adolescent patient's visit experience. Fortunately, patient and family respondents expressed no concerns with our current practice of direct observation, and some felt the care they received was better. We also learned that patients preferred direct observation over remote observation, despite high usage rates of technology in the population. Trainee and observer respondents also preferred direct observation.

This study has several limitations. We did not validate our data collection tools, and respondents may not have interpreted questions as intended. There is also potential for social desirability bias in our findings. Most importantly, we did not track response rates, and it is possible that our small sample contains response bias, with respondents differing in their views of direct observation from nonrespondents. Despite these shortcomings, we believe that our experience may resonate with medical educators grappling with milestone assessments of learners.

Conclusion

Direct observation is an important assessment tool in medical education, and adolescent/young adult patients and their families expressed comfort with it, and reported a preference for direct observations over the idea of remote observations using technology.