Assessing Residents' Veteran-Centered Care Skills in the Clinical Setting
ABSTRACT
Background
Despite their placement in Veterans Health Administration centers nationwide, residents' training and assessment in veteran-centered care is variable and often insufficient.
Objective
We assessed residents' ability to recognize and address mental health issues that affect US military veterans.
Methods
Two unannounced standardized patient (SP) cases were used to assess internal medicine residents' veteran-centered care skills from September 2014 to March 2016. Residents were assessed on 7 domains: military history taking, communication skills, assessment skills, mental health screening, triage, and professionalism, using a 36-item checklist. After each encounter, residents completed a questionnaire to assess their ability to recognize knowledge deficits. Residents' mean scores were compared across training levels, between the 2 cases, and by SP gender. We conducted analysis of variance (ANOVA) tests to analyze mean performance differences across training levels and descriptive statistics to analyze self-assessment questionnaire results.
Results
Ninety-eight residents from 2 internal medicine programs completed the encounter and 53 completed the self-assessment questionnaire. Residents performed best on professionalism (0.92 ± 0.20, percentage of the maximal score) and triage (0.87 ± 0.17), and they scored lowest on posttraumatic stress disorder (0.52 ± 0.30) and military sexual trauma (0.33 ± 0.39). Few residents reported that they sought out training to enhance their knowledge and skills in the provision of services and support to military and veteran groups beyond their core curriculum.
Conclusions
This study suggests that additional education and assessment in veteran-centered care may be needed, particularly in the areas of posttraumatic stress disorder and military sexual trauma.
Introduction
Each year, more than 50 000 residents and fellows rotate at Veterans Health Administration (VHA) locations.1 In spite of the large volume of trainees who care for veterans, the extent to which residents are trained and assessed on veteran-centered care (VCC) is unknown.
To ensure optimal health care through access to services and positive patient-provider interactions, VCC requires providers to be attentive to the exposures and implications of prior military service on patients' circumstances.2 Effective VCC is accomplished through a combination of military cultural consciousness, patient-centered communication, and keen clinical acumen. The principles of VCC are applicable not only to the 7 million veterans who access the VHA3 but also to the millions of veterans who receive care in civilian settings.
The aim of this study was to assess residents' ability to identify and respond to the specific health care needs of veterans using an unannounced standardized patient (SP). In the context of competency-based medical education, it is imperative for residents to demonstrate clinical competence in patient-centered care. Often, optimal patient care is measured by the absence of patient complaints with brief snapshots of unsupervised interactions. While direct observation remains the gold standard of assessment, it is difficult to accomplish in the clinical setting without influencing residents' behavior.4 Moreover, it is challenging for faculty to devote sufficient time and attention to reviewing and providing timely feedback on residents' performance.
Methods
Setting
This study occurred at the Veterans Affairs (VA) Ann Arbor Healthcare System (VAAAHS) and the Louis Stokes Cleveland VA Medical Center (LSCVAMC). The VAAAHS serves as a referral center for specialty care and operates 105 acute care and 40 community living center beds. The LSCVAMC primary care clinic serves approximately 12 000 veterans and was designated 1 of the original 5 Centers of Excellence in Primary Care Education in 2010. The VAAAHS and LSCVAMC care clinics facilitate the ambulatory primary care education of more than 125 and 50 internal medicine residents, respectively, from different affiliates.
Case and Instrument Development
We developed 2 SP cases: an Army veteran with combat-related posttraumatic stress disorder (PTSD) presenting with insomnia and irritability and a Navy veteran with military sexual trauma (MST) presenting with headaches and anxiety. The US Department of Veterans Affairs defines MST as sexual assault and/or repeated, threatening sexual harassment occurring during military service.5 Appropriate triage and treatment for PTSD and MST are critical due to their prevalence among veterans6–8 and because male and female9 veterans with PTSD from MST10 experience unique mental health consequences (eg, depression, behavior adjustment disorders).11
The assessment domains and case content (eg, patient characteristics and medical presentation) were developed by an expert panel of VHA-based internal medicine and psychiatry faculty, SP educators, a veteran with expertise in medical education, and staff with expertise in objective structured clinical examination assessment to ensure an accurate representation of common presentations of PTSD and MST within the clinic setting.
Data Collection
Unannounced SPs are an effective approach to assessing patient care.12 Employing actors to portray common patient presentations in the learning environment ensures appropriate clinical material, authentic clinical situations, and a low-stakes environment to develop key skills.4,12 Unlike conventional SPs, unannounced SPs assess learners' skills without their knowledge, thereby minimizing the Hawthorne effect.4 This method allows residents to receive training with limited interruption of their clinic duties.
Prior to their first encounter, the 7 unannounced SPs (4 men, 3 women) participated in over 12 hours of training to learn to portray these roles. All SPs were evaluated by 23 raters (senior medical students, residents, faculty, and SP educators) on their ability to respond to questions and their believability portraying the case.
We worked closely with program directors and coordinators at both locations to ensure the SP visits were scheduled during regular clinic hours as a new patient appointment.13 We provided residents with the same information (“door sheets”) they typically receive prior to entering the patient room (eg, vital signs, age, sex, chief complaint). Simulated patient information was used to populate the electronic health record and presented to the resident in clinic.13 Unannounced SPs were randomly assigned based on their availability and remained in the patient waiting area until they were called by the nurse, as per usual clinic practice. The study team informed the clinic staff (medical assistants, nurses, attendings, and mental health psychologists) about the study and sent e-mail reminders the day before.
All residents were informed about the study through program announcements and e-mail. From September 2014 to March 2016, unannounced SPs assessed residents' military history taking, communication skills, assessment skills, mental health screening, triage, and professionalism, using a 36-item checklist with some validity evidence.14 The checklist was scored using a 3-point scale (2, completed; 1, needs improvement; 0, not completed). These skills were mapped to the Accreditation Council for Graduate Medical Education competencies and key screening guidelines adopted by the VHA.
Guided by practice-based learning theory, which requires residents to investigate and evaluate their care of patients, appraise and assimilate scientific evidence, and continuously improve patient care based on self-evaluation, residents engaged in several self-assessment activities. After each SP-resident encounter, residents received 30 minutes to (1) review a gold standard video of a faculty member performing the medical interview; (2) respond to 2 open-ended questions in which they compared their performance to the video; and (3) complete select questions adopted from the Self-Assessment Checklist for Personnel Providing Primary Health Care Services.15 The questions used a 5-point Likert scale (from 1, strongly disagree, to 5, strongly agree).
The Institutional Review Board at the University of Michigan and the VA Research and Development Committees (Ann Arbor and Cleveland) approved this study.
Data Analysis
We conducted ANOVA tests to analyze mean performance differences across resident training levels. We also conducted 2-tailed t tests to determine if significant differences existed in residents' performance based on case (combat-related PTSD versus MST) and unannounced SP gender. Descriptive statistics were used to report response frequencies from the self-assessment questionnaire. In prior work, we noted many aspects of validity evidence for the checklist.14 Examination of overall score and each domain indicated significant differences by expertise level when comparing medical students, residents, and faculty. Faculty performed significantly better than trainees in this exercise, demonstrating validity evidence for the use of the unannounced SPs.
Results
Ninety-eight internal medicine residents (45 postgraduate year 1 [PGY-1], 33 PGY-2, 20 PGY-3) were assessed by an unannounced SP (50 PTSD, 38 MST) in the primary care clinics at the VAAAHC (n = 80, 82%) or LSCVAMC (n = 18, 18%).
Overall performance scores are presented as percentages of the maximum score of 69 possible points (table 1). The ANOVA results indicated that there were significant differences among training levels for the domains of assessment skills (P < .001), MST (P = .002), and overall performance (P < .001). Post hoc tests (Tukey-Kramer honest significant difference, global P = .05) indicated that PGY-3 participants performed better than PGY-1 and PGY-2 participants for these domains. Although performance improved by training level, all residents continued to have the lowest scores for both mental health screening domains (table 1).
Domain performance was also examined by case and SP gender to evaluate any potential bias (table 2). For the PTSD case, scores were higher in several domains (communication skills, MST, triage, and overall) when the SP was female compared with male SPs. For the MST case, scores were higher for the female SPs in only 1 domain (MST).
Fifty-three residents (54%) completed the self-assessment questions (table 3); 51 (96%) indicated that military culture may influence how service members respond to illness, disease, and death. Only 30 (58%) agreed that they sought out and engaged in professional development and training to enhance their knowledge and skills in the provision of services and support to military and veteran groups. Even fewer residents (n = 27, 52%) agreed they were well versed in the most current and proven practices, treatments, and interventions or major health problems among military and veteran populations served by their facility. As part of the self-assessment activities, residents provided narrative comments to the 2 open-ended questions that highlighted residents' discomfort with sexual history taking and broaching difficult subjects.
Discussion
In this study of exploring residents' skills during a simulated new patient visit with an unannounced SP, residents across PGY levels performed best on communication skills and professionalism and worst on 2 mental health screening domains. There were large variations in scores (higher standard deviations) in domains where residents performed poorly, indicating a wide range in resident skill levels, independent of their year of training.
Residents also performed worse on nearly all domains when the unannounced SP was male compared with female SPs. This finding is reassuring given concerns that female veterans may receive worse care despite their growing proportion (9.8%) among the veteran population.3 In response to the increase in female veterans, the VHA has begun integrating women's comprehensive care and services into their health care systems,16 potentially making residents more sensitive to their needs. Additionally, care for male veterans may present more of a challenge due to residents' lack of knowledge regarding the prevalence of male sexual assault victims or known barriers to care, including existing mental models of hypermasculinity, stigma and shame of male sexual assault, and perceptions of homosexuality.17,18 Our results show the need for continual educational efforts regarding VCC skills, especially mental health screenings.
Despite 2 decades of targeted efforts to improve screening health services for MST, in conversations following the SP encounters, many residents indicated that inquiries surrounding unwanted or uninvited sexual incidents were the most difficult part of the encounter.
This study has limitations. First, we did not capture follow-up measures to determine change in residents' performance as a result of their unannounced SP encounter. Also, despite efforts to capture data from 2 VHA hospitals, our sample was limited, and results may not be generalizable to other VHA hospitals. Finally, while the items we used on the self-assessment questionnaire provided useful information, they lack validity evidence and the low response rate limits the strength of the results.
Further research is needed to better understand how physicians develop VCC skills, especially in the initial visit while they are establishing rapport. In addition, as more than two-thirds of contemporary veterans receive their medical care in civilian locations, VCC principles should be taught beyond VA facilities.
Conclusion
This study suggests that additional education and assessment in VCC may be needed, particularly in caring for patients with PTSD and MST.
Author Notes
Funding: This work was supported by a grant from the University of Michigan Medical School, Office of Graduate Medical Education.
Conflict of interest: The authors declare they have no competing interests.
The authors would like to thank all the military veterans who serve the citizenry of their respective countries.
The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the US government.
