Handheld Ultrasound Device Usage and Image Acquisition Ability Among Internal Medicine Trainees: A Randomized Trial
ABSTRACT
Background
There is insufficient knowledge about how personal access to handheld ultrasound devices (HUDs) improves trainee learning with point-of-care ultrasound (POCUS).
Objective
To assess whether HUDs, alongside a yearlong lecture series, improved trainee POCUS usage and ability to acquire images.
Methods
Internal medicine intern physicians (n = 47) at a single institution from 2017 to 2018 were randomized 1:1 to receive personal HUDs (n = 24) for patient care/self-directed learning vs no-HUDs (n = 23). All interns received a repeated lecture series on cardiac, thoracic, and abdominal POCUS. Main outcome measures included self-reported HUD usage rates and post-intervention assessment scores using the Rapid Assessment of Competency in Echocardiography (RACE) scale between HUD and no-HUD groups.
Results
HUD interns reported performing POCUS assessments on patients a mean 6.8 (SD 2.2) times per week vs 6.4 (SD 2.9) times per week in non-HUD arm (P = .66). There was no relationship between the number of self-reported examinations per week and a trainee's post-intervention RACE score (rho = 0.022, P = .95). HUD interns did not have significantly higher post-intervention RACE scores (median HUD score 17.0 vs no-HUD score 17.8; P = .72). Trainee confidence with cardiac POCUS did not correlate with RACE scores.
Conclusions
Personal HUDs without direct supervision did not increase the amount of POCUS usage or improve interns' acquisition abilities. Interns who reported performing more examinations per week did not have higher RACE scores. Improved HUD access and lectures without additional feedback may not improve POCUS mastery.
Introduction
Point-of-care ultrasound (POCUS) usage is increasing across the clinical training spectrum as more investigations have demonstrated its positive effect on patient care.1–5 However, there is a risk trainees may be performing POCUS with minimal oversight (including for procedures).6 Outside of emergency medicine, there are few guidelines on how to effectively teach POCUS or measure competency.6–8 There is a growing need to investigate how to optimally teach trainees this ever-increasing diagnostic modality.3–5,8
Previous investigations of POCUS education have mainly focused on the effect of lectures on image interpretation.6,9–15 POCUS educational guidelines have advocated for improved device availability,8,16–18 but there have been no studies examining whether access to personal handheld ultrasound devices (HUDs) improves trainee competency. In theory, improved access to HUDs increases opportunities for deliberate practice, which is needed to improve competency with POCUS beyond the classroom setting.19 Such practice may improve technical skill with acquiring images, which is a key feature of POCUS competency.6,19
In this randomized controlled study, we investigated whether increased access to HUDs increases the frequency POCUS is performed and whether increased HUD access improves a trainee's ability to acquire POCUS images. We also investigated whether trainees who report performing more examinations per week are more skilled at acquiring images.
Methods
Participants and Setting
The 2017 intern class (n = 47) at an internal medicine residency program participated in this study. The Stanford University Institutional Review Board (IRB) approved this investigation.
Study Design
Interns (n = 47) received didactics related to POCUS from June 2017 to June 2018 (Figure). In addition, the interns were randomized 1:1 to receive personal HUDs that could be used for patient care and/or self-directed learning (n = 24) vs no-HUDs (n = 23). POCUS usage rates and trainee ability to acquire POCUS images were assessed.
Citation: Journal of Graduate Medical Education 13, 1; 10.4300/JGME-D-20-00355.1
Outcomes
Main outcome measures included self-reported HUD usage rates and differences in post-intervention assessment scores between HUD vs no-HUD interns (see “Assessments” below). Additional outcomes included the relationship between the number of self-reported scans and technical skill assessment scores, differences in pre- vs post-intervention assessment scores, and trainee confidence with POCUS via surveys administered pre- and post-intervention (provided as online supplementary data).
Curriculum Implementation and Design
All interns (n = 47) received a 1-hour weekly lecture whenever they rotated on the inpatient medicine wards rotation (Figure). This rotation is 4 weeks in duration. Each lecture contained 2 parts: (1) 20 to 30 minutes of traditional didactics via PowerPoint, and (2) 30 to 40 minutes of supervised practice using HUDs on standardized patients. Trainees were given personalized feedback on their image acquisition technique during this portion of the class. Four lectures were given each month: (1) introduction to POCUS, (2) thoracic ultrasound, (3) echocardiography, and (4) abdominal ultrasound. These 4 lectures were repeated each month as new interns rotated on service. Most interns experienced the same content multiple times (2–3 times/year), which was intentional to assess learning rates with repeated lecture exposure, as has been previously reported.20 The contents of each lecture were based on previously published guidelines or expert consensus for POCUS training.7,18,20–28
HUDs
This study used the Philips Lumify HUD, an FDA-approved device. Interns randomized to HUDs received their device at the start of the rotation. Based on our IRB protocol, any saved patient images could not be reviewed by the researchers. However, interns could share their findings with their attending physicians for clinical care purposes. Attending discretion and comfort with POCUS determined whether to incorporate these images into clinical decision making or to provide feedback, which were not measured. The interns not randomized to HUDs could access a single Lumify device that was shared among residents and fellows in the hospital. This control HUD was kept in a centrally stored container that participants had to sign out from the unit secretary. Interns reported the average number of POCUS examinations performed each week via a survey sent during the last week of the rotation.
Assessments
Trainees were assessed on their ability to obtain cardiac and abdominal ultrasound images during the pre- vs post-intervention periods. This study utilized 2 technique assessments: (1) the Rapid Assessment of Competency in Echocardiography (RACE) scale, and (2) the Brightness Mode Quality Ultrasound Imaging Examination Technique (B-QUIET).29,30 Both methods have excellent interrater reliability and are well-described assessment tools for image acquisition ability regarding cardiac and abdominal POCUS.29,30 The RACE scale assesses image acquisition skills for trainees performing bedside echocardiography (provided as online supplementary data).6,29 The B-QUIET is a technique assessment that can be applied toward abdominal ultrasound (provided as online supplementary data).6,30
All pre- and post-intervention assessments were performed on 3 healthy male volunteers who were screened to ensure they had easily obtainable image windows. Trainees were randomly assigned a volunteer for scanning. Trainees were asked to obtain the following views/structures: parasternal long, parasternal short, apical 4-chamber, subcostal, inferior vena cava, splenic-diaphragm interface (from the mid-axillary line), and longitudinal right kidney (provided as online supplementary data). Trainees were given 15 minutes to scan and were instructed to save an image when they believed they had obtained an optimal image for evaluation. A study author was present for the scanning sessions to provide instruction and to set up the device but did not directly observe or comment on the images being acquired. Three of the study authors who teach POCUS at our institution (J.K., L.W., A.K.) independently reviewed the saved images and assigned scores using the RACE and B-QUIET scales. The study authors were blinded to the participant's identification when reviewing images. RACE scores were applied toward cardiac images, and B-QUIET scores were applied toward splenic and renal images. The average score across the 3 reviewers was used for the final trainee score.
Statistical Analysis
Outcomes were compared between the 2 groups using unpaired 2-sample Wilcoxon signed-rank tests. Correlations between interns' self-reported confidence levels and assessment scores were evaluated using Spearman's rank test of correlation using the R Project for Statistical Computing. A priori power analysis was performed with assumptions of a 0.0125 type I error threshold, standard deviation of 0.5, and effect size of 0.3. A sample size of 128 interns (64 per group) would be needed for 80% power to detect statistical meaningful differences. Therefore, this study was underpowered because we could only perform it on the 2017–2018 intern class.
Results
Baseline Characteristics
There were 47 categorical and preliminary interns in the study. Image acquisition skills were evaluated for 47 interns at the beginning of the study and 25 interns (HUD, n = 11; no-HUD, n = 14) at the end of the study (51% completion rate; Table 1). Lecture attendance was similar between the groups (Table 1).
Outcomes
Ultrasound Usage Rates:
HUD interns reported performing POCUS assessments on patients a mean 6.8 (SD 2.2) times per week vs 6.4 (SD 2.9) times per week in the non-HUD arm (P = .66). There was no correlation between an intern's final RACE or B-QUIET score and the amount of times they reported performing POCUS per week (Table 2).
RACE/B-QUIET Scores:
There were no significant differences in post-intervention RACE scores (including overall image quality and image generation) or B-QUIET scores between the HUD vs. no-HUD groups (Table 1). The median RACE image generation score increased from 0.0 pre-intervention (IQR 0.0–6.0) to 17.3 post-intervention (IQR 15.0–19.0; P < .0001; provided as online supplementary data) for all interns. Similarly, the B-QUIET scores for kidney and splenic POCUS significantly increased pre- vs post-intervention for all interns (provided as online supplementary data).
Trainee Confidence:
Overall, the interns' final RACE scores did not correlate with their confidence to correctly interpret POCUS images of the heart (Table 3). Similarly, the B-QUIET composite scores for both renal and splenic POCUS did not correlate with the intern's self-reported confidence to identify these structures on ultrasound (provided as online supplementary data).
Discussion
POCUS educational guidelines have advocated for improved device availability,8,16,18,19 but it is unknown if increased device access improves POCUS learning. In this study, we found that personal HUDs and a lecture series were neither correlated with an increase in POCUS usage nor did they improve technique assessment scores for trainees. Furthermore, trainee RACE or B-QUIET scores did not correlate with higher self-reported rates of POCUS usage or trainee confidence. Together, these findings suggest that improved HUD access with a lecture series that provides hands-on scanning may not be enough to promote POCUS mastery. Supplementary pedagogies, such real-time feedback with actual patients, are likely needed.6,8,19
Regarding the observation that POCUS usage rates did not increase with HUD access, it is possible that no-HUD interns used the community device more frequently because of the intervention. Alternatively, it is possible that the demands of being a trainee prevented HUD interns from performing more examinations, especially since time-motion studies have revealed that internal medicine interns spend less than 15% of their time toward direct patient care.32 Future studies should stringently track POCUS usage among trainees using time-motion methodologies and device logs.
Irrespective of identical HUD usage rates, we observed that trainees who reported higher HUD usage did not achieve higher RACE or B-QUIET scores. Previous investigations have shown that trainees can become proficient in acquiring cardiac and abdominal POCUS images in as few as 20 to 30 examinations,29,33,34 while other authors have shown that the mastery of skills requiring manual dexterity takes years of deliberate practice.35–37 It is possible the interns quickly achieved a plateau effect with POCUS proficiency, as observed with Millington et al.33 Additional mastery to detect a difference between the HUD groups or superusers may have required additional practice beyond the time frame of this study. Finally, we observed that there was no correlation between trainee confidence and their actual skill. This phenomenon has been described as the Dunning–Kruger effect and it may be common in POCUS learning.6,38
There are several limitations to this study. It was performed at a single academic institution with an underpowered sample size. It examined only intern physicians due to funding and rotation constraints. We were unable to perform the post-intervention assessment on all interns due to scheduling limitations (eg, days off and off-site rotations). We were unable to track the timing/location of the devices' usage, and the interns' self-reported usage rates may be subject to recall bias. To our knowledge, there were no significant lapses in device availability/functionality, and interns were discouraged from sharing devices. Due to timing constraints, we did not perform interval assessments of learning over the year. Intern physicians in the HUD arm didn't receive formal feedback on personally acquired patient images, which may have limited the intervention's impact.
Conclusions
In this small randomized study, we did not observe an improvement in internal medicine interns' acquisition of cardiac, splenic, or renal POCUS images despite having personal HUDs. Additionally, interns did not scan more frequently than peers without HUDs, suggesting that personal device access may not be a major limiting factor in promoting POCUS usage.
Overview of Study
Note: In period 1, assessments were performed on incoming 2017 interns (pre-intervention/baseline measurement) who were at the beginning of their year. No point-of-care ultrasound (POCUS)–related didactics were administered prior to the pre-intervention measurement. In period 2, they all received weekly didactics in POCUS while rotating on the inpatient wards service. In addition, they were randomized 1:1 to have personal access to a handheld ultrasound device (HUD; n = 24) or no such device (non-HUD; n = 23). In period 3, post-intervention assessments were performed after completing their year of ultrasound training. We compared image acquisition performance between pre- and post-intervention groups, as well as HUD vs no HUD groups. Due to scheduling constraints, 25 interns completed the post-intervention measurement (HUD, n = 11; no-HUD, n = 14).
Author Notes
Editor's Note: The online version of this article contains the survey used in the study, RACE score sheet, B-QUIET assessment, pre-intervention vs post-intervention RACE and B-QUIET scores, correlation between trainee confidence with post-intervention assessment scores, and survey results pre- vs post-intervention by HUD and no-HUD.
Funding: The authors report no external funding source for this study.
Conflict of interest: Dr Kumar is a paid consultant for Vave Health, which manufactures a handheld ultrasound device not used in this study. This consultancy began after this study was conducted.
