Editor's Note: The online version of this article contains the performance checklist used in the study.

Introduction

In an effort to improve patient safety, residency programs are seeking new methods of teaching invasive bedside procedures.1–3 Studies confirm that simulation-based training,4,5 use of ultrasound,6,7 adherence to a checklist,8 team training,9 and direct observation10 decrease complication rates, thereby improving safe practices.

In 2007, the American Board of Internal Medicine (ABIM) ceased to mandate technical competency for some procedures, while it recommended active participation in a predetermined number of procedures subsequent to simulation-based learning.11 Despite the relative infrequency of internists performing procedures posttraining,12 some academic programs may continue to rely on residents to do so as a component of their training. In the absence of endorsed criteria for measuring competency, some training programs devised their own systems. Smith et al3 created a medical procedure service and evaluated patient outcomes but did not specifically define a competency threshold. Dong et al2 and Huang et al8 provided validity evidence for a central line checklist by using a task trainer but did not link the learner's skill to patient outcomes. Studies suggest that to provide a comprehensive assessment of trainees' abilities, a range of tools should be used.13

The purpose of our study was to build on the work of others in developing and evaluating new criteria for procedural competency that included knowledge, skill, and attitude coupled with the assessment of procedural outcome measures.

Methods

Results

A total of 171 residents participated in the procedure team rotation. Under direct attending physician supervision, residents inserted CVCs by using real-time ultrasound guidance for a variety of clinical indications. Reported information reflects only those residents with complete data, defined as postintervention written test, checklist scores, and attitude ratings needed to evaluate at least 1 CVC attempt. Excluding the 18 participants with incomplete records and the 5 who did not perform an insertion, we assessed a total of 148 residents during 639 procedures. Fifty percent of the patients were referred by medical teaching teams, and the majority were located on the general medical wards.

The greatest number of CVCs (416 [65%]) were inserted into the internal jugular vein, followed by 173 (27%) in the femoral and 49 (8%) in the subclavian veins. Insertion site was missing for 1 procedure. Each site was chosen based on the clinical indication for the procedure as well as patient-specific characteristics (eg, patency of the selected vein).

The sample population of trainees is described in table 1a. Applying our criteria to the first procedure in a patient, only 4 residents began the training with established competency. Of the 148 residents who completed all assessments, 53 (36%) ultimately achieved the primary outcome of competency, 40 residents (27%) were judged to be borderline competent, and the remaining 55 (37%) were deemed not competent.

TABLE 1A Procedure Team Resident Characteristics During Central Venous Catheterization at Jackson Memorial Hospital (2007–2011)

View Fullscreen

Sixty-nine percent of all procedures were scored as 100% on the checklist. For only 7 (1%) procedures did a resident rate himself or herself lower than 4 of 5, despite achieving a 100% checklist score and receiving 5 of 5 from the supervisor. Five of these 7 residents ultimately achieved competency. Conversely, there were 11 (2%) instances of a resident scoring less than 100% and receiving at least 1 faculty rating below 4 of 5, yet the resident self-assessed 4 of 5 or higher. None of these residents were ultimately deemed competent.

The data demonstrated significant improvement in the postintervention written test and checklist scores (P < .001) within each group. Residents reported performing an average of 7 (range: 0–25) insertions prior to the simulation-based program; in our study population, each resident performed an average of 4 (range: 1–12). Those who had performed a higher number of procedures before the training were more likely to be assessed as competent (P  =  .05). However, the number of procedures performed during the rotation was significantly different among the groups (P < .001), with those performing the most procedures categorized as competent and those with the least number of procedures, not competent.

Procedural outcomes were reviewed individually and as a composite measure (table 1b). There was insufficient evidence to demonstrate a difference in the rate of completed procedures between groups, but a significant difference was noted among groups in the rate of multiple attempts (P  =  .002), favoring the borderline and competent groups. While the overall difference among groups for complication rate was not statistically different (P  =  .08), a significant difference was demonstrated between those who were not competent and those who were (P  =  .03). Taken together, the composite success rate showed a significant difference among all groups (P  =  .001).

TABLE 1B Procedure Team Resident Outcomes During Central Venous Catheterization at Jackson Memorial Hospital (2007–2011)

View Fullscreen

table 2a shows outcomes by final self-assessment regardless of checklist score. There was a significant difference in team experience (P  =  .02) and multiple attempt rate (P  =  .02), favoring those with higher ratings. table 2b demonstrates the breakdown of outcomes based on individual self-assessed rating. Similarly, table 3a reports outcomes by faculty rating. Those whom the faculty rated highest performed more procedures while on the team than those not rated as highly (P < .001). More importantly, those rated highest by the faculty demonstrated a higher composite success rate (P  =  .02). table 3b lists the breakdown of outcomes based on faculty rating.

TABLE 2A Procedural Outcomes by Final Resident Self-Confidence Group, Regardless of Checklist Score

View Fullscreen

TABLE 2B Procedural Outcomes by Final Resident Self-Confidence Rating, Regardless of Checklist Score

View Fullscreen

TABLE 3A Procedural Outcomes by Final Faculty Rating of Confidence Group, Regardless of Checklist Score

View Fullscreen

TABLE 3B Procedural Outcomes by Final Faculty Rating of Confidence, Regardless of Checklist Score

View Fullscreen

Discussion

To date, there has been no universal standard for determining that residents are competent to perform invasive bedside procedures.3,18–20 The approach we developed combines objective assessments of knowledge and skill with subjective assessments of attitude, ultimately linking them to procedural outcomes. While others have reported the utility of a similar assessment instrument,8 historically, self-assessments of performance have been challenged.21,22 Our project builds on prior work that called for clearer standards to define competency.3

The prior experience average in our sample was higher than the recent ABIM numerical threshold, yet only 4 residents were competent at the outset. These results support work that has found that prior experience is not a sufficient predictor for competency.2 Of note, the number of procedures performed while on the team demonstrated an association between procedure team experience and attainment of competence. Perhaps the number of procedures is not quite as important as the educational process. This may provide further support for the “progression of competency.”19,20 Compared to those judged not competent, the competent group had a significantly higher composite success rate. Data support the contention that multiple attempts result in a higher complication rate.3,23 That no differences in the overall complication rates among groups were noted was not surprising given the low overall incidence rate (6%). However, the competent group had a significantly lower rate than those who were not competent.

With respect to subjective assessments, a rate of lower multiple attempts was noted for individuals who rated themselves higher on their self-assessment, and a higher composite success rate was found for the group rated highest by the faculty.

One limitation of our study is that participants were evaluated during a brief 4-week experience on the procedure team rotation, and it is possible that if the evaluation period had continued after their time on the team, others might have met the criteria necessary to be judged competent. Second, the proposed definition was used during a single procedure, CVC insertion. Third, our sample size was small. A more robust sample size may demonstrate an association between procedural outcomes and self-assessment. Finally, the study was conducted with internal medicine and medicine–pediatrics residents at a single institution. Further study is needed to evaluate generalizability to other institutions and specialties.

Conclusion

We have demonstrated that the application of our multi-faceted definition (assessments of knowledge, skill, and attitude coupled with procedure-specific outcomes) can result in the determination of an individual's competency for a particular procedure in a short time frame. During our brief 4-week study period, slightly more than one-third of residents achieved the threshold for competency, yet most did not. We offer this new model to other training programs that are challenged to define invasive bedside procedural competency and as a beginning of further work to define the optimal training model and training length to ensure competence and safe and effective patient care.