Setting

Each of the participating academic institutions (St. Christopher's Hospital for Children, the University of Wisconsin, and the University of Michigan) had existing EBM curricula for pediatrics residents, which included the preparation of a CAT and presenting findings to peers and faculty. Pediatrics residents completing this project were required to identify a clinical scenario, ask a clinical question, perform a literature search, select an article, critically appraise the selected article, and apply the evidence back to their clinical scenario. The EBM learning activity included creating and presenting a CAT synopsis to their peers and faculty members in a small group setting. The existing EBM curricula at each institution were not modified. Residents developed their EBM-learning activities for presentation using a standardized format originally developed by David Sackett.1

Evaluation Tool Development

Three investigators (the 3 authors), all with expertise in teaching and evaluating EBM for pediatrics residents, convened to develop the EBM C-PET tool (figure). The investigators are a part of a group of 7 colleagues, nationally recognized within the pediatrics community for their contributions to curriculum development in EBM.14,15 The skills under consideration for the EBM C-PET tool were derived from a comprehensive EBM curriculum developed by the 7-member working group after a rigorous, iterative process that included searching the literature and reviewing existing EBM curricula and assessment tools. We used a modified Delphi technique to reach consensus on the optimal areas for inclusion in an EBM curriculum.16 We identified the items from the comprehensive EBM curriculum that would be incorporated into the EBM C-PET, resolving disagreements through discussion. The final tool underwent further review by additional experts to ensure adequate representation of the items that would be included in the EBM C-PET tool.

View Figure

• Download as Powerpoint
• Download Figure

Once the relevant EBM skills were determined, the investigators sought to categorize resident performance into 3 categories: below expectation, developing, or proficient. The 3 assessment categories were initially based on an existing EBM evaluation at one of the institutions. Behavioral anchors were created to describe performance in each of the 3 assessment categories for each of the items measured. The 3 investigators independently pilot-tested the tool on 2 sample CAT presentations/synopses, and the criteria to measure those domains were refined using a frame-of-reference training technique.17,18 The frame-of-reference training included evaluating the sample presentations and comparing the 3 investigators' ratings of the same presentations.18 This process informed the development of the tool and demonstrated that the initial 3 assessment categories were insufficient to reflect the full range of resident skills and to show meaningful differences in resident performances. The initial 3 categories were expanded to 5, using the Dreyfus and Dreyfus model of skill development. That model was adopted because it is based on skill acquisition, is familiar to educators, and could be readily used in the development of EBM assessment tools.19,20 The 5 terms in the Dreyfus and Dreyfus model (novice, advanced beginner, competent, proficient, and expert) were chosen to assess resident performance with the evaluation tool. For each of the 14 items measured, specific behavioral anchors were created to describe performance that would fit each of the Dreyfus and Dreyfus categories. After additional pilot testing, those behavioral anchors were further refined using an iterative process. Again, we used frame-of-reference training on our final EBM C-PET evaluation tool, discussing the 3 investigators' ratings of the same pilot presentations to help “calibrate” the investigators so they were able to discriminate between different levels of performance.18

Data Collection

Pediatrics residents' EBM CAT presentations were videotaped, and written CAT synopses were copied for distribution to the 3 investigators. Each investigator independently reviewed and scored each EBM CAT presentation and its corresponding written CAT synopsis using the EBM C-PET. Data were stored anonymously in an Excel (Microsoft Inc, Redmond, WA) database.

Each institution's Institutional Review Board granted exemption to the project before the start of data collection.

Statistical Analysis

Cronbach α coefficients21 were calculated to assess internal consistency using the individual item scores from each participant, that is, items 1 to 14, for the complete EBM C-PET tool. A Cronbach α of 0.7 to 0.8 was considered acceptable, a value from 0.8 to 0.9 was considered good, and a value greater than 0.9 was considered excellent. The results of the scoring of the individual EBM presentations were analyzed with SPSS software (IBM Corporation, Armank, NY) to determine the interrater reliability of the EBM C-PET. An intraclass correlation coefficient (ICC) was used to assess the interrater reliability. An ICC of 0.5 to 0.75 was considered moderate, and an ICC above 0.75 was considered good. A sample size of 30 was selected to provide a sufficiently narrow confidence interval around the observed ICC for the overall instrument.

Tool Development

The final EBM C-PET assessment tool included 14 items (tool provided as online supplemental material). Eight of the items (57%) focused on EBM: develop a well-built question from a clinical scenario, understand and use key search terms, critically appraise the article, apply the critical appraisal to the clinical scenario, prepare a CAT, place the current study in context, explain EBM concepts, and provide an overall assessment of EBM skills. Five items (36%) emphasized the resident's teaching ability: organization and clarity, effective use of teaching tools, synthesis and integration, presentation conducted at level appropriate for audience, and an overall assessment of teaching skills. One global item (7%) assessed the resident's overall performance.

As an example of item development, the original scale for the item, “Places the current study in the context of other relevant research on the topic,” included: (1) Does not place the current article into the larger context (below expectation), (2) Makes reference to 1 other source of evidence on the topic (developing), and (3) Integrates information from the current study into the larger literature (proficient). After adaptation to the Dreyfus and Dreyfus model, the scale included the following 5 items:

• Novice: Does not place the current article into the larger context (ie, no background information)

• Advanced beginner: Makes reference to a few other sources of evidence on the topic but does not critically analyze their relationship to the current study

• Competent: Makes reference to a few other sources of evidence on the topic and begins to analyze their relationship to other studies

• Proficient: Integrates information from the current study into the larger literature using critical analysis

• Expert: Seamlessly integrates information from the current study into the larger literature using critical analysis and communicates how the article affects practice

Tool Performance

The EBM C-PET had excellent internal consistency, with a Cronbach α of 0.94 across all 3 raters. When considering the items on the tool that were used to assess the EBM presentation, Cronbach α was 0.91, and when considering the items assessing the CAT, Cronbach α was 0.84.

We used the ICC to assess interrater agreement. The ICC for each individual item on the assessment tool ranged from 0.31 to 0.74 (table), and the average ICC for the 14 items was 0.67. We examined the lowest performing items to see whether the tool could be modified further, but there was no clear pattern for excluding specific items. Thus, we maintained the original 14 items on the final EBM C-PET tool.

Table Agreement Among 3 Raters Across Sites

View Fullscreen