The “Zing Factor”—How Do Faculty Describe the Best Pediatrics Residents?
Abstract
Background
Faculty in graduate medical education programs may not have uniform approaches to differentiating the quality of residents, and reviews of evaluations suggest that faculty use different standards when assessing residents. Standards for assessing residents also do not consistently map to items on evaluation forms. One way to improve assessment is to reach consensus on the traits and behaviors that are (or should be) present in the best residents.
Methods
A trained interviewer conducted semistructured interviews with faculty affiliated with 2 pediatrics residency programs until content saturation was achieved. Interviewees were asked to describe specific traits present in residents they identify as the best. Interviews were recorded and transcribed. We used an iterative, inductive approach to generate a coding scheme and identify common themes.
Results
From 23 interviews, we identified 7 thematic categories of traits and behaviors: personality, energy, professionalism, team behaviors, self-improvement behaviors, patient-interaction behaviors, and medical knowledge and clinical skills (including a subcategory, knowledge integration). Most faculty interviewees focused on traits like passion, enthusiasm, maturity, and reliability. Examination score or intelligence was mentioned less frequently than traits and behaviors categorized under personality and professionalism.
Conclusions
Faculty identified many traits and behaviors in the residents they define as the best. The thematic categories had incomplete overlap with Accreditation Council for Graduate Medical Education (ACGME) and CanMEDS competencies. This research highlights the ongoing need to review our assessment strategies, and may have implications for the ACGME Milestone Project.
Editor's Note: The online version of this article contains the interview guide used in the study.
Introduction
In this era of competency-based assessment, clinical faculty are expected to assess a resident's performance in each of the 6 areas defined by the Accreditation Council for Graduate Medical Education (ACGME) competencies.1 When a resident is deemed competent in all of these areas, he or she is deemed competent overall and eligible for certification by specialty boards. However, it is not clear whether attending physicians use the framework of competencies or another framework when they assess and stratify residents, particularly when they are asked to identify the “best” residents.
There are many stakeholders who value greater differentiation among residents. Postresidency training programs (ie, fellowships), future employers, and patients often desire more than “satisfactory” or “competent” physicians; they want to know who the “best” are. “Best” is generally defined relative to overall performance of peers; letters of recommendation and evaluation forms commonly identify graduates at specific percentiles (or top percentage) relative to those of their peers. Stratifying physicians requires a greater understanding of how faculty evaluators form their assessments of residents and what they are looking for. Data suggest that faculty may take different approaches to assessing residents and determining who are the best (and worst).2–5 In pediatrics, Altmaier et al2 attempted to define categories of resident performance associated with success and were able to identify 6 categories. Since their work in 1990, we have seen the evolution of the ACGME competencies and the CanMEDS framework.1,6
Eliciting narratives, the stories faculty tell about trainees, is 1 way to understand how assessments are formed.7 One recent study highlighted the strong subjective influence in global assessments4; other authors suggest that a variety of factors influence faculty ratings of performance.8 In another study, researchers divided comments about internal medicine residents into thematic categories and noted that not all categories spontaneously offered by faculty could be mapped to the CanMEDS framework.9 These findings suggest different approaches to development of evaluation forms and the need for faculty development so that such forms are used in a consistent way. Building on this body of work, our research began with the core question of, “How do faculty members assess pediatrics residents?” In this study, we aimed to identify thematic categories of traits and behaviors that faculty use when describing the best pediatrics residents.
Methods
Design
We conducted a qualitative study using a general inductive approach. This allowed us to identify traits and behaviors directly from our data rather than imposing a preexisting framework. We interviewed faculty members who supervise residents in 2 different residency programs covering 3 clinical sites.
Subjects and Setting
Faculty members in 2 departments of pediatrics, University of California San Francisco (UCSF) and Children's Hospital and Research Center Oakland (CHRCO), were eligible to participate in this study. The UCSF pediatrics residency program has approximately 85 residents and is based at a tertiary care academic medical center. The CHRCO residency program has approximately 80 residents and is based at an urban free-standing children's hospital. The 2 programs are in the same geographic area, approximately 13 miles apart, and were chosen for convenience.
There was no predefined sample size. Rather, the goal was to interview 10 to 15 faculty members from each program, determine if saturation had been reached, and add additional participants if needed. This number of interviews on a site level is consistent with methodology described by Guest et al.10 We purposely sampled pediatrics faculty from a variety of academic levels and subspecialties, including some with formal positions in pediatric medical education, to ensure inclusion of potentially diverse perspectives.
Procedures
The authors created a semistructured interview guide (provided as online supplemental material) and designed interview questions to elicit from faculty specific characteristics, qualities, and/or behaviors present in residents they identified as “the best.” The interviews began with a trained research assistant (E.M.A.) who provided composite photographs of current and recently graduated residents to help stimulate participants' memories of residents. For confidentiality, the research assistant instructed participants not to name any specific residents. After displaying the photographs, the interviewer started by asking faculty, “How do you define best?” The guide provided consistency and structure to the interviews but also allowed room for probing and exploring emerging themes. The research assistant piloted the guide to improve clarity and check interview length (approximately 30 minutes). Interviews were recorded and transcribed by a third party, who removed potentially identifying information (including institution) prior to analysis.
All participant contact and interviews were conducted following procedures approved by the UCSF Committee on Human Research (CHR). The Institutional Review Board at CHRCO deferred to the UCSF CHR.
Analysis
We used an iterative, inductive approach to generate a coding scheme and identify common themes.11 Initially, 3 authors (G.R., E.M.A., C.S.C.), 1 of whom conducted the interviews (E.M.A.), each reviewed a set of transcripts to identify key traits and behaviors. These traits and behaviors were loosely categorized, discussed in detail, and refined.
In the first review of 4 transcripts, we identified many traits and behaviors and grouped them into 7 thematic categories based on overlapping themes. In a second round, we reviewed 2 additional transcripts and refined the descriptions of the 7 categories. In the third round, we reviewed 2 additional transcripts, resulting in further refinement of the 7 categories. Using this final set of 7 categories, 2 authors (G.R., C.S.C.) coded the same 3 interviews and compared their coding. The authors discussed all coding discrepancies (which were few) until they reached consensus.
Subsequently, 1 author (G.R.) coded all of the transcripts, including those used to develop the coding scheme. This coding was reviewed by a second author (C.S.C.), and discrepancies were discussed and resolved. Final coding was entered into NVivo software (QSR International [Americas] Inc, Burlington, MA) to facilitate review of themes.
Results
At 20 interviews, we determined that saturation had been achieved based on recurring descriptions of similar behaviors and traits with no new additions. At that point, the research assistant completed 3 additional interviews (scheduled prior to discussions about saturation) and confirmed that no new descriptions or themes surfaced. Interviews were completed with 10 UCSF and 13 CHRCO faculty members.
We identified 7 thematic categories of descriptors that pediatrics faculty used to identify the “best residents” as follows: personality, energy, professionalism, team behaviors, self-improvement behaviors, patient-interaction behaviors, and medical knowledge and clinical skills (including a subcategory, knowledge integration). Adjectives and descriptors of traits and behaviors within each category are described in the table. Each of these thematic categories was captured in at least 80% of the interviews.
Personality
General personality traits were grouped into a category together to encompass the various traits mentioned by faculty that were not directly related to the practice of medicine. They included sense of humor, optimistic, adaptable/flexible, humble, calm, confident, likable, mature, socially intelligent, socially graceful, empathic, creative, resilient, buoyant, curious, and thorough. All faculty participants mentioned at least one trait in this category.
Often, the focus was on positive attitudes, for example, “I would say often it's not academic. Often it's attitudinal” (Interview 7); and “I think just the overall pleasantness about them. They're just pleasant to be around” (Interview 2). “There's just a certain personality type where you just can't get them down … they just have a buoyancy … nothing gets them down” (Interview 5).
Energy
Many faculty (83%) mentioned elements related to energy and enthusiasm, including passion, excitement, interest, “zing,” motivation (particularly motivation to learn), and inquisitiveness. Energy stood out as unique from other personality traits as much more “active.” “There's a certain got it—‘zing’—that extremely competent residents exude. It is related directly to their performance. But it's a subjective assessment. It's colored by interpersonal skills and interactions” (Interview 3); and “We have some residents who are effective because their energy level is so high” (Interview 9).
Professionalism
Faculty described traits such as good work ethic, hardworking, reliable, responsible, and meticulous as an important aspect of being “the best.” Traits relating to professionalism were mentioned by 96% of faculty.
Faculty frequently described professionalism in terms of commitment to the job: “it's a professionalism … a commitment [that] they take their oath seriously” (Interview 7) or “seriousness with which they take what they're doing, the responsibility you feel as a physician in sort of making these decisions” (Interview 14).
Team Behaviors
Faculty described behavior that related to interactions with others: relates to staff, teamwork/team player, good communicator (with team), good listener (with team), leadership behaviors, and educator. There were references to team-related behaviors by 96% of interviewees.
The general focus was on residents' abilities to work with others: “I think of the best doctors as people who have the ability to work with others” (Interview 2). Specific behaviors were frequently given as examples, including “ … using both verbal and physical cues showing your support of others: complimenting people, patting them on the back when they need it, making eye contact, and paying attention to a student or an intern when they're talking to you. I think those kinds of behaviors carry with them a certain amount of leadership and confidence and kindness and caring and compassion that are part of what makes a good resident” (Interview 9).
For senior residents, there was also frequent mention of team skills: “The people, especially senior residents, who keep their teams together despite a diversity of personalities and skill sets, are I think among the top senior residents” (Interview 2).
Self-Improvement Behaviors
Reflective practice, solicitation of feedback, awareness of needs, learning, and self-assessment were consistently mentioned. Behaviors in this category were mentioned by 100% of interviewees.
One example of these behaviors included, “how that person deals with adversity, mistakes, errors, bad things that happen, failure, that kind of thing … those are things that test what is intrinsic to the best residents, those kinds of bad events; an ability to step back and reflect, to feel and to be affected by what goes on, but not to have it be devastating to them,” and “their ability to take feedback and use it positively for themselves to improve themselves” (Interview 9).
Patient-Interaction Behaviors
Ninety-one percent of faculty described behaviors that directly related to patients, as distinct from those indirect behaviors related to teamwork and self-improvement. Those behaviors included follow-through, taking initiative, going above and beyond, taking ownership, advocating/advocacy, acting independently, knowing how the hospital worked (to advance care), bedside manner, and being a good communicator (with patients) and a good listener (with patients).
Descriptions of those behaviors included, “the ability to find the voice that the patient needs. You have a wide spectrum of abilities in the patients and their parents, and you need to go in there and find what kind of person they need” (Interview 5). “One example is how I saw one resident actually holding a patient's hand or the parent's hand or putting her hand on her shoulder when she was upset. Not everyone feels comfortable with that” (Interview 12).
“What is your ability to communicate to a patient? What are your verbal communication skills? What are your nonverbal communication skills? What is your posture? How do you establish rapport with a family, or with a patient? Those are the things that residents hone in on and excel at that define them as being, I guess, the best” (Interview 21).
Medical Knowledge and Clinical Skills
Faculty described medical knowledge and other descriptions analogous to fund-of-knowledge, including global descriptions of intelligence (using a variety of adjectives) as well as references to being clinically strong. Most faculty (83%) included knowledge when describing the “best” residents.
Faculty generally held a global view of an individual's knowledge base, as in “They're incredibly smart. They know something about something. They come across as incredibly brilliant” (Interview 12). They also made specific references to clinical judgment: “The first and most important [quality] is clinical judgment. They have to have good clinical judgment” (Interview 18). References to procedural skills and test taking were extremely infrequent.
Within the category of medical knowledge and clinical skills, we included a subcategory, knowledge integration. This category referred to higher-order skills, such as synthesis, organization, prioritization, attention to detail (when referring to understanding of prioritization), application, and good presentations. These skills were described by 87% of interviewees.
The behaviors included, “quickly develop[ing] the ability to go beyond numerical reporting of data into synthesizing and doing a differential diagnosis and coming up with a detailed treatment plan” and “being able to synthesize a plan for the patient, taking all that you've read and learned, and saying ‘this is what I think we should do’ ” (Interview 3).
Discussion
In this qualitative study, we identified 7 categories of traits and behaviors associated with the best pediatrics residents: personality, energy, professionalism, team behaviors, self-improvement behaviors, patient-interaction behaviors, and medical knowledge and clinical skills (including a subcategory, knowledge integration). Our results confirm some of the components of the ACGME competencies1 and also overlap with the CanMEDS framework.6 However, they also highlight some gaps between areas where pediatrics faculty look to identify the best residents and the ACGME competencies.
For example, we were surprised to find that our faculty spent a large amount of time discussing residents' dispositions or other personality traits, something which is not specifically addressed in the CanMEDS framework or the ACGME competencies. Although some behaviors such as adaptability and maturity may be covered to some degree, others such as having a sense of humor, humility, or buoyancy have not previously been reported as explicitly desirable traits for pediatrics residents. However, taken in context, it makes sense that faculty would value those traits as so much of clinical practice occurs in teams. The identification of “energy” as a meaningful trait was also unexpected. Some elements of our energy category overlap with practice-based learning and improvement,1 although the descriptors did stray from this competency. Although this category could easily have fit under personality, we felt it warranted a separate category because it was mentioned so frequently and explicitly.
We expected all faculty interviewees to focus on medical knowledge, but 17% of faculty did not specifically mention traits or behaviors that we categorized as medical knowledge. They spent very little time discussing residents' overall intelligence, in-training examination scores, book knowledge, being current with up-to-date best practices, and/or using evidence-based medicine. One possible explanation is that the immediate accessibility of knowledge, particularly in the era of handheld devices, diminishes its value as a proxy for excellence in other areas.4 It is also possible that all residents in a program achieve a certain knowledge standard, obviating the need for further discussion.
Some may question the importance or desirability of identifying the best residents within a training program. On a very simple level, we stratified physicians' responses to be able to answer the question, “Who is the best physician for x?” or “Who should I hire for my practice or fellowship?” From an educational development approach, stratification is natural as we seek to determine who is excelling and who is not. The ACGME Milestone Project12 aims to define specific competency-based developmental outcomes for residents and will then ask training programs to report residents' performance on these Milestones. Although this approach ultimately aims to ensure that all residents will meet a uniform standard of competence before leaving residency, it may result in de facto stratification as residents will likely progress through the Milestones at different rates. Reporting this performance accurately will require assessment by faculty. Interestingly, prior studies,5,13 as well as ours, have shown that when asked to assess residents, faculty tend to rely on generalized impressions and global behaviors rather than specific behaviors. The Milestone Project may provide a standardized framework to assist in overcoming this tendency.
Our study has several limitations. We interviewed faculty from 2 pediatrics residency programs in a single geographic region, potentially limiting generalizability. Also, using a single-data collection method limited our ability to assess internal validity. With additional time, it would have been interesting to query the faculty awareness and perceptions of the ACGME competencies to understand possible explanations for the incomplete overlap between our categories and the ACGME competencies.
Conclusion
This study adds to the growing body of literature on resident assessment by describing thematic categories of traits and behaviors pediatrics faculty use to identify the best residents. The incomplete overlap with current competency frameworks suggests that we may need to reassess these frameworks in the context of new evidence. Further research in this area will be needed as we move to more behavior-based assessments in the ACGME Milestone Project.
Author Notes
Glenn Rosenbluth, MD, is Associate Clinical Professor of Pediatrics, Division of Pediatric Hospital Medicine, Division of Pediatric Medical Education, University of California San Francisco (UCSF) and UCSF Benioff Children’s Hospital; Bridget O’Brien, PhD, is Assistant Professor of Medicine, Office of Medical Education, UCSF; Emily M. Asher, MS, MPH, MPA, is a Student, School of Medicine, UCSF; and Christine S. Cho, MD, MPH, MEd, is Assistant Professor of Clinical Pediatrics and Emergency Medicine, Division of Pediatric Emergency Medicine, UCSF, UCSF Benioff Children’s Hospital, and Children’s Hospital and Research Center Oakland.
Funding: This study was funded intramurally with a Pediatric Educational Innovations grant through the Haile T. Debas Academy of Medical Educators at the University of California San Francisco.
The authors would like to thank Christy Boscardin, PhD, for providing guidance and feedback on study design.
