Introduction

Mastery learning is a form of competency-based medical education developed on the premise that all learners can achieve high levels of performance on a clinical task if given enough time to learn and practice.^1,2 Mastery learning is an effective educational approach with documented improved outcomes across many domains, including procedural skills,³ operative interventions,^4,5 and difficult conversations.^6,7 Additionally, programs implemented using a mastery learning approach have demonstrated improved patient care outcomes and significant cost savings,^8,9 suggesting the ability for this learning approach to improve individual and population health.¹⁰

There are 7 key components of the mastery learning bundle: (1) baseline testing; (2) clear learning objectives; (3) engagement in educational activities focused on reaching the objectives; (4) a set minimum passing standard (MPS); (5) formative testing; (6) advancement if test achievement is at or above the passing standard; and (7) continued practice until mastery (and the MPS) is reached.¹¹ The establishment of a predetermined, objective, defensible MPS is critical to determining when a learner has achieved mastery.

The first step in setting an MPS is selection of both the method of standard setting and the panel of judges. Historically, the Angoff and Hofstee methods¹² were used for standard setting in mastery learning curricula. Experts subsequently proposed new methods to better align with the goals of mastery learning.^13,14 The Mastery Angoff method¹⁴ has become a favored approach to establishing the MPS in the context of a mastery learning curriculum. Using this approach, judges are asked to consider the performance of a trainee who is ready and well-prepared for the next stage of training, practice, or learning, as opposed to the “borderline trainee” who is considered when using the traditional Angoff method.¹⁴ When comparing several different approaches to establish an MPS, one study found the Mastery Angoff method produced a much more stringent MPS, which aligns with the theoretical construct of mastery learning of ensuring that all learners are able to achieve a high level of performance in a mastery learning curriculum.¹⁵ Once the method is selected, judges are recruited. Guiding principles for judge selection include content expertise, appropriate knowledge of the learner group, willingness to follow instructions in the standard-setting process, and willingness to minimize bias.¹²

While mastery learning has been used extensively to teach clinical skills, these skills have focused primarily on procedures; thus, medically trained experts often establish the MPS.^16–20 With applications of mastery learning expanding to teaching communication and other patient-centered care skills, there is an opportunity to incorporate patients as experts when setting the MPS. The value of obtaining patient feedback to assess the adequacy of resident communication is highlighted in research that engaged patients to provide feedback on communication skills to surgical residents.^21,22 Furthermore, patient satisfaction questions regarding resident performance have been added to Press Ganey surveys.²³

Two prior studies incorporated patients for standard setting.^24,25 In the first, patients determined the MPS for procedures performed by patients and/or caregivers (LVAD battery changes, controller changes, and dressing change).²⁴ In the second, the authors compared both physician and patient responses to inform ultimate determination of an MPS for a communication survey.²⁵ However, to date, patients have yet to be used as judges to set an MPS for physician performance on a task using the Mastery Angoff method.

The first goal of this project is to describe the process of utilizing patients and emergency medicine (EM) residency program directors (PDs) as judges to set the MPS for an uncertainty communication mastery learning curriculum. The second goal is to compare the MPS generated by the 2 different judge panels using the Mastery Angoff method.

Methods

This work is part of a larger project to develop and test a simulation-based mastery learning curriculum to teach EM residents to have more effective discharge conversations for patients with diagnostic uncertainty. We developed a 21-item Uncertainty Communication Checklist²⁶ for use in our simulation-based training and assessment of these physicians' communication skills.

We recruited 2 separate panels of patients and physicians to serve as judges for the standard-setting process. The number of judges for each panel was based on previously published recommendations.¹²

Inclusion criteria for patients to participate as a standard-setting judge included being an English-speaking adult (≥ 18 years) with a recent emergency department (ED) visit within either the Thomas Jefferson University (TJU) Health System or Northwestern Memorial (NM) Hospital that resulted in discharge with a symptom-based diagnosis (ie, abdominal pain). Patient exclusion criteria included being admitted to the hospital as a result of their most recent ED visit; undergoing medical clearance for a detox center or any involuntary court or magistrate order; in police custody or currently incarcerated; 4 or more visits to the ED within the month preceding the study recruitment period; having a major communication barrier such as visual, hearing, or cognitive impairment (determined by 6-item screener)²⁷ that would compromise their ability to give written informed consent; or being unwilling or unable to comply with study protocol requirements, determined from research personnel's best judgment.

An electronic health record report was generated at both health systems to identify potentially eligible patients. Two study physicians independently reviewed the report to identify patients discharged with a symptom-based diagnosis and created a randomly ordered recruitment list.²⁸ Trained research personnel contacted patients by telephone to explain the study and further assess eligibility. Interested and eligible patients were invited to participate in focus groups.

Inclusion criterion for physicians included serving as a current or former PD or associate PD for an accredited EM residency program. An initial recruitment e-mail was sent to 28 individuals selected by the study team through our professional network to balance program geography (Northeast, Midwest, South, West), PD gender, and 3- versus 4-year program designation. As the initial PD response led to a predominance of respondents from the Northeast and from 4-year programs, we sent a second recruitment e-mail to an additional 14 PDs from 3-year programs in the underrepresented 3 geographic areas to balance recruitment, resulting in the final sample.

Results

Two hundred forty patients who met inclusion criteria were screened; 105 were able to be contacted, of whom 8 were ineligible and 55 declined participation. Twenty-five patients (26% cooperation rate) agreed to participate as judges in 1 of 4 standard-setting focus groups (2 at TJU, 2 at NU) occurring from May 22 to May 31, 2018 (Figure). Each session included 5 to 8 judges and lasted 90 minutes. We approached 42 PDs, with 20 indicating potential interest, and ultimately 13 PDs (65% cooperation rate) participated as judges in 1 of 4 standard-setting video conferences from April 3 to June 26, 2018. Each session included 2 to 4 judges and lasted 60 minutes. See Tables 1 and 2 for patient and PD judge characteristics, respectively.

View Figure

• Download as Powerpoint
• Download Figure

Table 1 Patient Judge Panel Demographics (n = 25)

View Fullscreen

Table 2 Program Director Demographics (n = 13)

View Fullscreen

Patients assigned higher scores on 5 items, and physicians assigned higher scores on the remaining 16 items. Only 3 of 21 items had statistically significant differences between patient and physician raters (Table 3).

Table 3 Comparison of Standard Setting Between Scoring Groups

View Fullscreen

The patient judges set an MPS of 84.0% (range 45.2–96.2, SD 10.2); PD judges set an MPS of 88.2% (range 79.7–98.1, SD 5.5). The impact of the difference in the MPS between the 2 judge groups is 1 item: the patient MPS equated to a passing score of 18 out of 21 items correct, whereas the physician MPS equated to a passing score of 19 out of 21 items correct. The overall MPS, calculated as an average of these 2 results, was 86.1% (range 45.2–98.1, SD 9.0), equating to perfoming 19 of 21 items correctly.

Discussion

We demonstrated that patients are able to serve as judges for a task performed by physicians using the Mastery Angoff method. Further, in this study, patients established an MPS that is nearly identical to that generated by a panel of residency PDs.

For any given checklist, there is no gold standard for the MPS. Therefore, we cannot infer if patient- or physician-generated scores are “better” or “more accurate” on our Uncertainty Communication Checklist. Instead, we can only comment on the similarities or differences among judges' scores. We found the difference between scores generated by patients and physicians to result in only 1 more item needing to be performed correctly for the more stringent physician-generated score. This high level of agreement reflects the importance that both patients and physicians place on communication, particularly in the context of uncertainty at the time of ED discharge.

We observed a wider range of scores among patients, represented by a standard deviation of 10.2 versus 5.5 among PDs. One outlier patient was the primary driver of the difference in score range. Although statistically an outlier, the team believed that the low percentage estimations of this patient were not due to misunderstanding of the standard-setting task, but rather to their true beliefs about what could be expected of the residents achieving mastery (based on negative prior experiences). Therefore, we retained all scores for the calculations as they represented the patient perspective. Notably, removal of the scores from the patient outlier would not have altered the MPS.

Statistically significant differences among patient and PD scores were identified on only 3 of the 21 Uncertainty Communication Checklist items (items 2, 7, 8). Based on participant feedback during the standard-setting process, we believe several explanations exist for these differences. Patients reflected on their own experiences in the emergency setting, which had never included such interactions (eg, items 2 and 7). While it might seem that patients would frequently experience a discussion of alternate diagnoses (item 8), in an era imbued with fear of giving the “wrong” answer, physicians may not routinely share this information unless specifically guided. PDs, however, commented that while their residents did not engage in these communication tasks routinely, the tasks were concrete items that a well-prepared resident would be able to achieve if instructed properly.

With this work, we sought to extend patient involvement in communication skills assessment by demonstrating the feasibility of engaging patients as judges in mastery learning standard setting for topics in which they have appropriate content expertise. This study extends the reach of 2 previous studies that utilized patients as judges to establish an MPS,^24,25 and is the first to integrate patient perspectives through their participation as judges in the context of mastery learning for the assessment of physician performance.

Our findings are similar to those of Barsuk et al,²⁴ which reported a difference in an MPS set by patients and physicians of only 1 checklist item. However, the Wayne et al study²⁵ reported a much larger difference between patient and physician standards. Such differences may be due to the standard-setting approach used. The Wayne et al study used the traditional Angoff method to standard setting, which asks judges to conceptualize a “borderline” learner.²⁵ In contrast, our study used the Mastery Angoff method, asking judges to consider a “well-prepared” learner. Patients may be less familiar than physicians with what would constitute a “borderline” learner, yet better able to conceptualize a “well-prepared” learner, which could explain the similarity in scores. Further, in the Wayne et al study, judges were provided baseline performance data, which has been shown to influence standard-setting scores.^30,31 In our study, the focus on “mastery” and the lack of provision of baseline data may have removed the influence of previous personal experience (for both patients and PDs) and served to create a shared mental model between all judges of the ideal masterful performance of the skill.

Our methodology incorporated use of a video conference to facilitate conduct of the standard-setting sessions, which allowed for a panel of judges spanning the geographical and programmatic variation inherent in EM training programs, potentially decreasing their bias. Such broad representation may increase the generalizability of the MPS for this checklist beyond a single institution.

Limitations of this study include generalizability; although physician judges were recruited nationally, patients were English-speaking and recruited from 2 institutions, which reduces diversity. The total number of physician judges represents a small percentage (18%) of overall EM training programs. It is possible that additional judges may have changed the ultimate MPS.

As this study used only one approach for setting the MPS, future studies comparing standard-setting approaches with patient judges may elucidate differences among various these approaches.

Conclusions

This study demonstrated that using patient judges on a panel to establish an MPS for the Uncertainty Communication Checklist, using the Mastery Angoff method, for a diagnostic uncertainty communication mastery learning curriculum for resident physicians is feasible. In addition, patient and expert physician judges found nearly identical MPS.