All Hands on Deck—Structuring the CCC for Successful NAS Reporting

Setting and Problem

In preparation for the Accreditation Council for Graduate Medical Education's Next Accreditation System (NAS), programs have been challenged to become innovative in their approaches to assessment. The University of Minnesota internal medicine residency is a large program with 100 residents rotating at 3 major hospitals in the Twin Cities. We created Milestone-based evaluations and utilized new evaluation software in the 2013–2014 academic year. Our main challenges were engaging faculty at 3 different sites in the new evaluation process, and recruiting faculty to participate in the Clinical Competency Committee (CCC).

Intervention

New Milestone-based resident evaluations were presented to faculty at various departmental meetings and in focus groups. Through this, we introduced faculty to the NAS and the Internal Medicine Milestones. We revamped the existing CCC into a larger committee called the Resident Assessment and Progression Committee (RAP-C). In the new model, 4 subcommittees review 25 residents each and report to the RAP-C biannually. An associate program director (APD) chairs the RAP-C and affiliate site APDs lead the subcommittees. Each subcommittee is made up of 8 faculty reviewers, including program directors, core faculty, and selected educators from each affiliated site. They are paired in dyads, with up to 4 dyads in each subcommittee. Each dyad performs a paired portfolio assessment of their assigned 6 residents and maps each learner onto the 22 NAS reporting Milestones for internal medicine. Residents' assignments to each subcommittee are determined by the APD who conducts their semiannual review.

The dyads present their Milestone-mapping recommendations to the subcommittee, who discuss them and approve them or determine if further discussion is needed. Residents for whom the subcommittee reaches consensus are added to a “consent” agenda; residents for whom consensus cannot be reached at the subcommittee or whose performance is at an unsatisfactory level are discussed at the biannual RAP-C meeting. All Milestone mapping recommendations are forwarded on to the program director for an ultimate placement determination. The APD subcommittee leads provide feedback to the residents during the semiannual review.

We piloted the new approach in November 2013 with the intern class. Thirty-two faculty members were recruited to participate, and were given tools that included a document detailing the RAP-C process, a timeline, and the criteria to reach consensus. Resident evaluation reports were disseminated with an optional reviewer worksheet, designed to guide the reviewer in their analysis of the resident's performance.

Outcomes to Date

The RAP-C subcommittees met in early November following a 2-week period allowed to review individual resident evaluation reports. The meetings were well attended by faculty members and consensus was reached on 30 of the 32 interns. Two interns requiring further discussion were reviewed at the November RAP-C meeting, and appropriate plans were put in place for each. No closed votes were necessary and the program director agreed with the Committee's recommendations for all interns. From start to finish, the entire process took a little over 1 month and a time commitment by each faculty member of approximately 3 hours.

For the next RAP-C meeting, faculty will receive evaluation packets further in advance of the subcommittee meeting. We will also provide reviewers access to the electronic evaluation system, allowing them to check any new evaluations that came in after the evaluation reports were distributed. The entire process is expected to take nearly 2 months and faculty members will likely need 5 to 10 hours to review their 6 assigned residents. It is expected that, as faculty becomes more familiar with these assessments, this time commitment may decline.

Our RAP-C system serves as an important faculty development tool on Milestone-based assessments, resulted in increased faculty engagement with the program, provided a valuable opportunity for the faculty to fulfill their service requirements for promotion and tenure, and will allow for an informed, fair review of each resident's progression along the Internal Medicine Milestones.