Development of an Ambulatory Geriatrics Knowledge Examination for Internal Medicine Residents
Abstract
Background
The number of older adults needing primary care exceeds the capacity of trained geriatricians to accommodate them. All physicians should have basic knowledge of optimal outpatient care of older adults to enhance the capacity of the system to serve this patient group. To date, there is no knowledge-assessment tool that focuses specifically on geriatric ambulatory care.
Objective
We developed an examination to assess internal medicine residents' knowledge of ambulatory geriatrics.
Methods
A consensus panel developed a 30-question examination based on topics in the American Board of Internal Medicine (ABIM) Certification Examination Blueprint, the ABIM in-training examinations, and the American Geriatrics Society Goals and Objectives. Questions were reviewed, edited, and then administered to medical students, internal medicine residents, primary care providers, and geriatricians.
Results
Ninety-eight individuals (20 fourth-year medical students, 57 internal medicine residents, 11 primary care faculty members, and 10 geriatrics fellowship-trained physicians) took the examination. Based on psychometric analysis of the results, 5 questions were deleted because of poor discriminatory power. The Cronbach α coefficient of the remaining 25 questions was 0.48; however, assessment of interitem consistency may not be an appropriate measure, given the variety of clinical topics on which questions were based. Scores increased with higher levels of training in geriatrics (P < .001).
Conclusion
Our preliminary study suggests that the examination we developed is a reasonably valid method to assess knowledge of ambulatory geriatric care and may be useful in assessing residents.
Editor's Note: The online version of this article contains the examination questions used in the study.
Introduction
Individuals 65 years and older currently represent approximately 13% of the US population, and this number is expected to grow to 1 in 5 by 2030.1 The increasing number of chronically ill, older adults will require experienced primary care and geriatrics physicians. However, the number of medical students matching to primary care internal medicine programs and family medicine programs has dropped 33% and 35%, respectively,2 and just 22% of the graduating internal medicine residents from October 2009 through October 2011 planned to pursue a career in general internal medicine.3 Given these figures, it is important to ensure that all medical residents, even those who intend to subspecialize in other areas, have a strong background in geriatric medicine.
Recognizing this need, the American Board of Internal Medicine (ABIM) has placed greater emphasis on education in geriatrics. The current ABIM Certification Examination devotes 10% of the questions to geriatrics, compared with 2% in 1980.4 Several attempts have been made to create tools to evaluate geriatrics knowledge, among them Palmore's Facts on Aging Quiz (1977); the University of California, Los Angeles, Geriatrics Knowledge Examination (1997); and the University of Michigan Geriatrics Clinical Decision-Making Assessment (2006).5–8 Although each of these assessments serves a unique purpose, updates are needed, and none focuses on the ambulatory care of the older adult. Our goal was to develop and validate a novel knowledge-assessment tool with a deliberate focus on ambulatory geriatrics medical care.
Methods
Instrument Development
The examination was designed to focus on a variety of geriatrics content areas relevant primarily to the ambulatory setting, including polypharmacy, dementia, and wound care, with a focus on medical knowledge, rather than sociodemographic issues and social support. We developed a list of topic areas addressed in the American Geriatrics Society Goals and Objectives, ABIM Blueprint for Certification Examination, and in-training examinations during the past 5 years.4,9 From this we derived, by consensus, the items most clinically relevant for the education of internal medicine residents and their future practices.
We then created a 30-question examination, mimicking the multiple-choice, case-based style of ABIM board questions and the guidelines from the National Board of Medical Examiners.10 We limited the length of the examination to ensure feasibility of completion on a busy clinical rotation.
The initial 30 questions were reviewed by 2 geriatricians and 2 internal medicine faculty physicians, all of whom were core faculty members in the internal medicine residency program. Based on the feedback received, the questions were edited and then reviewed again by 2 other geriatricians.
Analysis
Item Analysis
The preliminary version was administered to 55 internal medicine residents and a convenience sample of medical students, primary care faculty members (internal medicine and family medicine), and fellowship-trained, board-certified geriatricians. The responses from this administration underwent psychometric analysis. The item-difficulty and item-discrimination indices were determined based on the number of examinees who answered each question correctly and the point biserial correlation coefficient, respectively.11 In addition, Cronbach coefficient α was used to test reliability of the individual questions.11 Five questions were discarded based on poor discriminatory power, compared with the individual's total score and level of difficulty, and the quality of question construction based on the group's combined percentage of correct answers. This shortened examination underwent a second item analysis.
Ninety-eight individuals participated in the study, including 20 fourth-year medical students (20%), 57 internal medicine residents (58%), 11 ambulatory primary care faculty members (11%), and 10 geriatrics fellowship–trained physicians (10%).
Our study was declared exempt by the University of Kansas Medical Center Human Subjects Committee.
Examination Analysis
The final version of the examination consisted of 25 items, each of which was scored as correct or incorrect. An examination score was generated by summing the total number of correct responses, and the correct proportion was determined by dividing that number by the total number of items.
Results were analyzed within 3 groups: (1) learners (medical students and residents), (2) ambulatory primary care faculty members, and (3) geriatricians. Descriptive statistics were generated, both overall and by group, for the examination score and the percentage of respondents answering correctly. Mean, standard deviation, and minimums and maximums were included. To assess for trends in the anticipated order, simple linear regression was used, assigning scores in increments of 1 to the groups. The model assumptions were assessed by visual inspection of box plots, estimated variances across groups, and a quantile-quantile plot.
Results
Item Analysis of Resident Examinations
The 25-item examination had a Cronbach coefficient α of 0.48 based on examinations administered to interns and residents. However, it was not considered a homogenous test for which a Cronbach coefficient α is fully applicable, given the large variety of clinical topics on which the questions were based. These findings were congruent with other published geriatrics examinations.5,6
Analysis of All Examinations
We grouped all learners' scores together because we determined that the knowledge levels of fourth-year medical students and that of residents were similar. The medical students completing the examination had previously completed a rotation in geriatrics during their third-year curriculum. Only a few of the residents who took the examination had previously completed a geriatrics rotation.
Examination scores improved as training levels increased (figure; P < .001). The learners' mean percentage was 0.61 ± 0.13 (range, 0.28–0.88). The ambulatory primary care faculty members' mean percentage was 0.70 ± 0.11 (range, 0.48–0.84). The geriatricians' mean percentage was 0.80 ± 0.06 (range, 0.72–0.84). No participant had a perfect score, and the lowest overall score was 0.28.
Citation: Journal of Graduate Medical Education 5, 4; 10.4300/JGME-D-13-00123.1
Discussion
Our goal was to fill a gap in existing methods to assess residents' knowledge of ambulatory care for older adults. Given the importance of ambulatory care for primary care and subspecialty medicine, residency programs should have a reliable and validated tool to assess residents' knowledge in this area. Our 25-question examination meets those requirements and should be applicable to internal medicine and family medicine residencies.
The average score among residents was 15 (60%), leaving ample room for improvement on the same examination given as a postrotation assessment. Content validity was supported by the instrument-development process as described, including use of items from validated examinations and topic selection by local experts.12,13 In general, the postgraduate year (PGY)-1 group had a lower score than did a combined PGY-2 and PGY-3 group.
Our study has several limitations. It was conducted at a single institution with a single set of students and residents, limiting generalizability. We also grouped all learners together in our analyses despite differing amounts of experience in medicine.
Conclusion
Overall, our examination is a promising tool to assess knowledge and should be evaluated at other institutions among internal medicine and family medicine residents to further establish its validity in varied settings. Future work should establish the examination's ability to measure knowledge gained during a 1-month ambulatory geriatrics rotation for internal medicine residents.
Percentage Correct Among Each Group of Examinees on an Ambulatory Geriatrics Examination
Author Notes
All authors are with the University of Kansas School of Medicine. Jessica L. Kalender-Rich, MD, is Assistant Professor of Internal Medicine and the Landon Center on Aging, Geriatrician and Core Faculty for the Internal Medicine Residency, and Faculty Lead for the Required Geriatrics Rotation for Internal Medicine Residents; Jonathan D. Mahnken, PhD, is Associate Professor of Biostatistics; Lei Dong, MS, is Senior Research Analyst in Biostatistics; Anthony M. Paolo, PhD, is Director of Assessment and Evaluation in Biostatistics; Deon Cox Hayley, DO, is Associate Professor of Internal Medicine and Landon Center on Aging and Geriatrician and Core Faculty for the Geriatrics Fellowship; and at the time of writing Sally K. Rigler, MD, MPH, was Professor of Scholarly, Academic, and Research Mentoring and of Internal Medicine, a Geriatrician, and a Research Scholar.
Presented in the Presidential Poster Session at the Annual Scientific Meeting of the American Geriatrics Society May 2–5, 2012, Seattle, WA.
The authors wish to thank the Office of Scholarly, Academic, and Research Mentoring, Department of Internal Medicine, for biostatistical and mentoring support.
Funding: The authors report no external funding source for this study.
