Are United States Medical Licensing Exam Step 1 and 2 Scores Valid Measures for Postgraduate Medical Residency Selection Decisions?

Are United States Medical Licensing Exam Step 1 and 2 Scores Valid Measures for Postgraduate Medical Residency Selection Decisions?

Authors: McGaghie WC, Cohen ER, Wayne DB

PURPOSE: United States Medical Licensing Examination (USMLE) scores are frequently used by residency program directors when evaluating applicants. The objectives of this report are to study the chain of reasoning and evidence that underlies the use of USMLE Step 1 and 2 scores for postgraduate medical resident selection decisions and to evaluate the validity argument about the utility of USMLE scores for this purpose.
METHOD: This is a research synthesis using the critical review approach. The study first describes the chain of reasoning that underlies a validity argument about using test scores for a specific purpose. It continues by summarizing correlations of USMLE Step 1 and 2 scores and reliable measures of clinical skill acquisition drawn from nine studies involving 393 medical learners from 2005 to 2010. The integrity of the validity argument about using USMLE Step 1 and 2 scores for postgraduate residency selection decisions is tested.
RESULTS: The research synthesis shows that USMLE Step 1 and 2 scores are not correlated with reliable measures of medical students', residents', and fellows' clinical skill acquisition.
CONCLUSIONS: The validity argument about using USMLE Step 1 and 2 scores for postgraduate residency selection decisions is neither structured, coherent, nor evidence based. The USMLE score validity argument breaks down on grounds of extrapolation and decision/interpretation because the scores are not associated with measures of clinical skill acquisition among advanced medical students, residents, and subspecialty fellows. Continued use of USMLE Step 1 and 2 scores for postgraduate medical residency selection decisions is discouraged.
PMID: 21099388 [PubMed - as supplied by publisher]

Long-term retention of central venous catheter insertion skills after simulation-based mastery learning.

Acad Med. 2010 Oct;85(10 Suppl):S9-12.

20881713

Barsuk JH, Cohen ER, McGaghie WC, Wayne DB.
Northwestern University Feinberg School of Medicine, Division of HospitalMedicine, Chicago, IL 60611, USA. jbarsuk@nmh.org


BACKGROUND: Simulation-based mastery learning (SBML) of central venous catheter(CVC) insertion improves trainee skill and patient care. How long skills areretained is unknown.

METHOD: This is a prospective cohort study. Subjects completed SBML and wererequired to meet or exceed a minimum passing score (MPS) for CVC insertion on aposttest. Skills were retested 6 and 12 months later and compared with posttestresults to assess skill retention.

RESULTS: Forty-nine of 61 (80.3%) subjects completed follow-up testing. Although performance declined from posttest where 100% met the MPS for CVC insertion,82.4% to 87.1% of trainees passed the exam and maintained their high performance up to one year after training.

CONCLUSIONS: Skills acquired from SBML were substantially retained during oneyear. Individual performance cannot be predicted, so programs should use periodictesting and refresher training to ensure competence.PMID: 20881713 [PubMed - indexed for MEDLINE]

Cost savings from reduced catheter-related bloodstream infection after simulation-based education for residents in a medical intensive care unit.

Cost savings from reduced catheter-related bloodstream infection after simulation-based education for residents in a medical intensive care unit.

Cohen ER, Feinglass J, Barsuk JH, Barnard C, O'Donnell A, McGaghie WC, Wayne DB

Simul Healthc. 2010 Apr;5(2):98-102

Authors: Cohen ER, Feinglass J, Barsuk JH, Barnard C, O'Donnell A, McGaghie WC, Wayne DB

Interventions to reduce preventable complications such as catheter-related bloodstream infections (CRBSI) can also decrease hospital costs. However, little is known about the cost-effectiveness of simulation-based education. The aim of this study was to estimate hospital cost savings related to a reduction in CRBSI after simulation training for residents.

PMID: 20389233 [PubMed - in process]

Internal medicine residency graduates' perceptions of the systems-based practice and practice-based learning and improvement competencies.

Internal medicine residency graduates' perceptions of the systems-based practice and practice-based learning and improvement competencies.

Didwania A, McGaghie WC, Cohen E, Wayne DB
Teach Learn Med. 2010 Jan;22(1):33-6

Authors: Didwania A, McGaghie WC, Cohen E, Wayne DB

Resident education in Systems-Based Practice (SBP) and Practice-Based Learning and Improvement (PBLI) is required but underemphasized.

PMID: 20391281 [PubMed - indexed for MEDLINE]