Kaye AR, Salud LH, Domont ZB, Blossfield Iannitelli K, Pugh CM.
Northwestern University Feinberg School of Medicine, Department of Surgery.
Stud Health Technol Inform. 2011;163:271-3
PMID: 21335802
Abstract
This study introduces a novel way to implement simulation in medical education. We investigated the feasibility of integrating a newly developed breast examination simulator into a breast exam technique lecture while also collecting detailed data on medical students' breast exam skills. Results indicate that it is feasible to integrate simulation technology into the classroom environment and collect detailed performance data that can be analyzed and used for skills assessment.
PMID: 21335802
Monday, February 28, 2011
Friday, February 18, 2011
Comparison of Checklist and Anchored Global Rating Instruments for Performance Rating of Simulated Pediatric Emergencies
Adler, Mark D. MD; Vozenilek, John A. MD; Trainor, Jennifer L. MD; Eppich, Walter J. MD, MEd; Wang, Ernest E. MD; Beaumont, Jennifer L. MS; Aitchison, Pamela R. RN; Pribaz, Paul J. MS; Erickson, Timothy MD; Edison, Marcia PhD; McGaghie, William C. PhD
Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare:
February 2011 - Volume 6 - Issue 1 - pp 18-24
doi: 10.1097/SIH.0b013e318201aa90
Purpose: To compare the psychometric performance of two rating instruments used to assess trainee performance in three clinical scenarios.
Methods: This study was part of a two-phase, randomized trial with a wait-list control condition assessing the effectiveness of a pediatric emergency medicine curriculum targeting general emergency medicine residents. Residents received 6 hours of instruction either before or after the first assessment. Separate pairs of raters completed either a dichotomous checklist for each of three cases or the Global Performance Assessment Tool (GPAT), an anchored multidimensional scale. A fully crossed person × rater × case generalizability study was conducted. The effect of training year on performance is assessed using multivariate analysis of variance.
Results: The person and person × case components accounted for most of the score variance for both instruments. Using either instrument, scores demonstrated a small but significant increase as training level increased when analyzed using a multivariate analysis of variance. The inter-rater reliability coefficient was >0.9 for both instruments.
Conclusions: We demonstrate that our checklist and anchored global rating instrument performed in a psychometrically similar fashion with high reliability. As long as proper attention is given to instrument design and testing and rater training, checklists and anchored assessment scales can produce reproducible data for a given population of subjects. The validity of the data arising for either instrument type must be assessed rigorously and with a focus, when practicable, on patient care outcomes.
Simulation in Healthcare: The Journal of the Society for Simulation in Healthcare:
February 2011 - Volume 6 - Issue 1 - pp 18-24
doi: 10.1097/SIH.0b013e318201aa90
Purpose: To compare the psychometric performance of two rating instruments used to assess trainee performance in three clinical scenarios.
Methods: This study was part of a two-phase, randomized trial with a wait-list control condition assessing the effectiveness of a pediatric emergency medicine curriculum targeting general emergency medicine residents. Residents received 6 hours of instruction either before or after the first assessment. Separate pairs of raters completed either a dichotomous checklist for each of three cases or the Global Performance Assessment Tool (GPAT), an anchored multidimensional scale. A fully crossed person × rater × case generalizability study was conducted. The effect of training year on performance is assessed using multivariate analysis of variance.
Results: The person and person × case components accounted for most of the score variance for both instruments. Using either instrument, scores demonstrated a small but significant increase as training level increased when analyzed using a multivariate analysis of variance. The inter-rater reliability coefficient was >0.9 for both instruments.
Conclusions: We demonstrate that our checklist and anchored global rating instrument performed in a psychometrically similar fashion with high reliability. As long as proper attention is given to instrument design and testing and rater training, checklists and anchored assessment scales can produce reproducible data for a given population of subjects. The validity of the data arising for either instrument type must be assessed rigorously and with a focus, when practicable, on patient care outcomes.
Saturday, February 12, 2011
Use of a continuing medical education course to improve fellows' knowledge and skills in esophageal disorders
PubMed ID: 21309911
Dis Esophagus. 2011 Feb 10. doi: 10.1111/j.1442-2050.2010.01161.x. [Epub ahead of print]
Kim HC, Pandolfino JE, Komanduri S, Hirano I, Cohen ER, Wayne DB.
Departments of Gastroenterology andMedicine, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA.
Abstract
Advanced esophageal endoscopic procedures such as stricture dilation, hemostasis tools, and stent placement as well as high-resolution manometry (HRM) interpretation are necessary skills for gastroenterology fellows to obtain during their training. Becoming proficient in these skills may be challenging in light of higher complication rates compared with diagnostic procedures and infrequent opportunities to practice these skills. Our aim was to determine if intensive training during a continuing medical education (CME) course boosts the knowledge and skills of gastroenterology fellows in esophageal diagnostic test interpretation and performance of therapeutic procedures. This was a pretest-posttest design without a control group of a simulation-based, educational intervention in esophageal stricture balloon dilation and HRM interpretation. The participants were 24 gastroenterology fellows from 21 accredited US training programs. This was an intensive CME course held in Las Vegas, Nevada from August 7 to August 9, 2009. The research procedure had two phases. First, the subjects were measured at baseline (pretest) for their knowledge and procedural skill. Second, the fellows received 6 hours of education sessions featuring didactic content, instruction in HRM indications and interpretation, and deliberate practice using an esophageal stricture dilation model. After the intervention, all of the fellows were retested (posttest). A 17-item checklist was developed for the esophageal balloon dilation procedure using relevant sources, expert opinion, and rigorous step-by-step procedures. Nineteen representative HRM swallow studies were obtained from Northwestern's motility lab and formed the pretest and posttest in HRM interpretation. Mean scores on the dilation checklist improved 81% from 39.4% (standard deviation [SD]= 33.4%) at pretest to 71.3% (SD = 29.5%) after simulation training (P < sd =" 16.4%)" sd =" 15.8%),">
Saturday, February 5, 2011
New directions in simulation-based surgical education and training: Validation and transfer of surgical skills, use of nonsurgeons as faculty...
New directions in simulation-based surgical education and training: Validation and transfer of surgical skills, use of nonsurgeons as faculty, use of simulation to screen and select surgery residents, and long-term follow-up of learners
Daniel J. Scott MD, FACSa, Carla M. Pugh MD, PhD, FACSb, E. Matthew Ritter MD, FACSc, Lenworth M. Jacobs MD, MPH, FACSd, Carlos A. Pellegrini MD, FACS, FRCSI (Hon)e and Ajit K. Sachdeva MD, FRCSC, FACSf, ,
a University of Texas Southwestern Medical Center, Dallas, TX
b Northwestern University Feinberg School of Medicine, Chicago, IL
c Uniformed Services University, Bethesda, MD
d University of Connecticut School of Medicine, Hartford, CT
e University of Washington, Seattle, WA
f American College of Surgeons, Chicago, IL
Accepted 16 November 2010. Available online 5 February 2011.
The Consortium of American College of Surgeons-Accredited Education Institutes was created to explore new opportunities in simulation-based surgical education and training beyond the scope of individual accredited institutes. During the Third Annual Meeting of the Consortium of American College of Surgeons-Accredited Education Institutes Consortium, 4 work groups addressed the validation and transfer of surgical skills, the use of nonsurgeons as faculty, the use of simulation to screen and select surgery residents, and long-term follow-up of learners. The key elements from the deliberations and conclusions are summarized in this manuscript.
Article Outline
Validation and transfer of surgical skills
Use of nonsurgeons as faculty for simulation-based surgical education
Use of simulation to screen and select surgery residents
Long-term follow-up of learners after participation in simulation-based surgical education and training
Conclusions
References
Daniel J. Scott MD, FACSa, Carla M. Pugh MD, PhD, FACSb, E. Matthew Ritter MD, FACSc, Lenworth M. Jacobs MD, MPH, FACSd, Carlos A. Pellegrini MD, FACS, FRCSI (Hon)e and Ajit K. Sachdeva MD, FRCSC, FACSf, ,
a University of Texas Southwestern Medical Center, Dallas, TX
b Northwestern University Feinberg School of Medicine, Chicago, IL
c Uniformed Services University, Bethesda, MD
d University of Connecticut School of Medicine, Hartford, CT
e University of Washington, Seattle, WA
f American College of Surgeons, Chicago, IL
Accepted 16 November 2010. Available online 5 February 2011.
The Consortium of American College of Surgeons-Accredited Education Institutes was created to explore new opportunities in simulation-based surgical education and training beyond the scope of individual accredited institutes. During the Third Annual Meeting of the Consortium of American College of Surgeons-Accredited Education Institutes Consortium, 4 work groups addressed the validation and transfer of surgical skills, the use of nonsurgeons as faculty, the use of simulation to screen and select surgery residents, and long-term follow-up of learners. The key elements from the deliberations and conclusions are summarized in this manuscript.
Article Outline
Validation and transfer of surgical skills
Use of nonsurgeons as faculty for simulation-based surgical education
Use of simulation to screen and select surgery residents
Long-term follow-up of learners after participation in simulation-based surgical education and training
Conclusions
References
Tuesday, February 1, 2011
Monday, January 17, 2011
A report on the piloting of a novel computer-based medical case simulation for teaching and formative assessment of diagnostic laboratory testing
Clarence D. Kreiter, Thomas Haugen, Timothy Leaven,Christopher Goerdt, Nancy Rosenthal, William C. McGaghie and Fred Dee
Department of Pathology, University of Iowa Carver College of Medicine, Iowa City, IA, USA;
Northwestern University Feinberg School of Medicine, Chicago, IL, USA
Citation: Medical Education Online 2011, 16: 5646 - DOI: 10.3402/meo.v16i0.5646
Objectives: Insufficient attention has been given to how information from computer-based clinical case simulations is presented, collected, and scored. Research is needed on how best to design such simulations to acquire valid performance assessment data that can act as useful feedback for educational applications. This report describes a study of a new simulation format with design features aimed at improving both its formative assessment feedback and educational function.
Methods: Case simulation software (LabCAPS) was developed to target a highly focused and well-defined measurement goal with a response format that allowed objective scoring. Data from an eight-case computerbased performance assessment administered in a pilot study to 13 second-year medical students was analyzed using classical test theory and generalizability analysis. In addition, a similar analysis was conducted on an administration in a less controlled setting, but to a much large sample (n 143), within a clinical course that utilized two random case subsets from a library of 18 cases.
Results: Classical test theory case-level item analysis of the pilot assessment yielded an average case discrimination of 0.37, and all eight cases were positively discriminating (range 0.11 0.56). Classical test theory coefficient alpha and the decision study showed the eight-case performance assessment to have an observed reliability of s G 0.70. The decision study further demonstrated that a G 0.80 could be attained with approximately 3 h and 15 min of testing. The less-controlled educational application within a large medical class produced a somewhat lower reliability for eight cases (G 0.53). Students gave high ratings to the logic of the simulation interface, its educational value, and to the fidelity of the tasks.
Conclusions: LabCAPS software shows the potential to provide formative assessment of medical students’ skill at diagnostic test ordering and to provide valid feedback to learners. The perceived fidelity of the performance tasks and the statistical reliability findings support the validity of using the automated scores for formative assessment and learning. LabCAPS cases appear well designed for use as a scored assignment, for stimulating discussions in small group educational settings, for self-assessment, and for independent learning. Extension of the more highly controlled pilot assessment study with a larger sample will be needed to confirm its reliability in other assessment applications.
Keywords: computer-based simulation; clinical skills assessment; formative assessment; laboratory medicine; performance
assessment
Department of Pathology, University of Iowa Carver College of Medicine, Iowa City, IA, USA;
Northwestern University Feinberg School of Medicine, Chicago, IL, USA
Citation: Medical Education Online 2011, 16: 5646 - DOI: 10.3402/meo.v16i0.5646
Objectives: Insufficient attention has been given to how information from computer-based clinical case simulations is presented, collected, and scored. Research is needed on how best to design such simulations to acquire valid performance assessment data that can act as useful feedback for educational applications. This report describes a study of a new simulation format with design features aimed at improving both its formative assessment feedback and educational function.
Methods: Case simulation software (LabCAPS) was developed to target a highly focused and well-defined measurement goal with a response format that allowed objective scoring. Data from an eight-case computerbased performance assessment administered in a pilot study to 13 second-year medical students was analyzed using classical test theory and generalizability analysis. In addition, a similar analysis was conducted on an administration in a less controlled setting, but to a much large sample (n 143), within a clinical course that utilized two random case subsets from a library of 18 cases.
Results: Classical test theory case-level item analysis of the pilot assessment yielded an average case discrimination of 0.37, and all eight cases were positively discriminating (range 0.11 0.56). Classical test theory coefficient alpha and the decision study showed the eight-case performance assessment to have an observed reliability of s G 0.70. The decision study further demonstrated that a G 0.80 could be attained with approximately 3 h and 15 min of testing. The less-controlled educational application within a large medical class produced a somewhat lower reliability for eight cases (G 0.53). Students gave high ratings to the logic of the simulation interface, its educational value, and to the fidelity of the tasks.
Conclusions: LabCAPS software shows the potential to provide formative assessment of medical students’ skill at diagnostic test ordering and to provide valid feedback to learners. The perceived fidelity of the performance tasks and the statistical reliability findings support the validity of using the automated scores for formative assessment and learning. LabCAPS cases appear well designed for use as a scored assignment, for stimulating discussions in small group educational settings, for self-assessment, and for independent learning. Extension of the more highly controlled pilot assessment study with a larger sample will be needed to confirm its reliability in other assessment applications.
Keywords: computer-based simulation; clinical skills assessment; formative assessment; laboratory medicine; performance
assessment
Friday, January 7, 2011
Faculty evaluation of simulation-based modules for assessment of intraoperative decision making.
Pugh CM, Darosa DA, Santacaterina S, Clark RE.
Surgery. 2011 Jan 7. [Epub ahead of print]
Northwestern University, Chicago, IL.
PMID: 21216420 [PubMed - as supplied by publisher]
Abstract
BACKGROUND: Previous studies using simulation-based curricula have focused largely on technical skills. We developed a set of simulation-based modules that focus on intraoperative decision making. The objective of this study was to conduct a faculty evaluation of: (1) the usefulness of 4 newly developed, simulation-based modules; (2) the curricular need to train and assess intraoperative decision making skills of the residents; and (3) potential for resident benefit.
METHODS: Simulation-based modules were developed using a cognitive task analysis (CTA) framework. The CTA framework involved faculty interviews focusing on 4 operative tasks that span a range of complexity: (1) creation of small and large bowel stoma, (2) laparoscopic ventral hernia repair, (3) pancreaticojejunostomy, and (4) lymph node biopsy during a mediastinoscopy. An experienced psychologist conducted task-specific, one-on-one interviews with fellowship-trained specialists who perform these operations in their practice. Two faculty were interviewed for each procedure. The interviews lasted a minimum of 1 hour and focused on critical decisions, error prevention, error recognition, and error rescue strategies. The coded interview summaries were used as development guides for the simulation-based learning modules. Each module included locally developed physical models for the simulated operative tasks combined with oral and paper-based questions. The physical models were fabricated in such a way that simulated operative tasks could be performed using standard surgical instruments. To assess the newly developed simulation-based modules, 8 volunteer faculty (50% overlap with the interview pool) participated in a simulation-based exercise during a one-on-one session and then completed an 8-item survey cast on a 5-point Likert agreement scale (1 = strongly disagree, 5 = strongly agree). One of the items was worded negatively to ensure internal consistency. An independent observer recorded faculty session times and assessed faculty engagement in the task (1 = not engaged, 5 = extremely engaged).
RESULTS: On average, faculty spent 60 minutes completing each simulation-based exercise. Over 80% of this time was spent performing the operative tasks as they would during a real-life procedure. Mean engagement rating was 4.9 (maximum 5.0, SD = 0.3). Survey results show strong agreement on the importance of training and assessing intraoperative decision making, and that residents would likely benefit from the simulation-based modules.
CONCLUSION: We developed 4 high-fidelity simulation-based modules to assess intraoperative decision making. Faculty agree strongly on the importance and need for additional modules.
Subscribe to:
Posts (Atom)