Elsevier

Ambulatory Pediatrics

Volume 7, Issue 2, 1 March 2007, Pages 182-186
Ambulatory Pediatrics

Original article
Development and Evaluation of High-Fidelity Simulation Case Scenarios for Pediatric Resident Education

https://doi.org/10.1016/j.ambp.2006.12.005Get rights and content

Objective

Pediatric residency programs need objective methods of trainee assessment. Patient simulation can contribute to objective evaluation of acute care event management skills. We describe the development and validation of 4 simulation case scenarios for pediatric resident evaluation.

Methods

We created 4 pediatric simulation cases: apnea, asthma, supraventricular tachycardia, and sepsis. Each case contains a scenario and an unweighted checklist. Case and checklist development began by reaching expert consensus about case content followed by 92 pilot simulation sessions used for content revision and rater training. After development, 54 first-and second-year pediatric residents participated in 108 simulation test cases to assess the validity of data from these tools for our population. We report outcomes for interrater reliability, discriminant validity, and the impact of potential confounding factors on validity estimates.

Results

Interrater reliability (κ) ranged from 0.75 to 0.87. There were statistically and educationally significant differences in summary scores between first-and second-year residents for 3 of the 4 cases. Neither previous simulation exposure nor the order in which the cases were performed were found to be significant factors by multivariate analysis.

Conclusions

Simulation can be used to reliably measure and discriminate resident competencies in acute care management. Rigorous measurement development work is difficult and time-consuming. Done correctly, measurement development yields tangible and lasting benefits for trainees, faculty, and residency programs. Development studies that use systematic procedures and large trainee samples at multiple sites are the best approach to creating measurement tools that yield valid data.

Section snippets

Study Design

This study had a single-group, posttest-only quasi-experimental design.

Setting/Study Participants

We enrolled first and second year pediatric residents from Children’s Memorial Hospital’s (Chicago, Ill) pediatric residency program. During the development phase (July 2003 to June 2004), 51 residents were enrolled. In the evaluation phase (July 2004 to June 2005), 54 residents were enrolled, which represented 100% of the residents. First-year residents starting in 2003 participated in both years of the study. Participation

Results

All residents who were approached agreed to participate. Technical problems caused one session to be rescheduled. Seventy-two percent of subjects were female (78% and 67% of first- and second-year residents, respectively). Fifty-six percent reported previous simulation exposure in medical school (41% and 70% of first- and second-year residents, respectively).

Reliability data are listed in the Table. The mean adjusted κ coefficients for each case range from 0.75 to 0.87, consistent with

Discussion

In this study, we demonstrated that assessing residents with high-fidelity simulation cases can be accomplished with high reliability. Our case content was developed and reviewed by content experts in pediatric emergency medicine and medical education. The residents’ response processes produced by the cases using the simulator in the controlled environment closely approximated the behaviors needed to provide acute pediatric care in emergent situations. For 3 of the 4 cases, we demonstrated that

Acknowledgments

We are grateful for the provision of simulation laboratory time and personnel from the Patient Safety Simulation Center and the Department of Anesthesiology (M. Christine Stock, MD, Chair), Feinberg School of Medicine, Northwestern University. We also thank Leonard D. Wade for his help as simulation laboratory technician.

References (9)

  • G.E. Miller

    The assessment of clinical skills/competence/performance

    Acad Med

    (1990)
  • F.M. Nadel et al.

    Assessing pediatric senior residents’ training in resuscitation: fund of knowledge, technical skills, and perception of confidence

    Pediar Emerg Care

    (2000)
  • D.B. Wayne et al.

    Mastery learning of advanced cardiac life support skills by internal medicine residents using simulation technology and deliberate practice

    J Gen Intern Med

    (2006)
  • S.M. Downing

    Validity: on the meaningful interpretation of assessment data

    Med Educ

    (2003)
There are more references available in the full text version of this article.

Cited by (0)

1

Ms Siddall has served as a paid educator for METI, a simulator manufacturer.

View full text