Shop

The International Simulation Data Registry: Harnessing the Standardized Data

Aaron Calhoun
Cite The International Simulation Data Registry: Harnessing the Standardized Data icon
Share The International Simulation Data Registry: Harnessing the Standardized Data icon
$title

The International Simulation Data Registry (ISDR) was established in 2014 and is inspired by the increasingly numerous and widely used clinical registries. It aims to enhance simulation-based medical education (SBME) and improve clinical practices and patient outcomes. A recent strategic partnership was initiated between the Society for Simulation in Healthcare and the University of Toledo to update the ISDR to version 2.0. This collaboration aims to leverage a novel multimodal assessment platform, enhancing the ISDR’s capabilities. The project not only boosts medical training and patient care but also lays the groundwork for tackling research hurdles like standardized data sharing, and enhancing education within the simulation community, reinforcing its significance for the future of healthcare education and practice improvement.

DOI: https://doi.org/10.69079/simzine.H24.n14.00011

Relevant Background

The global healthcare simulation community has grown tremendously over the last two decades. Many simulation centers worldwide have contributed to education and research efforts to optimize simulation-based medical education (SBME) outcomes. However, SBME has not realized its full translational science potential by demonstrating that effective SBME approaches lead to improved clinical practices and, ultimately, better patient outcomes[1]. The lack of infrastructure for sharing standardized data from healthcare simulation activities and correlating it with practice worldwide presented a challenge in bridging the gap. 

The Emergence of the International Simulation Data Registry

The International Simulation Data Registry (ISDR) was established in 2014 to drive evidence-based SBME, better clinical practice, and improved patient outcomes[2]. The ISDR was inspired by the growing utilization of clinical registries, which facilitate care quality benchmarking, optimizing health outcomes, controlling costs, and conducting epidemiological research[3]. Accessible data registries are also vital for developing effective tools and technologies across all healthcare fields and specialties, especially as machine learning and artificial intelligence become more prevalent [4]. 

The initial rollout of the ISDR targeted standardized cases with clearly defined performance metrics based on American Heart Association Guidelines, such as pulseless cardiac arrest[2]. The ISDR was launched across 27 institutions, collecting standardized data including (but not limited to) patient simulator type, simulation setting, simulation participant discipline and years in training, and key objective performance measures (e.g., time to initiation of care). There was considerable interest and momentum within the ISDR after its launch; however, this was sidetracked due to the COVID-19 pandemic. 

Enhancing ISDR with PREPARE: ISDR 2.0

In June 2023, the Society for Simulation in Healthcare (SSH) and the University of Toledo (UToledo) initiated a strategic partnership to update the ISDR. The ISDR 2.0 will leverage a novel multimodal assessment platform (PREPARE) developed at the University of Toledo at Dr. Pappada’s laboratory[5]. The simplicity of the user interface enables standardization of curriculum generation and underlying assessments, making PREPARE highly synergistic with the vision and goals of the ISDR. A key feature of the platform is to collect, process, and synchronize data from multiple sources, such as observer-based performance assessments, operational/training environments (e.g., audio, video, simulator data), and learners. 

Addressing Simulation Challenges with ISDR 2.0

With the new functionalities afforded by PREPARE, ISDR 2.0 will address some of the challenges faced by the simulation community. One of these challenges is the lack of standardized scenarios, underlying data, and assessments to be collected during SBME activities.

The holy grail of simulation-based research is to implement outcome measures that can be collected in both simulated and real-world patient care settings.

Standardizing metrics and key performance measures in a way that can be seamlessly applied in both simulated and real environments ensures that key comparisons can be made. This will allow the community to evaluate how knowledge, skills, and practices trained and evaluated in simulated environments translate to real-world care settings.

Another challenge lies in the subjective performance evaluation that accompanies SBME, often resulting in a simplistic pass/fail assessment rather than acquiring richer data. Consequently, learners receive feedback that is often not personalized or tracked longitudinally to encourage reflection and inform repeated deliberate practice for improvement. The ISDR framework provides a standardized way to collect clinically relevant, objective data that can be consistently tracked over time. This data can be used for feedback and benchmarking across learners and simulation centers, optimizing the SBME processes and outcomes. Figure 1 shows the four key areas that ISDR will address. Two of these areas (**denoted) will be available with future development and advancement of the PREPARE platform through collaboration amongst the simulation community. 

[Figure 1. Challenges and Opportunities in Simulation-Based Medical Education and Assessment that ISDR will address]

The Society for Simulation in Healthcare and UToledo teams are collaborating on the initial deployment of ISDR 2.0, which introduces instructor-level assessment. These features enable standardized curriculum, scenario creation, and real-time assessment during curriculum delivery. Unlike the initial version, ISDR 2.0 allows assessments to be entered as the scenario unfolds rather than retrospectively. Furthermore, all assessments are time-stamped, facilitating the recording of objective performance metrics such as time to intervention or diagnosis.

PREPARE is platform-agnostic and can be utilized across various simulation centers or sites. Being web-based, users can access the platform on any device with internet browsing capabilities.

The instructor-level functionalities of ISDR 2.0 are detailed in an initial publication outlining the multimodal assessment capabilities of the PREPARE platform [6]. Additionally, certain learner-level data acquisition capabilities are available in ISDR 2.0, including customizable pre- and post-assessments, survey generation, and a collection of learner-level demographics and training frequencies.

PREPARE has additional capabilities that are part of a forward-looking development strategy [4]. The current capabilities of the PREPARE platform are shown in Figure 2, which demonstrates the data collection at the learner, simulation environment, and instructor levels. The automated data collection at the learner and simulation environment levels will not be included in the initial rollout of the ISDR, but may occur in subsequent phased roll-outs. 

[Figure 2. Overview of PREPARE’s Multimodal Assessment Capabilities]

PREPARE also enables real-time monitoring of simulation participants’ physiological data, which correlated with learner performance in our initial studies [6, 7]. Additionally, it can capture audio from the simulation environment to align verbalized events/interventions with expected events, employing a speech-to-text (STT) and natural language processing (NLP) module for event detection [8]. Future enhancements involve implementing computer vision for automated detection of procedural interventions and team dynamics.

The Future of Simulation in Healthcare with ISDR and PREPARE

Finally, as artificial intelligence (AI) continues to grow, existing registries are already being used to develop predictive models and machine learning (ML) approaches to optimize and personalize healthcare. The vision for the ISDR is to provide a cohesive and intuitive infrastructure to promote this for the simulation community. Figure 3 illustrates the vision for ISDR and PREPARE in generating the necessary data for this purpose. Although some of PREPARE’s functionalities will not be available in the initial release of ISDR 2.0, accumulating and sharing data through a common platform utilization will bring us closer to realizing this ultimate vision.

[Figure 3.  Vision of ISDR and PREPARE providing the necessary dataset and resources to personalize training/education.]

References

1. McGaghie, W.C., et al., Evaluating the impact of simulation on translational patient outcomes. Simulation in Healthcare, 2011. 6(7): p. S42-S47.

2. Calhoun, A.W., et al., Concepts for the simulation community: development of the International Simulation Data Registry. Simulation in Healthcare, 2018. 13(6): p. 427-434.

3. Hoque, D.M.E., et al., Impact of clinical registries on quality of patient care and clinical outcomes: a systematic review. PloS one, 2017. 12(9): p. e0183667.

4. Shah, P., et al., Artificial intelligence and machine learning in clinical development: a translational perspective. NPJ digital medicine, 2019. 2(1): p. 69.

5. University of Toledo College of Medicine and Life Sciences. PREPARE Multimodal Assessment and Intelligent Learning Management System Platform. 2024; Available from: https://www.utoledo.edu/med/research/prepare/.

6. Pappada, S., et al., Personalizing simulation-based medical education: the case for novel learning management systems. International Journal of Simulation in Healthcare, 2022.

7. Pappada SM, Owais MH*, Alvarado C, Schneiderman J, Brunner D, Casabianca A, Hofmann J, Papadimos TJ. . Novel Learning Management System to Revolutionize Generation, Administration, and Assessment of Simulation-Based Medical Education. in International Anesthesia Research Society (IARS)  Annual Meeting 2019. Montreal, Canada.

8. Paudel, P., S. Pappada, and L. Cheng. Automated Multimodal Performance Evaluation in Simulation-based Medical Education using Natural Language Processing. in Proceedings of the ACM/IEEE 14th International Conference on Cyber-Physical Systems (with CPS-IoT Week 2023). 2023.

READ ALSO

Aaron Calhoun
Author

Aaron Calhoun

MD, University of Louisville, Norton Children's Hospital View all Posts
Scott Pappada
Author

Scott Pappada

PhD, Department of Anesthesiology, University of Toledo College of Medicine and Life Sciences, United States Department of Bioengineering, University of Toledo College of Engineering, United States View all Posts
Serkan Toy
Author

Serkan Toy

PhD, Department of Basic Science Education, Department of Health Systems and Implementation Science. Virginia Tech Carilion School of Medicine, United States View all Posts
Michael Spooner
Author

Michael Spooner

MD, Mercy One North Iowa Heart Center, United States View all Posts

Leave a Reply

Join our newsletter

All the sim news, straight to your inbox.
Receive monthly the best research, innovations and stories on healthcare simulation

Join our newsletter