Use of Human Patient Simulation and Validation of the Team Situation Awareness Global Assessment Technique (TSAGAT): A Multidisciplinary Team Assessment Tool in Trauma Education

Objective

Situation awareness (SA) is a vital construct for decision making in intense, dynamic environments such as trauma resuscitation. Human patient simulation (HPS) allows for a safe environment where individuals can develop these skills. Trauma resuscitation is performed by multidisciplinary teams that are traditionally difficult to globally assess. Our objective was to create and validate a novel tool to measure SA in multidisciplinary trauma teams using a HPS—the Team Situation Awareness Global Assessment Technique (TSAGAT).

Setting

Memorial University Simulation Centre.

Design/Participants

Using HPS, 4 trauma teams completed 2 separate trauma scenarios. Student, junior resident, senior resident, and attending staff teams each had 3 members (trauma team leader, nurse, and airway manager). Individual SAGATs were developed by experts in each respective field and contained shared and complimentary knowledge questions. Teams were assessed with SAGAT in real time and with traditional checklists using video review. TSAGAT was calculated as the sum of individual SAGAT scores and was compared with the traditional checklist scores.

Results

Shared, complimentary, and TSAGAT scores improved with increasing team experience. Differences between teams for TSAGAT and complimentary knowledge were statistically significant (p < 0.05). Mean checklist differences between teams also reached statistical significance (p < 0.05). TSAGAT scores correlated strongly with traditional checklist scores (Pearson correlation r = 0.996). Interrater reliability for the checklist tool was high (Pearson correlation r = 0.937).

Conclusion

TSAGAT is the first valid and reliable assessment tool incorporating SA and HPS for multidisciplinary team performance in trauma resuscitation. TSAGAT could compliment or improve on current assessment methods and curricula in trauma and critical care and provides a template for team assessment in other areas of surgical education.

Introduction

The assessment of trainee performance in dynamic, intense clinical situations such as trauma resuscitation has been traditionally difficult. Properly functioning trauma teams, as part of a trauma system, are vital to improved outcomes in trauma resuscitation and have been shown to significantly reduce the rate of preventable trauma deaths.¹ Trauma resuscitation is usually carried out by the coordinated efforts of multidisciplinary trauma teams. Team members typically include a trauma team leader, an airway manager, a trauma nurse, technicians, and various other subspecialists based on patient presentation. Although there is knowledge overlap between specialties during trauma resuscitation, each team member also possesses unique complimentary knowledge and skills that facilitate successful team performance. There are currently few assessment tools that address the challenges in measuring team performance.

The current standard assessment tool in trauma education is a checklist.² The checklist puts emphasis on results with minimal insight into process.³ A trainee may perform well on a checklist despite a series of misconceptions, misinterpretations, and mistakes. Checklists may reward thoroughness rather than competence and may not allow for recognition of alternative approaches to the problem.⁴ Failure of this assessment tool to recognize these mistakes could result in poor trauma management in real-life scenarios. In addition, checklists are mostly intended for trauma team leaders, and therefore they do not necessarily apply to other members of a multidisciplinary trauma team. The checklist has been validated for individual trainee assessment. It is not designed to assess team performance. There is a need for trauma skills assessment evolution.

The optimal method of multidisciplinary trauma team assessment is yet to be established. A part of the complexity of the assessment lies in the many aspects of team performance, including communication skills, leadership, assertiveness, and situation awareness (SA). Options commonly employed include video review, observer review, medical notes review, or the use of simulation.⁵ New focus on the integration of simulation into trauma and critical care curricula has been enabled by the design of the human patient simulators (HPS) (Medical Education Technology Incorporated, Sarasota, FL). This life-size mannequin shares many realistic features with real patients and has been used extensively in training and assessment worldwide.

HPS has been shown to have training advantages over traditional moulage scenarios,⁶ and the use of HPS in conjunction with Advanced Trauma Life Support (ATLS) teaching appears to improve the development of trauma management skills.6, 7 HPS has shown significant potential to facilitate assessment of individual and team performance in practical trauma management.8, 9 As the environment in which individual and team trauma skills are practiced has evolved, the assessment methods for these skills have begun to evolve as well. SA and the Situation Awareness Global Assessment Technique (SAGAT) have recently emerged as areas of interest in trauma and critical care.3, 10, 11, 12

SA is defined as the perception of elements in the environment within a volume of time and space, the comprehension of their meaning, and the projection of their status in the near future.¹³ This psychological construct is critical to decision making in intense, dynamic environments. SA has been extensively studied in aviation, military, and nuclear power plant operations but only recently in medicine. Trauma resuscitation exemplifies the dynamic, stressful environments where SA is critical to good outcomes. SA is subdivided into 3 levels of understanding.

Level I SA refers to perceptions of elements in the environment.¹³ This includes all data and stimuli that appeal to the 5 senses. Examples of level I SA in trauma resuscitation would include pulse, blood pressure, airway status, and significant injuries.

Level II SA involves comprehension of level I stimuli. The trainee builds on the data they acquired during their initial patient assessment, for example, a rapid heart rate and low blood pressure may indicate hypovolemia secondary to ongoing blood loss.

Level III SA is achieved when a trainee makes projections based on their understanding of Level I and Level II information. Projection leads to predictions and decision making about events or actions that may occur or be required in the near future.

Endsley defines team SA as the degree to which each team member possesses the SA required for his or her responsibilities.¹⁰ SA is vital to individual performance, and team SA is critical to good team performance. Team members may have different subgoals in a given trauma resuscitation. Team SA can be subdivided into 2 types: (1) shared SA, in which team members possess the same SA and (2) complimentary SA, which represents unique, specialty-specific SA necessary for good team performance¹⁰ (Fig. 1).

The SAGAT is a tool designed to assess trainees based on the 3 levels of SA (perception, comprehension, and projection).¹³ This assessment method has been used in a variety of complex, dynamic environments and has recently been validated for individual assessment in trauma education.¹¹ Just as with individual performance, it would be useful to develop an assessment tool evaluating team performance. Team SAGAT (TSAGAT) could provide valuable information. Good or poor team performance and management decisions could be detected and analyzed to provide constructive feedback. Pervasive deficiencies in team SA across subjects could be useful to identify problems with current training programs.

In conducting this study, our goals were (1) to develop the TSAGAT assessment tool, (2) to use the TSAGAT tool to assess multidisciplinary trauma teams performing simulated trauma resuscitation, (3) to analyze TSAGAT scores across different levels of experience so as to establish construct validity for the tool, (4) to compare the TSAGAT scores with a traditional checklist assessment tool, and (5) to analyze participants overall satisfaction with the TSAGAT and with the HPS by way of an exit questionnaire.