Scholarly article on topic 'The Command Team Experimental Test-bed Stage 1: Design and Build of a Submarine Command Room Simulator'

The Command Team Experimental Test-bed Stage 1: Design and Build of a Submarine Command Room Simulator Academic research paper on "Educational sciences"

CC BY-NC-ND
0
0
Share paper
Academic journal
Procedia Manufacturing
OECD Field of science
Keywords
{Submarine / Simulator / Fidelity / "Work load" / "Situation awareness"}

Abstract of research paper on Educational sciences, author of scientific article — Aaron Roberts, Neville Stanton, Daniel Fay

Abstract The command room system has developed across a century of submarine operations and so reflects a high state of evolution, but that does not mean that the system cannot be improved upon. Technological advances have resulted in the retrospective fitting of upgrades which may not have maximized the potential improvements offered. Future challenges for command teams in almost every domain include increasing amounts of data coupled with more automated systems and reduced manning. To optimise functionality new physical layouts, team structures, allocation of system functions, communication media, interfaces, and work design will be required. The aim of the ComTET (Command Team Experimental Test-Bed) project is to examine how a submarine command team currently functions, with specific regard to how information flows around the socio-technical system. This information shall be used to evaluate limitations in the current system, promote ideas concerning where reduced crewing might be possible and highlight how extra data might best be integrated into the system. Phase 1 of ComTET involved the creation of a submarine command room with high physical and task fidelity. The ComTET team has designed and built a submarine command room simulator that is a representation of the currently operational ASTUTE submarine. The simulator is comprised of 10 workstations each with two stacked monitors, various input devices and a headset linked to a multi-channel communications network. The simulation engine is a custom build of Dangerous Waters software, a naval warfare simulation game. The software features many operator-controllable units from on board a submarine, allowing the completion of individual submariner command team tasks simultaneously to fulfil global (team) mission objectives. The ComTET laboratory has a range of devices for recording the personal communications of each operator, in addition to video recordings of each operator and ambient voice/video recordings. This will facilitate the construction of social, task and information networks to examine the command room from a socio-technical perspective. The laboratory is also equipped with physiological recording devices so that the workload of operatives can be examined using psycho-physiological approaches alongside commonly used standardised measures of workload, situation awareness and cognitive function. The data collected shall be based around three scenarios which capture the primary operations routinely completed by submariners in high and low work load conditions.

Academic research paper on topic "The Command Team Experimental Test-bed Stage 1: Design and Build of a Submarine Command Room Simulator"

CrossMark

Available online at www.sciencedirect.com

ScienceDirect

Procedia Manufacturing 3 (2015) 2800 - 2807

6th International Conference on Applied Human Factors and Ergonomics (AHFE 2015) and the

Affiliated Conferences, AHFE 2015

The Command Team Experimental Test-bed Stage 1:Design and build of a submarine command room simulator

Aaron Roberts, Neville Stanton, Daniel Fay

University of Southampton, Boldrewood Campus, Southampton, United Kingdom

Abstract

The command room system has developed across a century of submarine operations and so reflects a high state of evolution, but that does not mean that the system cannot be improved upon. Technological advances have resulted in the retrospective fitting of upgrades which may not have maximized the potential improvements offered. Future challenges for command teams in almost every domain include increasing amounts of data coupled with more automated systems and reduced manning. To optimise functionality new physical layouts, team structures, allocation of system functions, communication media, interfaces, and work design will be required. The aim of the ComTET (Command Team Experimental Test-Bed) project is to examine how a submarine command team currently functions, with specific regard to how information flows around the socio-technical system. This information shall be used to evaluate limitations in the current system, promote ideas concerning where reduced crewing might be possible and highlight how extra data might best be integrated into the system. Phase 1 of ComTET involved the creation of a submarine command room with high physical and task fidelity. The ComTET team has designed and built a submarine command room simulator that is a representation of the currently operational ASTUTE submarine. The simulator is comprised of 10 workstations each with two stacked monitors, various input devices and a headset linked to a multi-channel communications network. The simulation engine is a custom build of Dangerous Waters software, a naval warfare simulation game. The software features many operator-controllable units from on board a submarine, allowing the completion of individual submariner command team tasks simultaneously to fulfil global (team) mission objectives. The ComTET laboratory has a range of devices for recording the personal communications of each operator, in addition to video recordings of each operator and ambient voice/video recordings. This will facilitate the construction of social, task and information networks to examine the command room from a socio-technical perspective. The laboratory is also equipped with physiological recording devices so that the workload of operatives can be examined using psycho-physiological approaches alongside commonly used standardised measures of workload, situation awareness and cognitive function. The data collected shall be based around three scenarios which capture the primary operations routinely completed by submariners in high and low work load conditions.

© 2015 Published by ElsevierB.V. Thisisanopen access article under the CC BY-NC-ND license

(http://creativecommons.Org/licenses/by-nc-nd/4.0/).

Peer-review under responsibility of AHFE Conference

Keywords:Submarine; Simulator; Fidelity; Work load; Situation awareness

2351-9789 © 2015 Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license

(http://creativecommons.org/licenses/by-nc-nd/4.0/).

Peer-review under responsibility of AHFE Conference

doi: 10.1016/j.promfg.2015.07.745

1. Introduction

1.1. Submarine command rooms

The command room system has developed across a century of submarine operations and so reflects a high state of evolution, but that does not mean that the system cannot be improved upon [1]. The long service life of submarines is one factor that has led to the routine retrospective implementation of advanced technologies which may not have maximized the utility of such upgrades. The future of submarine platforms is likely to include new and additional sensors (e.g. Unmanned Underwater Vehicles), improved communication bandwidths and new technologies. The use of increasingly sophisticated technologies in many industries is creating greater amounts of data that an operator must attend to [2]. Future challenges for command teams in almost every domain include the handling of greater volumes of data coupled with reduced manning. This will require new physical layouts, team structures, allocation of system functions, communication media, interfaces, job aids, and work design. A better understanding of the current interactions between individuals and teams, together with the effects of interventions, will help to improve the performance of command teams of the future [3]. Due to issues such as security, economic costs and time constraints there is not a wealth of published research examining submarine command teams and what exists focuses on current operational capacities rather than next generation ways of working.

1.2. The Command Team Experimental Test-bed

The goal of the ComTET (Command Teamwork Experimental Test-bed) project is to develop a test bed to undertake repeatable experiments to provide evidence to show where performance benefits may be gained in future multi-role submarine platforms. The work will provide knowledge concerning the distribution of tasks, communications and information flow across time between different artefacts (command team members and supporting technologies). An assessment of the cognitive capacities, Work Load (WL) and Situation Awareness (SA) of the command team will also be completed. Initial baseline studies will examine how submarines currently operate; these studies will then be used as a template to which future manipulations shall be compared. The work will flow in a cyclical manner in which the results of each study, inform future experimental manipulations across a three year testing programme.

1.3. Phase 1: Development and build of a submarine command team simulator

The first phase of ComTET was to design and build a submarine command room environment that represented a current operationally active submarine, allowing the collection of relevant data to assess submarine command team functionality. A simulator is defined as an apparatus or system that generates/reproduces an imitation of an operational environment or real world process across a period of time [4,5]. Simulations are commonly used when accessing the real system or operational environment is problematic or dangerous [6,7]. Simulators can incorporate additional controls, allowing the collection of operator performance data (e.g. software logging, videotaping and strategy analysis), for research or training purposes [8, 9]. Simulators also allow temporal manipulation [5, 9], providing operatives with exposure to more simulated experiences in each session and allowing time to think though decision alternatives and action plans.

The ComTET submarine simulator was required to be representative of military training simulators. Non-military participants were required to be trained in a day, so that recruitment did not rely on a military trained cohort for studies with high statistical power. However, the submarine simulator was required to have adequate fidelity for currently operational submarine command teams to attend the facility and participate in studies as the 'gold standard' comparator. The simulator was also required to be reconfigurable allowing the flexibility to experimentally test radical future ways of working, whilst overcoming previous issues concerning research in this domain (e.g. security, recruitment, financial and time costs). The primary objectives when building the submarine command team simulator was to create an immersive environment with adequate fidelity to generate levels of WL and SA comparable to submarine command team simulators used to train Navy personnel to a sea worthy level.

2. Theoretical considerations

2.1. Workload and situation awareness

At the level of the individual, working memory is the cognitive system used for the temporary storage and manipulation of task relevant information [10, 11]. WM is a limited capacity system and is therefore susceptible to overload. SA is defined as perception of elements in the environment at a specific time and space, comprehension of their meaning and projection concerning future status [12]. If an individual's capacity to process information is being exceeded by the volume of information being presented or the cognitive processing required for manipulation/interpretation, then it is difficult for SA to be maintained at an adequate level. An over-reliance on cognitive shortcuts such as pattern matching and previous experience can lead to rapid decisions but is prone to errors bias and reduced overall productivity [13, 14].

The making of successful decisions and completion of objective achieving operations relies upon good teamwork [15]. Teamwork relies heavily upon communications between team members, such processes can become the limiting factor in determining workload of the team, rather than the work itself [16]. Examining cognition, WL and SA solely at the level of the individual, does not examine the complexities and emergent properties of the team structure. Distributed cognition is characterised by multiple individuals and teams working together in pursuit of a common goal for which high levels of communication and coordination are required and a reliance on technologies to facilitate this [17].

Deciding levels of fidelity is a delicate balance, overloading the cognitive capacity of operatives does not facilitate learning; however, impoverished simulated environments may lead to boredom and/or negatively impact on implicit procedural knowledge [18]. It is important that levels of fidelity are dictated by the objectives of the simulator use (e.g. cognitive and behavioural requirements) and not just operator/SME opinion [19]. It was important that the ComTET submarine command team simulator had a level of fidelity required to achieve the aims and ambitions of the ComTET project.

2.2. Simulator fidelity

Simulator fidelity is defined as the degree to which a simulated environment corresponds to or emulates a domain specific operational/real world environment [4]. Fidelity is not a simple high-low dichotomy; it is the interaction of many different dimensions [4, 19]. A comprehensive review of the literature indicates that simulator fidelity is commonly split into 3 broad dimensions; physical, functional and psychological.

Physical fidelity refers to whether a simulator looks, sounds and feels realistic [4, 20]. This can range from the look and feel of levers, buttons and stimulus displays, to visual scene presentation and the creation of whole body motion [21, 5]. Traditionally, high physical fidelity has been favoured above high functional fidelity by operatives [4]. A simulator with extremely high physical fidelity can be detrimental to learning by overloading WM with material irrelevant to the specific training task [22, 23].Functional fidelity concerns whether a simulator models critical aspects of an operational environment [4, 5, 24]. It is whether internal mental models generated by the simulation correspond to the cognitive essence of the operational environment [25]. Behavioural fidelity refers to whether a simulation elicits 'real world' responses from operatives [26, 27]. Operational and task fidelity refer to the degree to which a simulation can match the processes and domain specific task requirements of an operational environment across a period of time [5]. Psychological fidelity is defined as the degree to which the simulation elicits the same psychological (e.g. stress and fear), cognitive (e.g. processing demand) and sensory (i.e. perception) experiences as the real world operational environment [4, 5, 25]. It relates to how information is presented, perceived and processed; whether cortical activation is similar in a simulated environment to when the sensory array is stimulated in an operational environment, particularly with regard to long term memory representations [28, 29,

3. Design and build of the ComTET Submarine Simulator

Subject Matter Experts (SME's) from the Royal Navy and the Defence Science and Technology Centre (DSTL) were involved in the design and building of the submarine command room simulator at various stages. A large empty room (10ft x25ft) was cleared and stripped down, designs of the command room layout (based on operational ASTUTE class submarine) were developed and bespoke cabinets to house the operator station computers were designed. A 3ft high false floor was built, which made the height of the ceiling comparable to submarine command rooms. Dexion was used for the construction of false walls that could be easily moved for future physical experimental manipulations (see picture 1 for a pictorial progression of the laboratory development). Additional physical equipment included scribble pads, seat storage, a mock-up air-conditioning system, lockers and a functioning Tannoy system. During this part of the build the primary consideration was physical fidelity [4, 20, 5, 21] and the creation of an environment in which currently serving submariners would feel represented an operational environment. The layout involved the creation of separate sound and picture compilation rooms, configured in a fashion that was representative of the currently operational Astute class submarine.

Task fidelity [5] was the primary consideration when selecting a simulation engine and selecting operator roles for inclusion in the simulator. It was imperative that an environment was created in which the primary tasks completed by a submarine command team could be simulated. It was also important that the selection of roles was representative of an operational command team and that the operator stations could offer behavioural fidelity at the level of the individual (e.g. completion of sonar detection, designation, classification and speed estimation) alongside operational fidelity at the level of the team (e.g. communication between sonar operatives and target motion analysis operatives for completion of global mission objectives). The simulation engine and software stack chosen enabled the creation of an environment with high task fidelity. Alterations were made to the simulation engine to remove game play elements (e.g. removing auto crew voice commands), increase fidelity of team operability (e.g. the ability to assign multiple sonar stations) and increase the fidelity of the software functionality based upon feedback from SMEs (e.g. the ability to generate multiple target motion analysis solutions simultaneously).

Technical capability for the CoMTET submarine simulator comprised of ten participant systems with a combined Consumer-off-The-Shelf product (Dangerous Waters) and bespoke software stack (multi-channel communications system and automated data collection processes), in addition to a variable amount of experimenter systems. All operator and experimenter systems were connected via Ethernet cables to a twenty-four port switch, enabling fast data communication and interconnectivity. Each operator system comprised of a Viglen E8500 model, with adequate specifications to run the required software stack and power all additional hardware; a Dell S2240T ten point 22.5" touch screen monitor, acting as a primary Visual Display Unit (VDU); a 15.6" secondary VDU; One mouse, full keyboard and number pad keyboard; A Genius 720P webcam, with integrated microphone; A set of Plantronics 86050-05 of headphones, with integrated microphone and noise cancelling capabilities; A LED Display powered by an Arduino board. Windows 7 Enterprise (x64) OS is installed on all systems, with licensing provided by UoS site license. The simulation engine used is Dangerous Waters, an off the shelf naval warfare simulation game featuring many player-controllable units deployed in the armed forces of numerous countries.

The simulation engine has a multi-player function where teams of individuals can play the role of different operatives aboard the same submarine vessel (see picture 2 for example pictures of the Dangerous Waters screens). The game has been developed by the US military and its task fidelity is considered to be high. The version currentlyinstalled is a non-Commercial bespoke build of Dangerous Waters for the MoD; this includes a Dangerous WatersApplication Programming Interface, providing an interface to monitor platform information whilst running scenarios. A number of scenarios have been created in Dangerous Waters based upon feedback received from SMEs. The scenarios include: return to periscope depth, dived tracking (after initial short range detection) and inshore operations (e.g. periscope reconnaissance). Each scenario has a high and low workload condition. The ComTET team developed a bespoke communications and data collection software suite. This software enabled verbal communication between all systems, with access to over 5 communication channels, in addition to the ability to monitor and record communications (using Audacity software). The selection of operator stations to include in the simulator was guided by SMEs, the selected roles are detailed below along with a brief description of their primary responsibilities within the command team.

a. b. c.

Fig. 1. (a) The false floor and prototype cabinet; (b) Construction of reconfigurable false walls; (c) Operator workstations being positioned.

3.1. The sound room

The Sonar Operatives (SO) are responsible for monitoring hull and flank passive sonar arrays. Their primary tasks include completing regular sweeps of the visual sonar waterfall to detect new potential vessels surrounding the submarine (either visual or auditory detection). The (SO's) are also responsible for completing further analysis of the sound profile detected to identify the vessel type and generate a speed estimate.The Sonar Controller (SoC) is more senior than the SO. The SoC is responsible for delegating and quality checking the work of the SO's and has a more complete awareness of what sounds are being detected around the submarine (e.g. the combination of multiple sonar arrays). The SC provides authority for potential contacts to be designated as contacts which should be monitored (rather than organic noise) and instructing bearing cut information to be sent to the picture compilation room.

3.2. The picture compilation room

The two Target Motion Analysis (TMA) operatives are responsible for processing information received from the sound room concerning the location from which an acoustic signal is being received across time. The TMA operatives receive bearing cuts periodically and must intelligently integrate this information with estimates of vessel course and speed to provide estimates of the positioning and future behaviour of a vessel. The Operations Officer (OpsO) is second in command in the simulator. The OpsO is responsible for delegating and quality checking the work of the TMA operatives. The OpsO is required to have a more complete tactical picture, verifying information with SoC and passing critical information (e.g. priority contacts) to senior members of the command team.

3.3. Additional operator stations

The Periscope (PERI) Operator is responsible for visually searching for potential contacts when the periscope is raised. The periscope can only be raised at certain depths and speeds, therefore PERI is required to be aware of these parameters to operate the surface instruments safely. PERI relies on information from sonar and TMA to guide visual sweeps, focusing on bearings where vessels are believed to be positioned.The Ship Control Operator (SC) is responsible for steering the submarine in terms of course, depth and speed. The SC needs to constantly monitor the parameters of the submarine to check that everything is healthy and particular parameters are maintained (e.g. a particular depth when surface instruments have been raised). The SC has a number of other duties including maintaining adequate ventilation of pressurized air storage and streaming of certain instruments used by the sonar team. The Officer of the Watch (OOW) is second in command on the submarine (behind the Captain) and is responsible for the safety of the submarine, making sure any manoeuvres or operations (e.g. returning to periscope depth from deep) are completed safely and efficiently. The OOW is required to hold a complete tactical picture of what surrounds the submarine, to guide decision making. The OOW may quality check the work of any of the

operatives and will guide the tactical process (e.g. issue commands for steering the submarine and mission objectives).

3.4. Data collection

The flow of information throughout the command team shall be examined using the Event Analysis for Systemic Teamwork (EAST) [30]. EAST is a method developed specifically for modelling command team. Since its conception, the framework has been applied in many domains, including naval warfare [31]. EAST models complex collaborative systems through a network approach. Specifically, three networks are considered: task, social, and information. This triangulated approach enables the study of distributed cognition and awareness with correlations to task performance. In situations that require team work and information held and distributed by technology, this provides a systems oriented approach to the study of SA. The submarine simulator has the capacity to record all communications (verbal and non-verbal) that occur between operative with a collection of 10 web-cameras, 2 high resolution video-cameras, 2 ambient micro-phones and the installation of recording software to capture any transmissions over the five channel communication network. All actions completed by participants shall be logged (e.g. TMA solution entered vs. truth) allowing for performance to be assessed and an indication of data management/passing to be provided. This will allow the examination of command team function using the EAST method.

A variety of other measures shall be collected; a battery of cognitive tests has been selected to assess the cognitive function of operatives before and after completion of scenarios. The cognitive tests selected are digital and so can be run on the command stations of all operatives immediately before and after scenario completion. Participants will also be asked to complete cognitive tasks during the experiment to assess spare cognitive capacity using the dual task paradigm. A battery of system usability, subjective workload and situation awareness measures have also been selected for administration after scenarios, these measures are also digital, allowing efficiency ofcollection from an entire command team with minimal experimenter input. The simulator also has the capacity to objectively assess workload during scenario completion, as a psycho-physiological recording suite was included in the build allowing the collection of various measures (e.g. electrocardiogram) that are commonly used for the assessment of operator workload.

Fig. 2. (a) Example of Target Motion Analysis workstation; (b) Example of broad band sonar workstation.

Fig. 3. (a) SMEs completing simulations in picture room; (b) SMEs completing simulations in sound room.

3.5. Simulator usability

A primary consideration when designing the simulator was that it should be adequate to complete experiments using expert participants (e.g. a command team of submariners) but also by individuals with no military or maritime experience (e.g. a student cohort). This would allow the completion of a body of work with authenticity and high statistical power. A one day training package was compiled; this consisted of 6 hour long tutorials including detailed tuition on how an individual operator can use the tools available at their workstation to complete primary tasks (e.g. sonar profile monitoring and analysis), how the information provided should be communication between the entire command team and how individual operators contribute to the global objectives of the submarine command team. To examine the usability of the simulator a currently operational submarine command team (who had been SMEs advising development) and a selection of students from Southampton University were invited to participate in pilot studies (see picture 3 for example of currently operational command team completing scenarios in simulator). The student cohort received the full training day prior to completing a day of scenarios. The submarine command team received no additional training. Both teams were able to complete all scenarios and levels of performance and immersion observed by experimenters appeared to be comparable. The data collected from the pilot studies along with feedback questionnaires shall be used to make the final adjustments to the submarine simulator prior to phase 2 of ComTET, the collection of baseline data.

4. Conclusion

The first phase of ComTET was to design and build a submarine command room environment that represented a current operationally active submarine, allowing the collection of relevant data to assess submarine command team functionality. The facility was required to be cost effective, re-configurable (for testing future was of working) and usable by domain experts and novices alike. The ComTET team, with the guidance of a number of currently operational British Navy submariners have constructed a simulator with levels of task, physical and psychological fidelity [4,5] that balance usability by novice operatives with immersion of domain experts. The wealth of data that the simulator has the capacity to record will provide the foundation of a research programme examining future ways of working within submarine command teams.

References

[1] Stanton, N. A. (2014). Representing distributed cognition in complex systems: how a submarine returns to periscope depth. Ergonomics,

57(3), 403-418.

[2] Stanton, N. A., Ashleigh, M. J., Roberts, A. D., & Xu, F. (2003). Virtuality in human supervisory control: assessing the effects of

psychological and social remoteness. Ergonomics, 46(12), 1215-1232.

[3] Klein, G., & Miller, T. E. (1999). Distributed planning teams. International Journal of Cognitive Ergonomics, 3, 203-222.

[4] Alexander, A. L., Brunye, T., Sidman, J., & Weil, S. A. (2005, November). From gaming to training: A review of studies on fidelity,

immersion, presence, and buy-in and their effects on transfer in pc-based simulations and games. In The interservice/industry training, simulation, and education conference (I/ITSEC), NTSA, Orlando, Florida.

[5] Stanton, N. (1996). CHAPTER SEVEN Simulators: a review of research and practice. Human factors in nuclear safety, 7, 114.

[6] Sokolowski, J. A., & Banks, C. M. (2009). Modeling and simulation for analyzing global events. Wiley. com.

[7] Stanton, N. A., Young, M. S., Walker, G. H., Turner, H., & Randle, S. (2001). Automating the driver's control tasks. International Journal of

Cognitive Ergonomics, 5(3), 221-236.

[8] Lee, J. D., McGehee, D. V., Brown, T. L., & Reyes, M. L. (2002). Collision warning timing, driver distraction, and driver response to

imminent rear-end collisions in a high-fidelity driving simulator. Human Factors: The Journal of the Human Factors and Ergonomics Society, 44(2), 314-334.

[9] Young, M. S., & Stanton, N. A. (2007). Back to the future: Brake reaction times for manual and automated vehicles. Ergonomics, 50(1), 46-

[10] Andrade, J. (2001) Working Memory in Context Psychology. Hove, Sussex.

[11] Baddeley, A. D. (2000). The episodic buffer: a new component of working memory? Trends in Cognitive Science 4, 417-423.

[12] Endsley, M. R. (2012). Designing for situation awareness: An approach to user-centered design. CRC Press.

[13] Harris, D. (2011) Human Performance on the Flight Deck. Aldershot: Ashgate

[14] Klein, G., & Miller, T. E. (1999). Distributed planning teams. International Journal of Cognitive Ergonomics, 3, 203-222.

[15] Stanton, N. A. (2011) Trust in Military Teams. Ashgate: Aldershot, UK.

[16] Salas, E., Burke, C. S., & Samman, S. N. (2001). Understanding command and control teams operating in complex environments. Information Knowledge Systems Management, 2, 311-323.

[17] Stanton, N. A., Walker, G. H., Jenkins, D. P., Salmon, P. M., Revell, K and Rafferty, L. (2009) Digitising Command and Control: Human Factors and Ergonomics Analysis of Mission Planning and Battlespace Management. Ashgate: Aldershot.

[18] Dahlstrom, N., & Nahlinder, S. (2009). Mental workload in aircraft and simulator during basic civil aviation training. The International journal of aviation psychology, 19(4), 309-325.

[19] Tichon, J. G., & Wallis, G. M. (2010). Stress training and simulator complexity: why sometimes more is less. Behaviour & Information Technology, 29(5), 459-466.

[20] Baum, D. R., Riedel, S. L., Hayes, R. T., & Mirabella, A. (1982). Training effectiveness as a function of training device fidelity (No. 82SRC37). HONEYWELL SYSTEMS AND RESEARCH CENTER MINNEAPOLIS MN.

[21] Bürki-Cohen, J., Boothe, E. M., Soja, N. N., DiSario, R. D., Go, T., & Longridge, T. (2000, May). Simulator fidelity: The effect of platform motion. In proceedings of the Royal Aeronautical Society conference on Flight Simulation—The Next Decade, London

[22] Dahlstrom, N., Dekker, S., Van Winsen, R., & Nyce, J. (2009). Fidelity and validity of simulator training. Theoretical Issues in Ergonomics Science, 10(4), 305-314

[23] Kluge, A., Sauer, J., Schüler, K., & Burkolter, D. (2009). Designing training for process control simulators: a review of empirical findings and current practices. Theoretical Issues in Ergonomics Science, 10(6), 489-509.

[24] Steurs, M., Mulder, M., & Van Paassen, M. M. (2004, August). A cybernetic approach to assess flight simulator fidelity. In AIAA Modeling and Simulation Technologies Conference.

[25] Estock, M. J. L., Alexander, D. A. L., Gildea, D. K. M., Nash, C. M., & Blueggel, 2. L. B. (2006, January). A model-based approach to simulator fidelity and training effectiveness. In The Interservice/Industry Training, Simulation & Education Conference (I/ITSEC) (Vol. 2006, No. 1). National Training Systems Association.

[26] Bürki-Cohen, J., Boothe, E. M., Soja, N. N., DiSario, R. D., Go, T., & Longridge, T. (2000, May). Simulator fidelity: The effect of platform motion. In proceedings of the Royal Aeronautical Society conference on Flight Simulation—The Next Decade, London

[27] Wang, Y., Mehler, B., Reimer, B., Lammers, V., D'Ambrosio, L. A., & Coughlin, J. F. (2010). The validity of driving simulation for assessing differences between in-vehicle informational interfaces: a comparison with field testing. Ergonomics, 53(3), 404-420.

[28] Berka, C., Levendowski, D. J., Lumicao, M. N., Yau, A., Davis, G., Zivkovic, V. T., ... & Craven, P. L. (2007). EEG correlates of task engagement and mental workload in vigilance, learning, and memory tasks. Aviation, space, and environmental medicine, 78(Supplement 1), B231-B244.

[29] Kiili, K. (2005). Digital game-based learning: Towards an experiential gaming model. The Internet and higher education, 8(1), 13-24.

[30] Stanton, N. A., Baber, C. and Harris, D. (2008) Modelling Command and Control: Event Analysis of Systemic Teamwork. Ashgate: Aldershot.

[31] Stanton, N. A. and Baber, C. (2006) The Ergonomics of command and control. Ergonomics, 49 (12-13), 1131-1138.