Scholarly article on topic 'Measuring Situation Awareness with Probe Questions: Reasons for not Answering the Probes'

Measuring Situation Awareness with Probe Questions: Reasons for not Answering the Probes Academic research paper on "Psychology"

CC BY-NC-ND
0
0
Share paper
Academic journal
Procedia Manufacturing
OECD Field of science
Keywords
{"Situation Awareness" / "Situation Present Assessment Method" / SPAM / "Single pilot operations" / SPO / "Reduced crew operations" / RCO / "Response time-out" / "Question time-out"}

Abstract of research paper on Psychology, author of scientific article — James C. Cunningham, Henri Battiste, Sam Curtis, Elyse C. Hallett, Martin Koltz, et al.

Abstract Situation Awareness (SA) refers to an individual's understanding of what is happening in their environment, and what is likely to happen in the near future [1]. SA is an important construct to study in aviation because lower levels of SA are associated with increased aviation accidents and mishaps [2]. One technique to measure operator SA is through the use of online probe questions, where operators are queried about their task environment at various intervals while performing the task. The current study used the Situation Present Assessment Method (SPAM) [3] for administering online probes. With SPAM, a ready prompt appears on a panel and participants are instructed to accept the prompt when workload permits to be presented with a SA probe question. Latencies in accepting the ready prompt are indicative of the operator's workload level at that time. When ready prompts go unanswered, (ready time-outs; RTO), it is assumed workload is too high. When a prompt is accepted, probe questions are presented for a limited time (60seconds in this study). Thus it is possible they too will go unanswered by timing out (question time-out; QTO). QTOs can occur for the following reasons: 1) operator did not know the answer, 2) due to the dynamic changes in the task, the operator workload became too high to devote resources to answering the question, 3) question was unanswerable due to the current context, or 4) question required more time to answer than allowed. In the present study, we analyzed a subset of the data from a larger study (see Brandt et al. [4]) to examine reasons for why probe questions go unanswered. For QTOs, the majority of questions went unanswered because the operator did not know how to answer the question, or because the operator was dealing with high workload. RTOs, which occurred more frequently, were normally due to task/workload.

Academic research paper on topic "Measuring Situation Awareness with Probe Questions: Reasons for not Answering the Probes"

CrossMark

Available online at www.sciencedirect.com

ScienceDirect

Procedia Manufacturing 3 (2015) 2982 - 2989

6th International Conference on Applied Human Factors and Ergonomics (AHFE 2015) and the

Affiliated Conferences, AHFE 2015

Measuring Situation Awareness with probe questions: Reasons for not answering the probes

James C. Cunninghama, Henri Battistea, Sam Curtisa, Elyse C. Halletta, Martin Koltza, Summer L. Brandtb,c, Joel Lachterb,c, Vernol Battisteb,c, Walter W. Johnsonc

aCalifornia State University Long Beach, 1250 Bellflcwer Blvd, Long Beach, CA 90840, USA b San José State University, NASA Ames Research Center, Mcffett Field,CA, 94035, USA cNASA, NASA Ames Research Center, Moffett Field,CA, 94035, USA

Abstract

Situation Awareness (SA) refers to an individual's understanding of what is happening in their environment, and what is likely to happen in the near future [1]. SA is an important construct to study in aviation because lower levels of SA are associated with increased aviation accidents and mishaps [2]. One technique to measure operator SA is through the use of online probe questions, where operators are queried about their task environment at various intervals while performing the task. The current study used the Situation Present Assessment Method (SPAM) [3] for administering online probes. With SPAM, a ready prompt appears on a panel and participants are instructed to accept the prompt when workload permits to be presented with a SA probe question. Latencies in accepting the ready prompt are indicative of the operator's workload level at that time. When ready prompts go unanswered, (ready time-outs; RTO), it is assumed workload is too high. When a prompt is accepted, probe questions are presented for a limited time (60 seconds in this study). Thus it is possible they too will go unanswered by timing out (question time-out; QTO). QTOs can occur for the following reasons: 1) operator did not know the answer, 2) due to the dynamic changes in the task, the operator workload became too high to devote resources to answering the question, 3) question was unanswerable due to the current context, or 4) question required more time to answer than allowed. In the present study, we analyzed a subset of the data from a larger study (see Brandt et al. [4]) to examine reasons for why probe questions go unanswered. For QTOs, the majority of questions went unanswered because the operator did not know how to answer the question, or because the operator was dealing with high workload. RTOs, which occurred more frequently, were normally due to task/workload.

© 2015TheAuthors.PublishedbyElsevierB.V. This is an open access article under the CC BY-NC-ND license

(http://creativecommons.Org/licenses/by-nc-nd/4.0/).

Peer-review under responsibility of AHFE Conference

Keywords: Situation Awareness; Situation Present Assessment Method; SPAM; Single pilot operations; SPO; Reduced crew operations; RCO; Response time-out; Question time-out

2351-9789 © 2015 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license

(http://creativecommons.Org/licenses/by-nc-nd/4.0/).

Peer-review under responsibility of AHFE Conference

doi: 10.1016/j.promfg.2015.07.840

1. Introduction

1.1. Examining reasons for ready time-outs and question time-outs of SPAM probes

Situation Awareness (SA) refers to an operator's understanding of his/her task environment [1]. In aviation, it is important to measure operator SA because low levels of SA can lead to human errors that are associated with increased aviation accidents and mishaps [2]. The Situation Present Assessment Method (SPAM) of measuring SA has gained traction in studying complex and dynamic domains [3]. SPAM is an online probe method that queries an operator about elements in his or her environment while s/he is performing the task. The SPAM technique deconfounds operator SA from workload by presenting operators with a ready prompt before presenting the actual SA query. The operator is instructed to accept the ready prompt only when their workload is manageable. That is, the operator should only accept the ready prompt when s/he has enough capacity to read and answer the probe question. Ready response latencies are indicative of workload: low latencies equal low workload; higher latencies equal moderate workload; no response equals workload too high, no time for probes. When an operator fails to respond to a probe question's ready prompt within in a given period of time, data pertaining to the question is lost and the response is marked as a response time-out (RTO). RTOs can be indicative of high levels of operator workload at the time the prompt was administered [5, 6].

If the operator accepts the ready prompt, a probe question is then presented to the operator to measure the operator's SA. These probe queries are generated prior to data collection, with each question tailored to the scenario and verified for relevance by a subject matter expert. Using SPAM, both accuracy and response times to the probe queries are recorded. Correct answers to the probes are associated with having good SA and incorrect answers with poor SA. Response times to the queries provide further insight into the operator's cognitive processes. Short response times to correct answers suggest that the information necessary to answer the question was stored in the operator's memory and could be easily recalled by the operator. Longer response times to correct answers suggest the operator had to derive the answer or look it up from some external source [7]. The validity of SPAM as a measure of SA requires that most questions be answered. For accuracy, percentage correct responses should exceed levels that would be achieved just by guessing and response latency is based only on correct answers.

Typically, a probe question is only presented for a limited time (e.g., 60 seconds). Unfortunately, a limited presentation time makes it possible for the question itself to go unanswered by timing out. We refer to these instances as a question time-out (QTO). QTOs can occur for the following reasons: 1) the operator did not know how to figure out the answer to the question (indicative of no SA for the information that is being queried), 2) the operator's workload changed and s/he was no longer able to devote resources to answering the question, 3) the content of the question was unanswerable in the context of the task (queries are typically developed from a base scenario; with dynamic changes occurring, the current context may not match the original context on which the query was based), or 4) the question required more time to answer than the probe allowed (e.g., question timed out while the operator looking up the information or performing calculations). Thus, there are SA-related and SA-unrelated reasons behind QTOs.

To examine the cause of QTOs we examined data from a larger study using the SPAM technique that looked at the effects of SA on the feasibility of single pilot operations/reduced crew operations (SPO/RCO; see Brandt et al. [4] for further details about the study). In the next section, we provide a brief description of the SPO/RCO concept and review of prior work in this area to allow readers to understand the context in which the probe questions were presented.

1.2. Background on single pilot operations/reduced crew operations

Current flight operations require two pilots (Captain, First Officer) for commercial and cargo aircraft. While the Captain retains the primary leadership role, crews typically trade-off flying and monitoring roles. These roles are specified in each airline's standard operating procedures. Typically pilot flying is concerned with controlling the aircraft while the pilot monitoring handles radio communications, operating the aircraft system, completes required checklists throughout various flight stages, and assists with other cockpit responsibilities. Pilots also crosscheck each other to reduce and identify errors. This current setup requires two experienced pilots for each flight. Some estimates

put crew costs as high as 35% of total operating costs for flights less than 200 nm, carrying less than 50 passengers [8]. At the same time, recent improvements in cockpit automation and remote piloting technologies developed for unmanned aircraft systems (UAS) have raised the possibility of reducing the number of pilots to one. The idea would be to considerably reduce crew costs by having some of the responsibilities of the First Officer distributed to automation and ground support making a second cockpit pilot unnecessary.

Since air travel is expected to double in the next 20 years and there is a concurrent predicted decline of commercial and cargo pilots [9], reducing pilots needed in a cockpit would ease a predicted pilot shortage. This decline in the number of available pilots has been attributed to factors including retirement, strict hiring requirements, low paying entry-level positions, and high cost of flight schools [10, 11]. Without enough pilots, the airline industry may face the challenge of ineffectively using the pilots that are available. One potential solution is moving towards SPO/RCO. However, in order for SPO/RCO to be successful, the work currently done by the First Officer must be performed by other people or automation. The current study is the third in a series of studies examining the ability of ground-based personnel to perform some of this work.

In 2014, Lachter et al. [12] reported on a study examining the effects of non-collocation on a crew's ability to work together. In one condition, a two-pilot crew flew off-nominal scenarios while seated next to each other to simulate current day operations. In another, the same scenarios were flown with the First Officer seated in a separate room where the right side of the flight deck was recreated. The results of this study found little difference in objective performance between the two conditions. However, most pilots preferred flying together and rated the separate condition more poorly for safety of flight, communication and coordination. One area that seemed particularly challenging within the separate condition was simply trying to understand what the other crew member was doing. An analysis of the pilots' interactions found many more incidents of confusion with the other pilot in the separate condition than in the together condition. With these findings and further feedback from the participants, collaboration tools were developed to help pilots enhance collaboration and become more aware of actions taken by the other pilot.

In a follow-on study, Lachter and colleagues [13] examined a suite of tools developed to address the Crew Resource Management (CRM) challenges stemming from non-collocated crews found in the initial study. In this second study, ground operators were asked to uplink suggested reroutes to multiple aircraft that were headed into convective weather cells or turbulence. Operators were told to reroute as many aircraft as possible, even while assisting other aircraft. However, almost no aircraft were rerouted after an off-nominal aircraft called in at the beginning of the scenario. In debriefings, pilots commented that the rerouting task slipped out of their awareness once they became involved in working on the off-nominal flight. An argument was made that a ground operator working on off-nominal aircraft should not be asked to simultaneously service other aircraft. From this, two distinct concepts of operation emerged:

• Specialist - a ground dispatcher performs normal dispatch functions and hands the aircraft to a separate person (Specialist) who provides assistance to the aircraft when one-on-one piloting assistance is needed (we refer to such situations as "dedicated assistance").

• Hybrid - a ground operator, who can perform both dispatcher and pilot functions, keeps the aircraft, provides DA as needed, and hands off all other aircraft to another operator.

As noted earlier, the current study is based on analysis of part of the data from the third study in this series that examined these two different ground operator concepts [4]. This study looked at a concept of SPO/RCO wherein a flight crew is reduced to one (the airborne Captain) and the duties of the First Officer are performed, in part or in full, by someone on the ground (a "ground operator"). In particular, Brandt et al. [4] were interested in whether any increase in SA garnered from operators observation of the aircraft and airspace environment prior to a call for dedicated assistance. Brandt et al. [4] referred to this observation as "Situation Preview." Using the Specialist and Hybrid ground operator concepts, Brandt et al. [4] examined the ability of ground operators with limited SA to come to the aid of a single airborne pilot in an off-nominal or high workload situation. In their study, Specialists were called upon to provide dedicated assistance (DA) to an aircraft without having previously seen the aircraft or the airspace in which it was flying (No Situation Preview), whereas Hybrid operators were assumed to have previously

observed (Low Situation Preview) and possibly interacted with aircraft needing DA (High Situation Preview) as part of their dispatch role.

During the simulations, SA measures were collected using the SPAM technique, but some probes went unanswered. Either the operators did not respond to the ready prompt (RTO) or they were presented the probe question and did not answer (QTO). In either case, some SA probe data was lost due to these time-outs. In an effort to more fully understand why these time-outs occurred, this paper examined the conditions related to RTOs and QTOs under the two ground operator concepts.

2. Methods

2.1. Participants

Thirty-five pilots (3 retired, 32 current) took part in the present study. At the time of the study, twenty-one participants had logged over 10,000 hours of flight time, 11 had logged between 5,001 and 10,000 hours of flight time, two had logged between 3,001 and 5,000 hours of flight time, and one had logged between 1,001 and 3,000 hours of flight time. Seven participants had prior military flying experience.

2.2. Design

The present study utilized a within subjects design with a single fixed factor, Situation Preview with three levels: No (Specialist concept), Low (Hybrid concept with no prior contact from the aircraft needing dedicated assistance), and High (Hybrid concept with prior contact from the aircraft needing dedicated assistance). Two scenarios were run for each condition for a total of six scenarios per participant.

Participants acted as ground operators responsible for assisting single piloted aircraft through a sector of airspace centered near Denver. In two of the conditions, the participant acted as a Hybrid operator (High Situation Preview and Low Situation Preview) and in the third condition the participant acted as a Specialist operator. In the Hybrid conditions, the participant first provided support to ten aircraft in the sector. Around halfway into the scenario (~15 min), an aircraft with a high workload or off-nominal event would request dedicated assistance. The participant would then transition into dedicated assistance mode and provide First Officer support. The remaining aircraft were automatically redistributed to other ground operators (virtual operators in this simulation). In the Specialist concept, the ground operator began the scenario by entering dedicated assistance for the high workload or off-nominal aircraft, without ever providing support to any other aircraft.

2.3. Materials

The ground station consisted of four primary displays that provided global information about the airspace, a shared set of cockpit instruments, weather information, and crew resource management (CRM) tools. Two of the screens were touchscreens. The right touchscreen was used for manipulating shared, cockpit-related materials such as charts. The left touch screen had control display units (CDUs, used for entering data into the flight management system) and a tool used to support CRM awareness. Probe questions and probe responses were also presented and collected on this screen. There were two additional monitors which provided an out-the-window view of weather and a video feed of the flight deck. These were only active during dedicated assistance. The airspace and aircraft were simulated using the Multi Aircraft Control Systems (MACS) [4]. Participants managed the ground station while all other aspects of the simulated environment (piloting, air traffic management, and company services such as maintenance and weather forecasting) were handled by confederates. Participant ground operators communicated with confederates through a headset and a push to talk microphone. A more detailed summary of the ground operator workstation is included in Brandt et al. [4].

2.4. Procedure

At the start of the day, participants completed a demographics questionnaire and were briefed on their roles and responsibilities as ground operator under the two concepts. Approximately 3.5 hours were spent on training displays and controls, procedures and practice trials. Participants then completed six experimental scenarios, each lasting approximately 15 - 30 minutes, depending on the concept of operations being run. Following every scenario, participants completed a post-trial questionnaire.

Starting five minutes into each scenario and continuing every 2-4 minutes, participants were presented with a probe question regarding the state of the system and airspace using SPAM. Each probe began with a ready prompt during which participants indicated when they were ready to see the probe question. Participants were instructed to accept the ready prompt only when their workload was manageable. That is, the operator should only accept the ready prompt when s/he has enough capacity to read and answer the probe question. If the participant did not respond to the ready prompt within 60 seconds, this was considered a ready time-out (RTO) and the prompt disappeared. If the participant indicated being ready, a multiple choice probe question was displayed. If the participant failed to answer the probe question within 60 seconds, it was called a question time-out (QTO). To help understand the cause of a QTO, a second multiple choice question was displayed asking the participant for an explanation of why he or she did not answer the probe question. The participant was told to choose one of the following choices as a QTO reason:

• "I did not know how to determine the answer."

• "Workload was too high."

• "The question required too much time to answer."

If no response was given and the second set of questions timed out, "No response given" was recorded for the probe. Figure 1 displays examples of the three SPAM probe screens presented to ground operators during trials.

Following the final trial, participants were debriefed and other subjective measures about the experiment were collected. For the purpose of the present study, a subset of this data, relating to the RTOs and QTOs are reported. For more information on probe response and subjective measures, see Brandt et al. [4].

Ready?

How far are you from your destination airport?

<10 nm 11-20nm 21-30tim 31-40nm >40 nm

Why couldn't you answer that question? The question required too much time to answer

1 did not know how to determine the answer

Workload was too high

Fig. 1. Examples of probe screens presented during trials. From top to bottom: ready prompt, probe question, QTO reason question.

3. Results

3.1. Overall RTOs and QTOs: Percentages and reasons

RTO and QTO percentages for occurrences were calculated across ground operator conditions (see Table 1). As described in Table 1, RTOs and QTOs occurred for about 20% of the probe questions. Most of the time-outs were RTOs (16.3% RTOs compared to 2.83% QTOs). Thus, the main reason for why the questions were unanswered is that the questions were presented during times when the operator was busy or experiencing high levels of workload.

Table 1. Percentage of RTO and QTO occurrences by condition.

Ground Operator Condition; ; # of probes presented RTO (%) QTO (%)

Specialist (No Preview); n = = 204 17.2 0.5

Low Situation Preview; n = 410 15.6 4.6

High Situation Preview; n = 411 16.1 3.4

Frequencies were calculated for the specific probe questions that resulted in QTOs (n = 34; Table 2) and overall QTO reasons given (Table 3). From these tables, it is clear that several questions were left unanswered more frequently than others. The reasons for not answering were "I did not know how to determine the answer" and "The question required too much time to answer" and were attributed to poor SA (the operator did not have SA of that information and/or the operator did not want to devote the effort to acquire SA for that information). Additional reasons of "No Response Given" and "Workload was too high" were attributed to non-SA reasons. From Table 3, 70% of QTOs were due to operators not having SA for the information being queried and 30% of the QTOs were due to other reasons, including high levels of workload.

Table 2. Frequencies of QTO-related probe questions; arranged by frequency in descending order.

QTO-Related Probe Question Frequency

How long would NASA## be able to fly the holding pattern before having to divert? 9

What was the last command issued to your AC by ATC? 6

How far are you from your destination airport? 4

Which of the following airports is experiencing delays due to weather? 3

How far is NASA57 from its destination airport? 2

Which AC will most likely need to deviate for weather? 2

Rate your workload

What is the closest airport to your current position?

What is the next LEG on your flight plan?

What will be the next command put into the MCP?

Which AC is most likely to be the next AC that will be diverted?

Which of the following AC is currently having a system failure?

Which of the following airports is currently being most impacted by weather?

Which of the following airports is not impacted by weather currently?

Table 3. Frequencies of QTO reasons given; arranged by frequency in descending order.

QTO Reason Given Frequency

I did not know how to determine the answer 20

Workload was too high 7

The question required too much time to answer 4

No Response Given 3

3.2. RTOs and QTOs during dedicated assistance: Percentages and reasons

To examine the effect of SA preview/operator role on RTOs and QTOs, multiple repeated measures ANOVAs were run on the mean RTOs and QTOs during the dedicated assistance portion of the trials (the period where all roles received the same number of probe queries). No significant main effect of Preview was found for RTOs during dedicated assistance, F(2, 52) = 1.533, p = .226; High Preview mean = 22.93%, Low Preview mean = 23.29%, No preview (Specialist) mean = 17.02%. These findings are consistent with that reported by Brandt et al. [4] showing that operator reported workload did not differ across conditions. Again, the mean number of QTOs were small, and no significant main effect of Preview was found for QTOs during dedicated assistance, F(1.616, 42.020) = 2.22, p = .130; High Preview mean = 3.45%, Low Preview mean = 4.10%, No preview (Specialist) mean = 0.46%.

Frequencies were calculated for QTOs that happened during dedicated assistance (n = 16) and their respective QTO reasons; see Table 4. Similar to the above analysis, Reasons of "I did not know how to determine the answer" and "The question required too much time to answer" were attributed to poor SA (the operator did not have SA of that information and/or the operator did not want to devote the effort to acquire SA for that information). Reasons of "No Response Given" and "Workload was too high" were attributed to non-SA reasons. From Table 4, 60% of QTOs were due to operators not having SA for the information being queried and 40% of the QTOs were due to other reasons, including high levels of workload.

Table 4. Frequencies of DA related QTOs and QTO reasons given; arranged by frequency in descending order.

QTO-Related Probe Question during DA QTOs QTO Reason Given Frequency

I did not know how to determine the answer 5

No Response Given 2

Workload was too high 1

The question required too much time to answer 1

What was the last command issued to your AC by ATC? 6 I did not know how to determine the answer 4 The question required too much time to answer 1 No Response Given 1

How far are you from your destination airport? 4 I did not know how to determine the answer 3 Workload was too high 1

How far is NASA57 from its destination airport? 2 The question required too much time to answer 1 Workload was too high 1

Rate your workload 1 Workload was too high 1

What is the closest airport to your current position? 1 Workload was too high 1

What is the next LEG on your flight plan? 1 I did not know how to determine the answer 1

What will be the next command put into the MCP? 1 I did not know how to determine the answer 1

How long would NASA## be able to fly the holding 9

pattern before having to divert?

4. Discussion

We examined some of the reasons why operators failed to answer probe queries. In ideal implementation of SPAM, all probes questions would be answered. Unfortunately, this outcome relies on an operator's ability to answer the probes in real-time throughout a scenario, under potentially high workload conditions. Indeed, the present analysis found that, regardless of ground operator condition, RTOs occurred around 15-17% of the time when operators were presented a SPAM ready prompt.

Across conditions, QTOs were rather rare, occurring on less than 5% of the questions. Of the QTOs that did occur, roughly 60-70% of them were because the operator did not know how to answer the question, while 30-40% were because the operator was dealing with high workload. Even in the higher workload situation of dedicated assistance [4], the most common explanation for a QTO was that the operator did not know how to answer the question. This finding suggests that it may be that a probe question's answerability may rely more on the context of the scenario and less on workload. For example, in this study, previous context-specific probes may have lost their meaning due to actions taken by the operator that made what the probes referenced incorrect. This would make certain probes confusing and unanswerable. It may also be that operator's may actually know the answer to the probe but the answer is not listed as an option on the screen, thereby making it confusing and unknown to the operators how to correctly answer. If either of these is the case, future work on probes should be focused on questioning task specific information that is less likely to be changed by operator action. This would also make probe answer screen options more relevant and transparent, making it more likely they will select one of the given choices.

Acknowledgements

We would like to acknowledge NASA's Airspace Systems Program - Concepts and Technology Development Project, which funded this research, and both NASA Ames Research Center's Flight Deck Display Research Laboratory and San Jose State University, who conducted the research. Special thanks to Kim Vu and Thomas Strybel for assistance editing this document.

References

[1] M.R. Endsley, Human Factors, 37:1 (1995), 32-64.

[2] M.R. Endsley, in: R. Fuller, N. Johnston, N. McDonald (Eds.), Human Factors in Aviation Operations, Ashgate Publishing Ltd., Aldershot, England, 1995, pp. 287-292.

[3] F.T. Durso, A.R. Dattel, S. Banbury, S. Tremblay, A cognitive approach to situation awareness: Theory and application, 1, 2004.

[4] S. Brandt, J. Lachter, V. Battiste, W.W. Johnson, in: the Proc. 6th Int. Conf. Appl. Hum. Factors Ergon., Las Vegas, 2015.

[5] S.V. Ligda, A.Q.V Dao, T.Z. Strybel, K.P.L Vu, V. Battiste, W.W. Johnson, in: the Proc. 54th Int. Conf. Hum. Factors and Ergon. Society, 2010, pp. 55-59

[6] K.P.L. Vu, T.Z. Strybel, V. Battiste, J. Lachter, A.Q.V. Dao, S. Brandt, S. Ligda, W.W. Johnson, International Journal of Human-Computer Interaction, 28:2 (2012), 107-118

[7] D. Chiappe, T.Z. Strybel, K.P.L. Vu, Journal of Cognitive Engineering and Decision Making, 9(1) (In press), 33-43.

[8] Harris, D., A Human-centered Design Agenda for the Development of Single Crew Operated Commercial Aircraft, Aircraft Engineering and Aerospace Technology, 2007, pp.518-526

[9] US Department of Transportation FAA, FAA Aerospace Forecast - Fiscal years 2012-2032, 2012.

[10] J. Blair, J. Freye, Natl. Assoc. Flight Instr. (2012). http://www.nafinet.org/whitepaper.aspx.

[11] S. Carey, J. Nicas, A. Pasztor, Wall Str. J. (2012). http://www.wsj.com/articles/SB10001424052970203937004578079391643223634.

[12] J. Lachter, V. Battiste, M. Matessa, Q.V. Dao, R. Koteskey, W.W. Johnson, in: Int. Conf. Human-Computer Interact. Aerosp., Santa Clara, 2014.

[13] J. Lachter, S. Brandt, V. Battiste, S. Ligda, M. Matessa, W.W. Johnson, in: Int. Conf. Human-Computer Interact. Aerosp., Santa Clara, 2014