Preliminary results: actual workload vs workload perception

Decision making during different phases of flight is a crucial task that consumes finite crew cognitive resources, which sometimes can be compromised due to co-existence of several duties. Pilots recurrently monitor aircraft systems and  environment while attend communication and aviation tasks. The crew is constantly involved in decision-making processes affecting the flight. Which factors affect the quality of the decision making process in the flight deck and which cognitive supporting tools could be designed to improve their performance are some of the main aspects which will be analysed by the e-Pilots project.

Setting the scene

In Multi-Crew Operations (MCO), the decision-making process is carried out by both pilots within the cockpit. However, in Single-Pilot Operations (SPO), this capability is reduced due to the lack of redundancy in the decisions input by a second pilot. In single-pilot operations, the situational awareness is reduced, affecting the perception of the different situations which might emerge from the aircraft operation, their consequences and interpretation.

Improving the situational awareness and safety of aircraft operations has been the mainstream of development for aircraft manufacturers. Nowadays, in order to rapidly detect failures and improve situational awareness, modern aircrafts are equipped with warning and alerting systems for the early detection of system malfunctions, abnormal operations, stalling, fire on board, or electrical failures. These events are often assessed by on-board assistance systems such as Airbus’ Electronic Centralized Aircraft Monitor (ECAM) or Boeing’s Engine-Indicating and Crew Alerting System (EICAS).

The e-Pilots project is devised with the clear objective of improving the decision-making process during aircraft operations aiding the pilot by increasing its situational awareness tasks and improving the process. The e-Pilots system aims at co-existing with the current crew-alerting systems on-board. The proposed tools apply cognitive computing (machine learning techniques) to identify situations that require further actions from the pilot.

The e-Pilot use-case

In order to assess the relevance and potential efficiency of the mechanisms proposed within this project, historical data on accident/incident information has been analysed. Between 2008 and 2017, 49% of fatal accidents involving commercial jets worldwide took place during a final approach and landing. This statistic has not changed throughout the last 40 years despite efforts both from industry and regulators.

These accidents could have been prevented by a go-around instead of continuing approach and landing (Flight Safety Foundation, 2012). Making a timely decision to execute a go-around manoeuvre has the potential impact on the overall aviation industry accident rate (Boeing, 2014).

Approach and landing accidents/incidents include runway excursions (overruns or lateral excursions), controlled flight into terrain (including landing short of runway) or loss of control. The project’s use case is focused on conducting go-arounds based on the risks of runway excursions, which has been identified as one of the top safety concerns shared by EASA Member States and the US National Transportation Safety Board & Federal Aviation Administration (FAA).

Characterizing the pilot’s cognitive state

Figure 1

In terms of mental resources, pilots can be characterized by their cognitive state. The grouping is done by a set of nine factors arranged into three categories: factors resulting from an interaction between individual and environment (workload, fatigue, stress), factors referring to cognitive processes (attention, vigilance, situational awareness), and factors resulting from individuals acting within a team (communications, teamwork, trust). These nine factors had been previously identified by experts as factors that could have a large impact on ATCO (air traffic controller) performance. Moreover, each of these factors do not act independently, but are also coupled among them, showing bi-directional or one-direction relations, as can be seen in figure 1.

The resulting e-Pilot decision support system will adapt its strategy according to the cognitive state of the pilot. Consequently, it will be key for the successful implementation of the project to be able to detect when the pilot’s cognitive state is degraded.

In order to achieve this, the e-Pilot team has designed a net of physiological sensors (based on electrocardiogram, airflow characterization, electroencephalogram, landmarks recognition by computer vision,) to set the basal state of subjects under study and analyse how it evolves in different workload scenarios.


First results

Figure 2

The system proposed by the project regarding the pilot’s cognitive state has been validated using cortisol analysis and the NASA TLX survey (a multidimensional assessment tool that rates perceived workload in order to assess a task) while subjects were playing three different mathematical games that required the execution of concurrent operation in parallel (see figure 2 B).

As seen, a higher workload perception was translated in a decrease of performance (figure 2 C). In this stage of the project, the physiological sensor net is being used to characterize (in a flight simulator) how a set of pilots react in different complexity scenarios. This data will be used to train, individually, the e-Pilot algorithm that will detect and predict a degradation of their cognitive state.


Workload perception vs actual Workload

Figure 3

Preliminary results were obtained from a set of experiments in which project members validated workload perception versus actual workload. The subjects under study had to perform three different experiments for 30 minutes in which they had to manage several processes simultaneously. The experiments, based on the Dual N-back test, consisted of (see figure 3):

Test 1 WL (workload) low: A square appeared every 4.5 s in one of eight different positions on a regular grid on the screen. By using the keyboard, the subject had to indicate if the position of the square was the same as the one that was presented just before.

Test 2 WL medium: An integer number between 0 and 9 appeared every 4.5 s on the screen. For each number, a math operator (add, subtract, multiply, or divide) was presented via an audio message. The subject had to apply the math operation on the shown number and on the one presented before (1-back task). The result of the calculation had then to be entered on the keyboard.

Test 3 WL high: the two former position and arithmetic tasks were combined. An integer number between 0 and 9 appeared every 4.5 s in one of the eight different positions on a regular grid. For each number, a math operator (add, subtract, multiply, or divide) was presented via an audio message. The subject had to respond if the position of the shown number was the same as the one that had been presented two positions earlier (2-back task). In addition, the subject had to apply the math operation on the shown number and on the one that had appeared 2 positions before. The result of the calculation had then to be entered on the keyboard.

The performance of each subject in each of these experiments was evaluated (providing a mark after the 30min activity). Moreover, each subject made a self-assessment of the workload perceived by making use of the NASA TLX multi-dimensional evaluation tool.

Preliminary conclusions

Figure 3 presents how high-level workload perception (figure 3 top) impacted negatively on the subjects’ performance (figure 3 bottom). This opens a window of possibilities in order to address two key aims: comprehend the role of memory items within decision-making regarding amount of workload, and identify which is the threshold in which situational awareness could decline.

No Comments

Sorry, the comment form is closed at this time.