An innovative methodology to evaluate workload

Assessing operator workload is critical in the industry, especially during the design process of a new system or a new tool or interface. The designer has to ensure at every stage of the process that the impact of the design... [ view full abstract ]

Assessing operator workload is critical in the industry, especially during the design process of a new system or a new tool or interface. The designer has to ensure at every stage of the process that the impact of the design will not produce a level of workload that may impair the performance and safety. This paper describes an innovative methodology applicable in an industrial context such as the railways. The Workload Assessment Methodology (WAM) was firstly developed in the context of the design process of the Airbus military aircraft (A400M). It combines traditional and well validated measures of mental workload such as cardiac measures, subjective evaluation (NASA-TLX) and task observation. However, based on a literature review, it is clear that none of these measures gives a clear benchmark of an acceptable workload level although this is an important criterion for the designer to decide changes in the design or in the procedure. The main innovation brought by this methodology is the individual calibration of the measures based on a complex task (the Multi-Attribute Task) which simulates several of the cognitive processes involved in real tasks such as those of pilots or train drivers. The Multi-Attribute Task (MAT) is presented to the operators before the evaluation with four different scenarios of increasing cognitive load. The fourth scenario has been designed to produce a very high level of workload, not manageable to maintain nominal performance. The workload evaluations are applied during these 4 scenarios and the workload measures are « calibrated » on these four scenarios. It is then possible to identify, for each participant, four associated levels of workload which are thus customised to each individual. Then, the measures are used during the evaluation together with video recording and task observation. All the data are synchronised using a dedicated software for subsequent analysis. After the evaluation, a preliminary data processing enables to identify all the specific phases associated with a significantly high workload, i.e. higher than the level associated with the fourth level of the MAT scenario. During a debriefing phase, specific questions are asked to the operators to analyse the causes of these high levels of workload (design, tasks sharing, procedure, lack of training…). This methodology was validated with aircraft pilots and applied during the design process of the A400M. Results suggest that the calibration methodology was helpful to identify significant levels of workload on the NASA-TLX. Heart rate was sensitive to high demanding tasks (e.g. air refuelling). From the results, a decision tree combining various criteria taking into account all the data collected has been built to assist the designers in their decision making. These results show that the method was useful to identify high workload periods and from this identification, to propose recommendations for the designers. This methodology can be adapted to other specific domains. Further refinements and adaptations of the method will be discussed.