IRT SystemX announced the launch of its collaborative R&D project Cockpit and Bidirectional Assistant (CAB). This is the second project of the “Artificial Intelligence and Augmented Engineering (AI2)” program launched on February 4, 2020. It brings together 4 manufacturers (Dassault Aviation, Orange, RTE, SNCF) around concrete use cases for an enhanced decision support for complex piloting.

How to develop a virtual assistant that teaches “from” and “to” the operator of critical systems or sensitive networks (vehicle, network, infrastructure) to control? By hybridizing artificial intelligence (AI) and human-machine interaction (HMI), the CAB project aims to improve man-machine cooperation.

It aims to develop an intelligent cockpit demonstrator, integrating a bi-directional and multimodal virtual assistant to support the operator in his decision making. Among the main scientific and technological aspects of the project are: the representation and hierarchization of knowledge, the relevance of the analysis and recommendations, the explicability and personalization of the expertise according to the operator’s profile.

Research work in response to growing demand from operators of critical systems or sensitive networks

SystemX, the only IRT dedicated to the digital engineering of the systems of the future, is launching the CAB project, the second project of its IA2 research programme. It aims to define and evaluate an intelligent cockpit integrating a virtual agent that will increase in real time the operator’s decision-making capacities when faced with complex and/or atypical situations, and whose particularity is that it learns from and to the expert. This research work linking the fields of AI and HMI is unprecedented and responds to a growing demand from operators of critical systems or sensitive networks.

The piloting of critical systems (airplane, car, train, etc.) or sensitive networks (transport, electricity, telecom, etc.) implies for the operator to manage a large amount of data coming from the system to be piloted, related to its environment and the complexity of the situation in which it is located. This leads to a significant increase in its cognitive load.

Prototyping a generic test cockpit

Automation and the provision of virtual assistants are commonly used in situations where the final decision rests with humans. The quality of cooperation and complementary learning between man and his virtual assistant are essential. The objective of the CAB project is the development and prototyping of a generic test cockpit – open in terms of industrial applications – in which it will be possible to evaluate the forms of exchange between the expert and an AI that continuously learns, both from the information flows received and from the decisions taken by humans.

The explanatory aspect of AI recommendations is central to this project to give added value to the operator in his decision making. The virtual assistant will be able to determine the profile of the operator, his level of cognitive load, and adapt the information flows to the operator in order to manage a complex and/or atypical situation in the best conditions.

Scientific locks to be lifted

Among the main scientific obstacles to be overcome: the representation and development of knowledge models, the bidirectional learning of the operator and AI, the characterization and understanding of a complex situation by the virtual assistant and the relevance of the analysis, the personalization of the recommendations to the situation and to the level of expertise of the operator (junior, senior), the explicability of the recommendations made by the virtual assistant or the acceptability by humans.

This project will also study the notion of multimodality, to understand with which sense(s) of the operators it is most relevant to make the bidirectional assistant interact: visual, auditory/voice, tactile or combined interaction modes, or even operator monitoring (eye tracking, biometric data, etc.).

4 use cases studied

• Dassault Aviation: providing assistance to aircraft crews in the face of increasing mission complexity, while consolidating safety.
• Orange: benefit from a two-way wizard that brings together measures to check that telecom applications and infrastructures are operational and efficient, warn of any current or anticipated malfunction, and display measures that have reached a critical threshold to identify and repair the cause of the malfunction.
• RTE: to offer operators a learning assistant, adapted to their level of expertise and the criticality of the situation, to assist in decision-making in the context of power system operation in anticipation and in real time, and to facilitate cooperation within the control room, between different professions and when handing over from one team to another in 3/8. Objective: Guarantee the continuity of service of particularly critical and complex systems.
• SNCF: Design a proof of concept for a bidirectional virtual agent enabling real-time increase in the capacities of operators (PCC and/or Driver) facing complex and/or atypical situations under strong time constraints.

At the end of the project, the following will be developed/proposed: an open intelligent cockpit demonstrator to address various use cases, datasets, algorithms relevant to the hybridization of AI with multimodal HMIs and the proof of concept of the use cases proposed by the 4 industrial partners.

“This project aims to design hybrid systems involving the use of artificial intelligence techniques (knowledge representation, machine learning, decision support) and bi-directional multimodal Man-Machine Interfaces (man-machine cooperation, customised GUIs, adaptive GUIs, data visualisation), while taking into account human factors (cognitive load, control modes, human model).

Two-way virtual assistants are designed to make operators even more efficient. By learning from them and anticipating future actions, they are part of a true cooperation for operational efficiency,” explains Walid Achour, CAB project manager.

The IA2 programme

CAB is the second project of the “Artificial Intelligence and Augmented Engineering (IA2)” programme launched by the IRT SystemX on 4 February 2020. This programme is unprecedented both in its scope and format and meets the expectations of industrialists who design, develop and validate systems.

Through 6 collaborative R&D projects, headed by an upstream project to pool scientific results, coordinated by Marc Schoenauer, a renowned researcher at Inria Saclay, the IRT SystemX proposes to develop solutions in 5 years that combine three approaches to modelling and simulation: physical modelling of systems, the use of business knowledge or behaviour models expressed by experts, and machine learning models based on real data.

Translated from SystemX lance le projet Cockpit et Assistant Bidirectionnel (CAB), une aide à la décision augmentée pour les pilotages complexes