Industrial assembly processes can be monitored in real time thanks to the Madeira project, led by CT.

CT engineers, in collaboration with their partner CEIT-IK4, are conducting research to monitor industrial assemblies in real time, achieving greater efficiency, reliability, and effectiveness.

The Madeira project will provide industrial assembly processes with a flexible software platform that integrates virtual reality, augmented reality, and machine learning.

Madeira, whose name stands for ‘Machine And Deep Learning in an Industrial Environment using Augmented Reality,’ is an R&D project developed by CT and CEIT-IK4 with the objective of integrating new technologies into industrial processes to obtain real-time information and increase efficiency.

Currently, for example, during the assembly of an aircraft interior, any unforeseen event such as a shortage of parts in stock or issues with a robot will halt the assembly process until the problem is resolved. Madeira provides the system with complete knowledge of the environment, allowing the robots to adapt and change their mode of operation when faced with any unforeseen event, without needing to stop. They can perform other tasks until the problem is solved.

To achieve this objective, three technological challenges must be overcome: automated systems (industrial and collaborative robots) must be able to perceive their environment; the process must reconfigure itself automatically according to the environment; and both of these points must be fully integrated into the existing overall control of the industrial process within the company.

Based on these challenges, the three main technical objectives are as follows:

Firstly, the application of Deep Learning techniques to the augmented reality matching process, to gain a visual understanding of the industrial environment. To achieve this, the system must have an operator with access to a camera-equipped device (mobile phone or Hololens), which would be used to monitor the entire area where industrial operations are being performed. This device will have software installed, based on Deep Learning (DL) techniques, to recognise the environment and, using CAD models, locate where the parts of interest are, including their angles and scale.

The steps of the system are as follows:

1. Define the type of scenario at which this application will be aimed.
2. Define the most appropriate hardware architecture for the selected scenario.
3. Train the neural networks to recognise industrial images of the environment.
4. Perform image recognition using the CAD model.

Secondly, implement Machine Learning (ML) in the process simulation, to adapt to the variable conditions of the environment. Although this implementation can be carried out in a wide range of different industrial scenarios, there are a series of common steps that are performed either manually or in an automated way using robots.

This objective is considered consecutive and is therefore addressed from a predetermined industrial scenario defined in the first objective, significantly simplifying the identification of the tasks to be simulated in the process.

The goal of this operation is to create a system capable of improving the entire process in terms of development time, ergonomics, efficiency, and any other parameter that might subsequently be considered. To achieve this, a process tree is modelled along with a set of other steps:

1. Generate artificial data from the process simulation.
2. Export these data in a format readable in the ML development environment.
3. Define which technique will be used, whether supervised or unsupervised, along with its subcategories.
4. Implement the ML technique and read the data.
5. Improve the results obtained in ML.

Thirdly, develop middleware for the high-level integration of systems/applications involved in the industrial process. This objective is based on the premise that the device resulting from meeting objectives 1 and 2 must be portable and wireless, allowing it to be carried by the operator from one location to another during the industrial process.

To create a useful tool, achieving a device fully connected with the process is of vital importance. Therefore, it is necessary to develop a solution that guarantees communication between the operator’s device and a robotic arm, an AGV, or any other automated device or equipment involved in the operation.

Each device may have a different communication method and a different Software Development Kit (SDK), making a generic implementation virtually impossible. The aim is to develop middleware that allows a developer to use the robot’s SDK and connect it to the platform through middleware operations. This objective involves the following steps:

1. Evaluation of existing alternatives.
2. Design of the architecture and communication protocols for the middleware.
3. Development of the protocols.
4. Incorporate the various protocols into the architecture.
5. Integrate the architecture with the operator’s device.
6. Develop communication with a specific automated device.

 About MADEIRA

The project, carried out by CT in collaboration with CEIT-IK4, is supported by the Ministry of Science, Innovation and Universities and the European Regional Development Fund (ERDF) as part of the ‘2018 Collaborative Challenges’ programme of the 2017-2020 State Programme for Scientific, Technical, and Innovation Research, under file number RTC-2017-6418-6.