Data Logging in Autonomous Driving Easily Leads to Huge Efforts in Further Processing and Selecting the Right Data
Data logging in autonomous driving is a challenging discipline and represents a significant budget item in the development and testing of autonomous or driver assistance functions. The costs are driven by the complexity of the world on one hand and huge data volumes on the other.
The complexity of the world will not change but the data volumes can be reduced – significantly – using some smart techniques to optimize the logging process.
In a prequel to this blog post, we addressed how to reduce data volume by eliminating redundancy in the data (see the previous blog post here). Because the data is collected for development and validation of functions based on neural networks, we use neural networks as well to determine which data is redundant and which is novel. The approach is very efficient and most importantly, very effective. In most driving situations on the highway, 80% to 90% of data volume can be eliminated.
Today, we describe another method of how to record data with consideration of its future use. The principle is based on triggered recording (unlike continuous recording) that stores only data that meets certain conditions of data relevance. Data relevance is usually related to the operational design domain (ODD) of the development program and description of the environment and situations where the vehicle is driving. In addition, the amount of data already collected for the ODD is considered.
Recording Only the Data of Interest
Triggered recording refers to an approach of data recording when the data is stored (written to a storage device) only when a certain predefined event occurs. The recording can be stopped by another event or by a time-out. Unlike triggered recording, continuous recording stores the data the whole time, until the storage device is full.
Triggers can be internal or external. Internal triggers are in-vehicle events (originating within the vehicle, e.g., emergency braking signal, velocity signal, acceleration signal) and external triggers are determined by particular traffic situations or traffic scenes around the ego vehicle. External triggers can be lighting conditions (e.g., night-time, glare), surrounding road infrastructure (e.g., highway, intersection, traffic sign, toll gates), or the existence and behaviour of the traffic participants.
During road testing, the most interesting data to record is from situations when the system under test does not perform as required. Such situations must be analyzed in detail and the root cause investigated. The required triggers are defined based on driver input, direct ECU output, or indirectly based on internal ECU variables.
Starting the recording exactly when the event occurs might provide an incomplete picture. Imagine that the trigger is set to record data only if a brake signal is detected. If this brake signal is the result of an advanced driver assistance system (ADAS) function that relies on the obstacles detected, it will be difficult to improve the function if it is braking too late. In that case, the situation occurring before the braking signal must be recorded and used for improving the detection part of the function.
Therefore, it is important to have a data sequence preceding the trigger event. To address that, it is common to use circular buffers to implement such ‘pre-trigger’ recording. In that case, data is continuously written to a circular buffer (a specific part of a RAM memory) and if no trigger occurs before the buffer is full, the data from the beginning of the buffer is overwritten. When the trigger occurs, the entire buffer is emptied, and the recording sequence from the buffer is written to a storage device.
Trigger settings are an inherent part and a feature of a data logging system. However, their configuration depends on the project needs derived from the targeted ODD and the state of the development. Autonomously controlling a vehicle in good lighting conditions and sparse traffic is relatively easy to achieve in the early development phase. There is enough data for training and there are many situations to validate it. However, the reliability decreases when considering heavier traffic with various traffic participants and unknown manoeuvres. Similarly, glare or dense road construction can reduce the performance of the system. Therefore, such situations might be in the focus of data collection and road tests while the most common situations can be excluded from recording since their data is already sufficiently represented in a data pool.
Before we move on to requirements and solutions on implementing recording triggers, there is one thing to highlight. Whatever logic is behind the implementation of triggers (decision if the trigger fires or not), the decision must happen in real time. There is no slack and all the data processing for the trigger evaluation, all the comparison logic, and the trigger execution itself must happen in real time.
Later, we will go into more detail on the implementation of triggers with the dSPACE data logging solution based on AUTERA and RTMaps. The flexibility of the tools provides comprehensive means of setting a broad variety of triggers with various complexity covering the entire range of potential trigger events.
There is no slack: All the data processing for the trigger evaluation, all the comparison logic, and the trigger execution itself must happen in real time.
The simplest way to trigger a recording is to provide an interface to a driver or a test engineer to manually control the recording. It can be a hardware button or a control software application. The recording can be controlled by a logging configuration application, but this is usually not very convenient as such applications offer many functionalities and the GUI is correspondingly more complex than necessary for a simple logging control. dSPACE is addressing this need for a simple control interface with a dedicated product that can be used together with AUTERA and RTMaps. It is a mobile application called RTag that can be installed on a tablet mounted on a dashboard. Therefore, it is easily available and does not distract the driver. Recording can be started and stopped by a simple touch. RTag offers tagging audio input too that makes the usage even more convenient.
RTag is not just a ‘software button’ but it allows monitoring the status of the logging system and tagging of the data for successive steps.
Automatic Triggers Based on Bus Signals and ECU State
The most common way to trigger a recording is based on analysis of a particular bus signal or network communication. For example, the recording trigger can be set to record only if the vehicle is moving with velocity above 20 km/h. In such cases, a velocity signal is extracted from a CAN bus and periodically compared to the threshold. Another example is recording in a predefined geographical area only. In that case, the position received from a GNSS receiver is used as a trigger signal.
The prerequisite for such a trigger setting is an ability of the logging system to extract information from buses and networks, most commonly CAN, LIN, FlexRay, and Ethernet. Having a communication matrix and appropriate tools solves the issue.
In E/E architectures for ADAS/AD, smart sensors often provide an object list describing the objects around the vehicle via buses and networks which might be also leveraged as a recording trigger. Situations with traffic participants around are usually more interesting than empty roads. Furthermore, the information about the position of the objects can be taken into consideration for a trigger as well (e.g., if the moving object is within 50 m of the ego vehicle).
AUTERA bus and network interfaces with RTMaps provide an easy way to extract bus signals and use them for recording triggers. As a demo example, we have selected a CAN bus and object list from a front and rear camera. The trigger condition is defined as following: start the recording when a truck is detected less than 30 meters from the ego vehicle (in any direction).
We have used a simulator to generate the CAN signals and AUTERA with RTMaps as a logging system. For the entire demo and details on the implementation you can check the full video in the Learning Center platform.
The AUTERA bus and network interfaces provide an easy way to extract signals from automotive buses such as CAN, LIN, FlexRay, and Ethernet and use them for recording triggers.
In addition to bus signal, AUTERA and RTMaps offer the possibility to monitor signals in ECU and set relevant triggers. Via XCP (universal measurement and calibration protocol) and ECU measurement interfaces, the ECU input can be logged along with the rest of the data and can be used as a trigger at the same time. This allows recording of sensor data only from situations when the ECU under test does not perform as required (e.g., the confidence level of perception function is below a certain threshold).
Automatic Triggers Based on Perception
A little more advanced logging system with sufficient computing power does not have to rely on smart sensors to detect the objects around and analyze the situation. The detection algorithms can run directly on the logger and in such a way to provide independent assessment of the scene for logging purposes. While AUTERA and RTMaps provide the suitable platform and framework, the advanced algorithms themselves run in dedicated components.
Detection of objects from cameras is provided with a 2 D detection component. The RTMaps component contains a neural network model and outputs an object list that can be processed in the same way as in previous example of processing bus signals. A step further are trajectories and manoeuvre detections that allow you to set triggers on dynamic scene situations.
Detection of objects in 3D point clouds from lidar data on AUTERA
In similar way, data from lidars can be used and analyzed on the data logger directly providing relevant triggers. Here, RTMaps can be extended with components from Exwayz. These components not only detect the objects in 3 D space but also extract trajectories or the ego position using SLAM algorithms. See possibilities of various compatible algorithms in RTMaps AI Store.
AUTERA and RTMaps are the appropriate platform for automatically creating triggers from the outputs of sensor perception algorithms, which in turn can run on AUTERA or a connected ADAS ECU.
Leveraging Triggered Recording to Save Cost and Time
Versatile triggers provide a powerful means for data reduction and thus, reduction of costs during the recording process. With limited effort and an increase in the complexity of the logging configuration recording triggers bring significant benefit. To fully leverage the potential of triggers for data recording, a suitable platform is necessary; a platform that offers the flexibility in analyzing data content from various data sources and a platform that provides sufficient computational resources to execute the necessary pre-processing and data analysis in real time. The AUTERA data logger, which can also execute complex algorithms for the precise generation of trigger events, provides a suitable solution for these tasks. It is complemented by the software tools RTMaps and RTag, which support the convenient creation of automated and manual triggering.
This compact in-vehicle solution is easy to use. Contact dSPACE to learn more about how AUTERA and the software tools can optimize the data acquisition process and costs for your specific requirements.