For a better experience on, enable JavaScript in your browser. Thank you!

Resolving Sensor Interface Challenges for Autonomous Applications

Jacob Perrin, Applications Engineer, dSPACE Inc.

April 25, 2017: In recent years, the demand for advanced driver assistance systems (ADAS) and active safety systems has reached an unprecedented high. As companies in the automotive industry look to break into the autonomous vehicle market, engineers are spending much time and effort trying to integrate the large number of available sensors in a real-time manner.

Those who investigated sensor fusion realize the challenges related to topics such as multi-threaded programming, data sample timestamping and resynchronization, data latencies measurement and estimation, system optimization and performance assessment, code reuse, and software applications maintenance. The list is simply endless.

An attempted solution for some companies has been to develop and maintain in-house tools or adapt existing tools that are not suited for the real-time sensor fusion problem. The companies that tried to address this challenge with the development and maintenance of in-house tools often found them inadequate.

Model-based design has been an important pillar of the automotive embedded development process for a long time. It is well known that model-based design approaches increase efficiency in development, improve product quality, and reduce cost.

A similar approach is necessary for taking on the challenges of sensor fusion that will help engineers to focus on developing core algorithms by taking away the efforts for writing sensor interface drivers, data synchronization, etc.

dSPACE has found the solution to this engineering challenge with Intempora, a Paris-based company, and its product RTMaps (Real-time Multisensor Applications). As a result, the company has partnered with Intempora for international distribution to make RTMaps available to all our customers directly from dSPACE. 

RTMaps enables users to efficiently prototype new algorithms and new systems by providing a graphical, modular environment. This way, engineers can easily test and evaluate functions based on different sets of sensors, in different configurations, and with different sets of processing and data fusion strategies. RTMaps data logging and real-time playback capabilities allow engineers to work offline on real data, in a reproducible manner to develop and benchmark sensors and algorithms.

As the number of US-based adopters for this tool has grown, we have seen a wide variety of use cases for RTMaps. When developers first begin to work with the ADAS pool, they mainly deal with the problem of interfacing to the various sensors from various vendors. This is a daunting task and it may take weeks or months to develop proper interfaces to collect and synchronize the data and make it available for further processing.

With RTMaps, the task of gathering and recording data from a sensor is done in only a few clicks. This means that there is no redevelopment of the same basic device driver/SDK interface, on which nearly every company seems to have spent engineering resources in the past without actually generating any intellectual property. In the RTMaps environment, you simply drag and drop the RTMaps component for your desired sensor into the diagram, click run, and you are able to visualize, manipulate, and integrate the data stream.

RTMaps – a modular, multithread frame-work for real-time, multisensor applications.

After extracting the data from the sensor, the question becomes what to do with it? Software components that will consume the data in some way are located downstream from the initial sensor interface. If no software component currently exists for your application, one can be easily created by using the provided C++/Python SDK. You can also use SDK to create components for additional sensor interfaces. Often, sensor providers include a library for interfacing to a sensor. These APIs can be exploited to have a component built in hours or days.

dSPACE and our partners at Intempora have experience in creating components for customers on demand. When we create a component, it is generally integrated into the RTMaps rolling update scheme within a few days so all RTMaps users are able to use the component. However, quite a few pre-algorithmic components are included, which can be exploited for use in your own application. For example, many OpenCV functions are included as a component package by default, in addition to a wealth of other processing components. 

Many of our RTMaps customers today simply go into the field, record data, and then use that data in the lab to refine their algorithms or evaluate the performance of sensors. One of the best features of RTMaps is the ability to collect data in a standardized format and then synchronously replay that data as if the stream were coming directly from the sensors. Video files can be stored as raw images (AVI, MPG, binary, etc.). Sensor interfaces can be saved directly to .m files to use them in the MATLAB environment. Customers can even define their own recording formats. A task that has become much easier for these users.

RTMaps also offers the option to run in a distributed manner allowing for the data to be collected from the different platforms. For playback, the player component simply points to the .rec file, which aggregates the timestamps from each of these independent data sources, and synchronously replays the data. Timestamps can be created by either utilizing the system clock or using a device clock, such as UTC time coming from a GPS.

With regard to the system clock, it is necessary to mention operating systems supporting RTMaps. RTMaps is middleware, meaning that it will perform only as well as the underlying system (processor, hardware accelerators, operating system optimizations, etc.).

For ideal performance, RTMaps uses multithreaded, copyless, event-based processing mechanisms. The current system architectures supported are x64/x86 on both Windows and Linux. However, Intempora will realease a new version of RTMaps soon, called RTMaps Embedded. This version will extend the RTMaps engine to target ARM processors in an embedded cross-development-like manner. With this, Intempora is gearing itself towards being able to support targets with production-intent operating systems such as QNX and Green Hills.

As Intempora prepares to release its new software, dSPACE will be releasing a new hardware platform optimized to be a target embedded platform. This new platform will feature interfaces such as GigE, GMSL, CAN/CAN FD, LIN, GNSS/IMU, USB, HDMI, WLAN, LTE, and Bluetooth. All of these are paired with a six-core ARM CPU and the latest NVIDIA GPU. This platform, known as the MicroAutoBox Embedded SPU (Sensor Processing Unit), is the robust, in-vehicle solution our customers have been asking for. When included in the stack with MicroAutoBox II, the two combined platforms allow customers to prototype both the perception and control portions of their autonomous solutions.

RTMaps is an exciting addition to the dSPACE tool chain, and its prevalence is definitely growing. I have really enjoyed working with our customers to build their autonomous system solutions, and our customers have all been pleased with the results. One particular customer put RTMaps to the test by letting his two-man, US-based team compete against a dozen foreign engineers, with both teams attempting to interface a variety of sensors to create a solution for perception in their vehicles. Within weeks, the US-based team using RTMaps was able to interface all sensors and place the sensors in a common coordinate frame. Months later, the other team was still working on hand-coding the preliminary interface to even just one of the sensors. By this time, our customer had nearly completed the project and became a champion for RTMaps.

RTMaps is a powerful tool, and I look forward to seeing how it will be applied to further the autonomous vehicle race into the future.

Further Information Product Information