How does a driverless heavy-duty vehicle move safely through rough terrain? SANY provides the answer to this question – with the development of a drive control system for an autonomous dump truck. For the impressive proof of safety, SANY relied on simulation solutions from dSPACE.
The automation of traditional open-pit and underground mining by using autonomous work and transport machines opens up promising prospects. The mining industry's growing demand for unmanned vehicles is supported by initial findings that mining productivity and efficiency can be increased, worker safety improved, and the environmental impact of mining minimized.
The autonomous dump truck from SANY is equipped with a range of perception sensors.
Dump trucks on the way to the mine.
High Level of Safety for Autonomous Transportation
For the visionary concept of autonomous transportation to become a reality, one important requirement must be met: Absolutely reliable controls for autonomous driving are needed. An exciting task for SANY, which is developing an autonomous dump truck for use in mining.
“The dump truck is equipped with an autonomous driving (AD) control unit, high-precision satellite navigation (global navigation satellite system, GNSS), and various sensor technologies such as radar, lidar, and camera," explains Huijun Zhen from SANY. With the help of these systems, it can detect its surroundings and follow the routes planned on a cloud platform. It finds the best route through the constantly changing mining area and avoids obstacles. Further features in "The control system of the autonomous dump truck from SANY".
However, a mine is a very complex environment in which a standardized transport infrastructure cannot be assumed – a particular challenge for the autonomous dump truck, which, for example, must be able to transport several hundred tons of payload between the extraction site and the conveyor system under adverse conditions in an absolutely safe, reliable, and economical manner.
Efficient validation of newly developed control unit functions through tests with the real vehicle is therefore not possible under safety-critical conditions.
The control system of the autonomous dump truck from SANY
Navigation and localization
Use of high-precision maps, BeiDou navigation, GPS, and other technologies for precise positioning and autonomous navigation of mining vehicles.
Perception of the environment
Fusion of lidar, camera, and radar data, real-time detection of the environment to ensure the safety of mining vehicles.
Cloud platform planning
The cloud platform enables intelligent multivehicle planning and helps to optimize the efficiency of transport in the mine.
Route planning and decision
The optimum route for the dump truck is planned based on the results of the sensor fusion. This ensures efficient and safe operation of the mining vehicle.
Control and implementation
Highly developed control algorithms for precise control of the mining vehicles, ensuring safe travel along the planned routes.
How can the AD control unit be validated for use in mining?
SANY opted for a simulation-based approach with two combined test methods to validate the AD control unit of its autonomous dump truck:
- Hardware-in-the-loop (HIL) simulation
- Vehicle-in-the-loop (VIL) simulation
HIL simulation takes place in the development laboratories of ECU manufacturers and vehicle manufacturers and is used in early development phases to continuously check the maturity of the algorithms under deterministic real-time conditions. It is therefore the key instance for the approval tests of new software releases through to the production approval of the AD control unit.
The VIL simulation enables safety-critical tests to be carried out safely with the real vehicle. For this purpose, a simulator is carried along that creates a virtual world for the vehicle's perception sensors, in which the vehicle, including the AD control unit, is evaluated on a test track or in a mine. A safety driver is responsible for monitoring the correct function and intervenes in the test process if necessary.
First, the autonomous dump truck has to prove itself in the virtual world.
Digital twin of the mine: the workplace of the autonomous dump truck.
In Search of a Simulator with Talents for Autonomous Driving
To validate the autonomous dump truck, the synchronous simulation of the sensor system is of fundamental importance. At the same time, all vehicle functions and the vehicle environment must be simulated realistically in order to be able to take a comprehensive view of the vehicle's behavior by simulating a realistic, digital twin. Last but not least, the simulator must provide interfaces for all necessary input and output signals of the control unit to be tested. SANY opted for a HIL simulator from dSPACE after successfully evaluating the system's suitability for elementary requirements. This includes:
- Real-time simulation of heavy-duty trucks vehicle dynamics and traffic environment
- Time-correlated simulation of the perception sensors
- Generation of a realistic virtual test environment
Procedure for HIL Validation
The HIL simulator designed for SANY's requirements profile is based on the SCALEXIO real-time system, on which the simulation models of the ASM (Automotive Simulation Models) tool suite are executed. The raw sensor data for the camera and lidar are calculated on four sensor simulation PCs using the sensor-realistic simulation software AURELION. As three different lidar sensor types are installed in the vehicle, individually implemented and parameterized lidar models were created to simulate the different properties such as range, scan pattern, field of view (FoV), frequency, etc., in the sensor simulation with physical accuracy. The radar data is processed in the form of object lists with the ground truth sensor models from ASM Traffic. The ASM Truck driving dynamics model provides additional signals for IMU (inertial measurement unit), navigation, etc. The SCALEXIO real-time platform processes all data synchronously and makes it available to the control unit via interfaces such as automotive Ethernet and the Environment Sensor Interface (ESI) unit extension. In this test setup, the ESI unit converts the simulated data from the four camera sensors into signals and protocols in accordance with Gigabit Multimedia Serial Link (GMSL) technology. A detailed overview of the simulator components is summarized in "HIL tools and services in use at SANY".
A Realistic 3D World Is Required
To be able to carry out tests in the virtual world, a digital twin of the mine first had to be created. "For this purpose, we scanned the approximately 3x3 km area with drones, which recorded both the contour and the texture of the surface. With the help of dSPACE tools, we were able to derive a model of the drivable surface from this data, including gradients, inclinations, and surface properties," reports Huijun Zhen. In the process, 4 million polygons were processed. As the autonomous dump truck also uses public roads for feeder trips, the model of the mine was extended with imported map data. The autonomous dump truck itself was integrated into the virtual world using the ASM Truck simulation model. The model can be used to configure and parameterize important features such as double axles, twin tires, and loading so that the driving dynamics of a heavy dump truck can be realistically simulated off-road.
Which tests can be carried out with the HIL simulator?
To be able to carry out virtual test drives with the model of the autonomous dump truck, the developers at SANY have defined a catalog of relevant tests and processes.
The test catalog enables SANY developers to carry out functional tests efficiently. It can be combined with FIU (Failure Insertion Unit) tests to check the behavior in the event of technical defects.
How does the VIL simulation contribute to validation?
In the vehicle-in-the-loop (VIL) test carried out by SANY, a simulator is carried in the real vehicle. It generates a georeferenced virtual world that is available to the AD control unit instead of the real sensor data. This makes it possible to test the algorithms for route planning and decision-making under real conditions. By feeding in obstacles and errors to the virtual world, scenarios that would lead to collisions in the real world can be safely tested. "With the VIL system, we can easily place virtual dynamic and static objects in the real mining world as we need them," summarizes Huijun Zhen.
The VIL simulator was realized with a robust dSPACE AutoBox designed for vehicle use, in which a SCALEXIO system functions as a real-time computing platform. The virtual world and the vehicle are calculated on this with the Simulation Tool Suite ASM. The AD control unit "sees" a virtual environment that is simulated with the ASM sensor models for camera, radar, and lidar and made available by the simulator as object lists via the CAN bus. The simulator in turn is supplied with signals from the real vehicle so that a closed, realistic vehicle dynamics and traffic simulation is created. Details of the simulator components can be found in "VIL tools in use at SANY".
The special charm: Selected tests such as collision avoidance, which were defined for HIL testing, can also be reused with the VIL system in the context of the real world, for example, to evaluate the behavior of the loaded vehicle when braking or swerving in unpaved, rugged, and hilly terrain, or to investigate the influence of non-linearities.
Evaluating the Validation Project for the Autonomous Dump Truck and the dSPACE Tools Used
The successful completion of the project to validate the AD control unit for the autonomous operation of a mining dump truck was a significant milestone for SANY. "Through the intensive use of hardware-in-the-loop (HIL) and vehicle-in-the-loop (VIL) simulation, we were able to comprehensively test and verify the functionality and safety of the AD control unit," confirms Qiyan Jin. Thanks to thorough testing with these simulation methods, the dump truck drives autonomously, safely, and reliably in the mine.
dSPACE's contribution to safe autonomous driving:
The tools and services from dSPACE contributed significantly to the success of the project. "The simulation solutions are characterized by high reliability, quality, and flexibility," summarizes Qiyan Jin. An essential foundation for the simulation-based approach is the effortless creation of highly accurate digital twins of the vehicle and mine. "The synchronous simulation of different sensor types with physical accuracy is crucial for success, as this is the only way to create a plausible virtual world for the perception and decision-making algorithms," emphasizes Huijun Zhen. He adds: "The coordinated software and hardware tools from dSPACE are not only valuable and useful for development and testing, but also indispensable when there is no room for compromise in terms of safety."
User-friendliness and efficiency of dSPACE tools:
The dSPACE tools proved to be easy to use, which enabled a quick familiarization. Within six months, two members of the project team were able to complete the entire validation of the AD control unit and keep to the specified schedule. The fast learning curve and intuitive usability of the dSPACE tools made a decisive contribution to achieving the project goals on schedule.
Simulations show the operation in the mine. The atmospheric backdrop is created with the sensor-realistic simulation software AURELION.
Whether real or virtual – With the right safety solution, autonomous dump trucks drive safely in the mine.
"The seamless integration of the realistic simulation environment with the ECU of the real vehicle enables fast validation of the algorithms and iteration of the software, which significantly reduces the development time."
"The hardware-in-the-loop (HIL) simulator from dSPACE has all the features needed to supply our AD controller synchronously with the required sensor signals and protocols. It paves the way for safe autonomous driving."
"With the highly accurate sensor-realistic simulation software AURELION, algorithms for the visual perception and decision-making of autonomous driving can be reliably verified."
"The coordinated software and hardware tools from dSPACE are not only valuable and useful for development and testing, but also indispensable when there is no room for compromise in terms of safety."
Conclusion and Outlook: Reliable and Trustworthy in the Mine
The project impressively demonstrates how reliable and effective dSPACE tools and services are for autonomous driving. The user-friendliness and efficiency of the tools make dSPACE a valuable partner.
Following this successful project, SANY will now focus on developing more autonomous machines to help its mining customers ensure a safer working environment and increase efficiency.
SANY is planning to use software-in-the-loop (SIL) simulations during development for these projects. In addition, the SIL method supports the simulation of complex software architectures, such as those used in the field of software-defined vehicles (SDV). To ensure a seamless SIL-HIL transition, SANY is evaluating SIL solutions from dSPACE.
Courtesy of SANY
dSPACE MAGAZINE, PUBLISHED JANUARY 2025
The article was created in close cooperation with the following persons:
Huijun Zhe
Huijun Zhen is head of the development team for autonomous dump trucks at the SANY Group in Beijing, China.
Qiyan Jin
Qiyan Jin is an engineer in the development team of the SANY Group's autonomous dump truck in Beijing, China.
HIL tools and services in use at SANY
- dSPACE Full-Size SCALEXIO system for real-time simulation
- 2 SCALEXIO Processing Units
- 1 SCALEXIO LabBox: Versatile, scalable interfaces
- 4 sensor simulation PCs: Simulation of 4 camera and 4 lidar sensors in real time
- 4 high-performance GPUs: Accelerating the simulation of physics-based camera and lidar models in AURELION
- 1 Environment Sensor Interface unit (ESI unit): Time-correlated transformation and feeding of raw sensor data
- Simulation Tool Suite ASM (Automotive Simulation Models)
- Open, modular, and scalable real-time simulation models: simple expansion and parameterization
- ASM Truck, ASM Environment: Simulation of vehicles, chassis, and terrain
- ASM Traffic: Simulation of the traffic environment as well as radar object lists, navigation data, etc.
- Sensor-realistic simulation software AURELION
- Generation of camera and lidar raw data based on physical principles
- Realistic 3D visualization of the virtual scenario
- PTP (Precision Time Protocol) and PPS (Pulse Per Second) for time-correlated processing of relevant signals
- dSPACE Engineering: Importing a real mine scenario into AURELION for sensor simulation
VIL tools in use at SANY
- SCALEXIO AutoBox for real-time simulation in the vehicle
- SCALEXIO DS6001 Processing Unit
- SCALEXIO CAN board
- Simulation Tool Suite ASM (Automotive Simulation Models)
- ASM Truck: Simulating a "digital shadow" of the real dump truck
- ASM Environment: Transferring the terrain of the real world into the simulated virtual world
- ASM Traffic: Simulating dynamic and static objects, incl. ground truth (GT) sensors for camera, radar and lidar object lists, for example
