The independent operation of advanced driver assistance systems and, in the future, fully autonomous systems in ever-changing driving environments leads to increasing levels of complexity. The demands on the E/E architecture in the vehicle are growing and causing a shift from independent ECUs to a multizone architecture for highly complex, computation-intensive vehicle functions. In this context, compliance with functional safety (ISO 26262) and Safety of the Intended Functionality (SOTIF) standards is crucial.
Automotive industry is subject to numerous regulations and standards in order to maintain homologation and to prove the absence of an unreasonable risk in product development.
The validation of new systems is undergoing a paradigm shift due to intelligent environment sensors and complex cause-and-effect chains. A significant increase in effort throughout the validation process is clearly evident. This requires not only an increase in the number of tests, but also a comprehensive view of the development and V&V (verification and validation) process. Data-driven development and the close integration of virtual test methods are fundamental pillars in the development of reliable systems, as they enable developers to go through the homologation process reliably and efficiently.
This calls for a suitable and comprehensive test strategy that uses simulation as a central element and consistently reuses the measurement data in the simulation – not only to validate the system being developed, but also to ensure that the virtual test methods can fulfill their purpose. Simulation is not only a tool for efficient development. It is also indispensable for homologation in order to comprehensively validate known risks and new hazards in (as yet) unlearned driving scenarios and thus prove the absence of unreasonable risk.
How do you prove that a System is Sufficiently Safe?
Various standards and regulations form the framework conditions that must be taken into account during development. In line with the principle of precaution, vehicle manufacturers must meet standards for (functional) safety, cyber security and process requirements to demonstrate that they have taken all necessary measures to ensure that a system is as safe as possible before it is allowed on the road. Additionally, the safety argument is based on the idea of positive risk balance, which states that the system under test (SUT) is allowed on the road if it can be shown to perform at least as well as an attentive human driver.
The area of safety is addressed by ISO 26262 and SOTIF (ISO 21448). ISO 26262 ensures the absence of unreasonable risk resulting from any system fault. This includes sporadic hardware failures and technical errors at system level.
A safety-critical application must ensure the integrity of the environment. The functional constraint of the driving function with respect to real use cases and specific conditions under which the system is to operate (operational design domain - ODD) are not considered in the context of ISO 26262. For example, if a system is developed with a camera sensor, it is possible that the sensor technology will function properly from a technical point of view, but will not be able to detect an object in a particular scenario. The consideration of real use cases and potential misuse therefore motivated the publication of ISO 21448 - Safety of the Intended Functionality, which is complementary to ISO 26262.
When does a scenario become dangerous?
The system and its functionality are designed for a specific use case that addresses multiple scenarios. The scenarios may contain triggers that lead to a functional limitation and thus to a dangerous behavior.
As the level of automation increases, the triggers become increasingly complex and difficult to identify. SOTIF includes a risk assessment of scenarios that could lead to a hazard in reality. In addition, the probability of a scenario becoming dangerous is considered. There are four types of scenarios:
- Type 1: known and safe scenarios
- Type 2: known and unsafe scenarios
- Type 3: unknown and unsafe scenarios
- Type 4: unknown and safe scenarios
SOTIF activities aim to systematically identify and iteratively evaluate potentially unsafe scenarios (types 2 and 3). The known unsafe risks (type 2) are evaluated through combinations of suitable test environments and test methods, such as requirements-based testing (verification). Through technical measures in the system design, the residual risks are reduced to achieve a positive risk balance and re-evaluated to provide safety evidence that the residual risk caused by the scenario is sufficiently low. Unsafe and unknown scenarios are identified by simulation and virtual test procedures, such as scenario-based testing (validation). The now unsafe but known risks are subsequently verified.
The result is reduced residual risk in types 2 and 3, while the number of safe and known scenarios increases, thus increasing confidence in achieving SOTIF.
Verifying the Expected Behavior
The basis for a best practice procedure for the development of ADAS/AD functions that meets legal and technical requirements is the ASAM Test Specification Study Group Report . Suitable test methods and environments are listed in a test matrix, taking specific use cases into account.
According to the ASAM test matrix, one way to test the known and unsafe scenarios is requirements-based testing in a software-in-the-loop simulation. Based on the acceptance criteria, these scenarios set the requirements for the system under which it must operate safely. The test specifications are created on the basis of the requirements. This includes the specification of scenario, test case, and parameters.
In software-in-the-loop simulation, a virtual ECU is integrated into a PC-based simulation platform. Requirements-based testing can thus be used to verify the system's response to a predefined scenario. In addition, requirements-based testing provides a tool to evaluate the effectiveness of the technical measure in the system design.
Validating Unknown Risks
The overall goal of validation is to reduce unknown hazards within the ODD. One possible method is scenario-based testing in a software-in-the-loop simulation environment.
The idea of scenario-based testing, in terms of positive risk balance, is to compare the system under test with the behavior of a human driver. By integrating real-world data into the validation process, abstract scenarios are generated that represent a virtual environment. The abstract scenarios are then used to generate logical and specific scenarios for simulation. Interfaces to the vehicle and map material are also virtualized and integrated into the test setup.
Intelligent test control randomly scans parameter spaces to identify potentially hazardous scenarios. In parallel, data from real drives is used to evaluate the representativeness of the test results in the ODD. Thus, unknown risks can be continuously identified and reduced, which increases the overall security of the system.
Conclusion - What does the ideal V&V process look like?
To define an effective V&V strategy, the overall goals of SOTIF activities must be addressed:
- What is the validation goal?
- How can trigger events and functional insufficiencies be identified and evaluated?
- Is the evaluation of potentially hazardous scenarios sufficient?
- Are the relevant scenarios sufficiently covered?
- What methods does the provision of proof support?
- Are appropriate verification and validation methods selected?
There is no one-fits-all answer to these questions. Instead, the questions have to be answered for the respective SUT by considering the test objective and a combination of selected test environments and test methods at a certain test level.
Risks arising from hazardous behavior must be systematically identified and assessed. As the level of automation increases, identifying and evaluating conditions that lead to potentially hazardous behavior (triggers) becomes increasingly complex, requiring multiple analysis techniques in conjunction with real-world test drives to identify hazardous scenarios. For example, one way to examine the impact of potential triggers on the system is to perform a system weakness analysis. Here, a risk analysis is performed for each identified system element and associated functional insufficiency and trigger to define actions to improve the SOTIF, test its effectiveness, and assess the residual risk with appropriate justification.
In addition, the overall release strategy needs to integrate the SOTIF verification and validation process.
Taking these factors into account, it is possible to provide a SOTIF release argumentation which is ideally implemented in a digital homologation process. This sets the theoretical framework for a process that will enable complex autonomous systems to respond to ever-changing driving environments.
Based on ISO 26262 and ISO/PAS 21448, dSPACE Consulting helps develop test strategies for scenario- and requirements-based designs with professional risk analysis and integrate them seamlessly into established development processes.
Published, February 2024
