KANBrief 3/22
Where AI-enabled systems cannot be assessed by conventional methods owing to their high complexity or capacity to develop autonomously, how can their functional and operational safety be verified? Assurance cases are the tool of choice when new, potentially safety-critical technologies are deployed for which the existing real-world experience is not sufficient.
Despite many years of discussion in the context of standardization and regulation, a consensus has still not been reached on what constitutes an “AI system”. In the European regulatory sphere, there appears to be widespread agreement that an AI system is a certain type of software. However, it seems somewhat unclear how it should be differentiated from conventional software.
In autonomous and semi-autonomous systems, standardized procedures for assessing safety are increasingly reaching their limits. Even the simplest of safety concepts can become very extensive when complex tasks are automated in complex operational environments. A range of measures, such as management of uncertainties in environment recognition1, interact and form multiple layers of protection (“layers of protection architecture”). The operational environments and the tasks to be automated by these autonomous or semi-autonomous systems may be highly complex. This requires their protection layers to be based on software that under the European regulatory proposal is deemed to be an AI system.
For safety concepts of such complexity, a safety argument must be formulated that ensures that the overall concept is truly and sustainably valid. The assurance cases defined in ISO/IEC 15026 (Systems and software assurance) would appear to be a suitable approach for this purpose. These assurance cases are generally considered suitable where sufficient experience has not yet been gained with a particular technology in a safety-critical context2.
An assurance case comprises a claim, which is to be substantiated, regarding the desired level of safety, and an associated argument based on a body of supporting evidence.
As shown in the diagram, the argument can be structured hierarchically by the explicit formulation of discrete reasoning steps. Each reasoning step combines a claim to be demonstrated (e.g. that the product is safe) with premises (e.g. that the electrical hazard is controlled). At the next level, the premises are treated as new claims and are predicated in turn in further reasoning steps on further premises (e.g. that the power cable is not damaged <- insulation is adequate).
Often, the logical predication of a claim on certain premises is valid only on condition that certain assumptions are made, such as that of a particular operational environment (e.g. the user possesses experience, electrical currents are below a certain level, etc.). These assumptions are formulated during development and documented explicitly in the assurance case. Any claim that is not further refined must be supported by evidence such as documentation or the results of verification.
A formulated assurance case offers a number of benefits. It merges, in modular form, all the elements (artefacts) required for the safety argument, and can be integrated into the software of the system as a whole by way of special program modules (digital dependability identities3). It thus enables the fulfilment of key assumptions and claims to be monitored during operation, weaknesses in the assurance case thereby to be detected early, and the assurance case to be continuously improved and adapted to changes in the operational environment4. In particular, however, assurance cases offer a high degree of flexibility in structuring of the argument. This enables specifics of the application under consideration and the technologies used to be addressed.
Practical tools exist with which this flexibility can be exploited productively. The AMLAS method5 for example describes generic procedures for structuring a safety argument. However, AMLAS does not define what constitutes “sufficiently safe” for an AI system.
In the ExamAI project, a proposal has been developed for a form that test methods for AI systems might take. It is based on two independent lines of argument6.The first aims to show that the safety risk has been reduced, as far as is practicable, by selection of the most effective combination of safety measures and their best possible implementation with consideration for the cost-benefit aspect. The second has the purpose of providing quantitative evidence that the attained risk reduction is in fact sufficient.
The current LOPAAS research project7 combines these approaches with others from the research community. The project partners are also submitting the scientific consensus to standardization activities. These include the VDE-AR-E 2842-61 application rule for autonomous/cognitive systems, the ISO and IEC TR 5469 technical report on AI and functional safety, and BSI’s PAS 8800 for safety and artificial intelligence in road vehicles.
First, regulation and standardization should develop consistent definitions for the terms “AI system” and “autonomous system.” This is essential in order for the gaps in regulation and standardization concerning safety and other legally protected interests to be understood and closed. Second, research into assurance cases, and participation of the researchers in standardization, should be promoted and knowledge concerning assurance cases made available to stakeholders. Third, regulatory requirements should be formulated such as to provide a good starting point for the development and application of standards governing assurance cases. Regulatory requirements should focus on the claims that are essential for safety. These are usually located at the top level of an assurance case. Lower-level requirements, on the other hand, may present problems; depending on the argumentation or use case, they do not necessarily constitute part of a valid safety argument. Regulatory requirements governing such details may unnecessarily constrain the scope for implementation or give rise to unnecessary expense.
1 Management of uncertainties in environment recognition
2 Understanding What It Means for Assurance Cases to "Work", (pdf, non eAccessible)
3 Digital Dependability Identities
4 Assurance Cases as Foundation Stone for Auditing AI-enabled and Autonomous Systems
5 The AMLAS method
6 The lines of argument
7 The research project “LOPAAS”
Rasmus Adler, Fraunhofer Institute for Experimental Software Engineering IESE
rasmus.adler@iese.fraunhofer.de
Michael Kläs, Fraunhofer Institute for Experimental Software Engineering IESE
michael.klaes@iese.fraunhofer.de
