Scholarly article on topic 'Research on Elicitation of Safety Testing Requirements for Airborne Software'

Research on Elicitation of Safety Testing Requirements for Airborne Software Academic research paper on "Mechanical engineering"

CC BY-NC-ND
0
0
Share paper
Academic journal
Procedia Engineering
OECD Field of science
Keywords
{"software safety" / "safety testing" / "requirements elicitation" / "safety testing requirements" / "testing requirements"}

Abstract of research paper on Mechanical engineering, author of scientific article — Hongbing Li, Xiaohong Bao, Shujuan Ji

Abstract Software safety testing requirements elicitation method was put forward based on the safety-critical software including safety testing requirements classification, safety testing requirements decomposition and the finishing of elicitation sources in order to get more perfect software safety testing requirements. And it was compared with the safety testing requirements of current test methods. Finally the elicited testing requirements were applied in certain engine control software, and was validated more sufficiently.

Academic research paper on topic "Research on Elicitation of Safety Testing Requirements for Airborne Software"

(I)

CrossMark

Available online at www.sciencedirect.com

ScienceDirect

Procedía Engineering 80 (2014) 303-312

Procedía Engineering

www.elsevier.com/locate/procedia

3rd International Symposium on Aircraft Airworthiness, ISAA 2013

Research on Elicitation of Safety Testing Requirements for

Airborne Software

Li Hongbing, Bao Xiaohong, Ji Shujuan a*

School of Reliability and Systems Engineering, Beihang University, Beijing, 100191, China

Abstract

Software safety testing requirements elicitation method was put forward based on the safety-critical software including safety testing requirements classification, safety testing requirements decomposition and the finishing of elicitation sources in order to get more perfect software safety testing requirements. And it was compared with the safety testing requirements of current test methods. Finally the elicited testing requirements was applied in a certain engine control software, and was validated more sufficiently.

© 2014 Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.Org/licenses/by-nc-nd/3.0/).

Selection and peer-review under responsibility of Airworthiness Technologies Research Center, Beihang University/NLAA. Keywords: software safety; safety testing; requirements elicitation; safety testing requirements; testing requirements

1. Introduction

Software safety testing is a key link to ensure the quality of software. It is different from conventional software testing. It focuses on events that lead system to catastrophic accidents. Its fundamental purpose is to reduce the risk of catastrophic accidents of system. Because of different objects, software safety testing is quite different from routine testing in testing requirements elicitation.

It is one of the hot spot of current research for safety-critical software to carry out safety testing. There have been many research results. Software testing research facing RTCA DO-178B mainly focuses on meeting technical method of MC/DC coverage rate demands[5][6]; In addition, there are software safety testing methods which are based on fault tree analysis[8][9]and Petri net analysis[10][11], etc. The methods above met the following problems in practical application: 1) Because of the software safety testing requirements were not elicited sufficiently, it can't only rely on the cover test to ensure safety; 2)

* Corresponding author. Tel.: +1-336-683-6765; fax: +0-000-000-0000 . E-mail address: baoxh@buaa.edu.cn..

1877-7058 © 2014 Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/3.0/).

Selection and peer-review under responsibility of Airworthiness Technologies Research Center, Beihang University/NLAA. doi:10.1016/j.proeng.2014.09.089

Lots of software testing contained safety testing this testing type, but it only used enumeration method to give the testing requirements, the consideration was not quite comprehensive; 3) Various safety testing methods put forward its own safety test practice, but lacked of systematic consideration towards to adequacy of the whole software safety testing.

Therefore, how to comb software safety testing requirements systematically about software safety testing and achieving more comprehensive coverage of safety test for software safety events and elements is a problem to be solved.

This paper engaged in research on elicitation of safety testing requirements for airborne software. It was on the basis of the existing software safety results. It was to get more sufficient safety testing requirements to improve the adequacy of software safety testing and ensure the safety of products.

Nomenclature

A software safety

Software safety refers to software has the ability of not leading to the accident [7].

B software safety testing

There is no authoritative definition at present. The definition is given referring to [7]: Software safety testing is the test to verify the safety of software.

C software safety requirements

Software safety requirements are decomposed from system safety requirements. It assure that systems maintain in a safe condition and make a full reaction to potential failure at the same time [2].

D software safety testing requirements

Software safety testing requirements mainly solve the problem that what are tested in software safety testing. That is specified what need safety testing about the object under test. Software safety testing requirements usually use safety requirements in software development as the foundation to carry out the analysis. It forms the testable contents through refining, decomposing and supplying the safety requirements.

2. Elicitation of Software Safety Testing Requirements

2.1. Software safety requirements classification

This paper divided software safety requirements into three classes broadly. They are identified safety requirements, unidentified safety requirements and the missing safety requirements. Software safety requirements classification was shown in Figure 1.

Fig. 1. software safety requirements classification

The identified safety requirements are identified by system. It is identified in software requirements or development documents. The unidentified/missing safety requirements are not identified in specification. They need safety analysis further to supply and improve themselves.

Software safety testing requirements cover these software safety requirements.

2.2. Safety testing requirements decomposition

Software safety testing requirements decomposition is further refining of the classified software safety requirements. And they can be decomposed into safety test items, namely refining testing requirements. Safety testing requirements decomposition proposed in this paper was shown in Figure 2.

Fig. 2. safety testing requirements decomposition

The testing requirements can be divided into two categories: general test requirements and specific test requirements. General test requirements cover the missing safety requirements. Specific requirements include two categories: identified safety testing requirements and unidentified safety testing requirements.

General test requirements include general safety constraints, general failure mode constraints, safety coding standard, safety design criterion, design constraints and similar software failure appeared on the history. General safety constraint is the accumulation of historical experience, such as general requirements list gives in NASA - STD - 8719.

There are many types of software failure mode. And among them there are three kinds of failure types are common. They are operation type failure, sequence type failure and value type failure. We can communicate with experts and developers or refer to the test documentation to obtain similar software failure appeared on the history.

Identified safety requirements include specified safety-critical function, specified safety-critical performance and specified safety-critical interface. Unidentified safety requirements include safety-critical scenario, risk cut set constraint determined by safety analysis, operating requirements of system in different environment, system robustness requirements, operator/computer system interactive anomaly constraint, interface data anomaly constraint and code anomaly constraint.

Safety-critical scenario considers the combination of each safety-critical function in order to determine the testing method of combinatorial functions more efficiently. We usually test each function in the conventional function test. And test some combinatorial functions in comprehensive test. But this combinational test completely depends on tester's experience at present. It has no sp ecific guidance method. It is important that how to choose the function to combine in limited test resources.

Robustness requirements range proposed in this paper was greater than abnormal condition test and boundary test proposed in conventional test. It also include input data of failure mode determined by FHA or FMECA, not allowed state or mode conversion and system anomaly constraint, extreme anomaly constraints, etc.

In addition to contain the conventional data boundary anomaly, interface data anomaly also include order, time, dynamic input/output and interrupt, power, mains fluctuation, momentary power failure, system error or data exception generated by hardware failure.

Software safety testing requirements decomposition in this paper not only covered the identified software safety requirements, but also covered the unidentified/missing safety requirements. Therefore, this paper gave the relatively perfect software safety testing requirements from the top. And we could test them based on the level of the testing requirements. We could use unit testing, integration testing and system testing, etc.

2.3. Elicitation sources of software safety testing

Based on the software safety testing requirements decomposition above, the elicitation sources were shown in Figure 3.

The identified safety testing requirements are mainly from software requirements specification, software development program, specification, software interface document, software design documents, and safety analysis results. Unidentified/missing safety testing requirements are mainly from the standards, experience and other requirements.

The standard demands include safety design standard, test standards, as well as aerospace software standards, such as GJB/Z 102, NASA - STD - 8719, and software system safety manuals, etc.

Fig. 3. elicitation sources of software safety requirements

2.4. Elicitation steps of software safety testing requirements

The main steps of software safety testing requirements elicitation are as follows.

• Elicit system and software information;

• Tailor general software safety testing requirements;

• Elicit specific software safety testing requirements;

• Decompose test items.

First of all to elicit system documents and software background knowledge. Documents include software requirements specification, interface documents, software user documents, design documents, etc. In addition, we can refer to the relevant standards and safety standards, and communicate with user, developers, domain experts to know more detailed information.

We can tailor general software safety testing requirements according to the system characteristics, and get safety testing requirements applicable to the software, Then we decompose test items and identify test items to get the corresponding general software safety testing requirements.

Elicitation of specific software safety testing requirements was analyzed according to the characteristics of the target software. The elicitation of safety-critical scenario testing requirements in software system safety-critical requirements should consider safety-critical function combination that can lead to dangerous state. Risk cut set constraint determined by safety analysis, the use of systems in different environment and the elicitation of robust demand should be carried out gradually. The elicitation of safety-critical software interface requirements should identify all the safety-critical data variables and time selection, and also identify interface failure or the potential failure mode resulted by data error to get specific software safety requirements.

3. Software Safety Testing Requirements Analysis and Comparison

Software safety test based on testing requirements proposed in this paper was compared with safety test used in engineering at present. We found that most standards currently and safety testing demand in

practice are all general. Their guidance is not strong. Safety testing inspects safety and security existing in software be effective or not. Specific testing requirements are as follows.

• Test the processing power and protective capability of software in a standard configuration;

• Test the processing power and protective capability of software under abnormal conditions;

• Must contain testing of boundary, out of bounds and boundary attaches;

• Test the safety-critical operation mistakes.

These testing requirements above are more confined to the concepts. They are very limited for how to develop actual testing, what the testing object is and how to design test cases.

Compared with common testing requirements related safety, this paper proposed more comprehensive safety testing requirements from the top. Common testing requirements include specified safety-critical function, specified safety-critical performance and specified safety-critical interface. In addition to include test requirements of routine testing types, we need increase test requirements categories which should consider the test item. They are robustness requirements, interface data anomaly constraints, system operator/computer system interactive anomaly constraints and code anomaly constraints. Testing requirements categories completely new are general safety constraints, general failure mode constraints similar software failure appeared on the history, safety coding standard, safety design criterion, design constraints, safety-critical scenario, risk cut set constraints determined by safety analysis, operating requirements of system in different environment. The detailed comparison was shown in table 1.

Table 1. Contrast of software safety requirements and commonly used safety testing demand

Software safety testing requirements proposed in this paper The requirements of the current test system

Requirement Type Software safety requirements category existin g Need to increas e newly increase d

General safety testing requirements general safety constraints, general failure mode constraints similar software failure appeared on the history, safety coding standard, safety design criterion, design constraints safety coding standard f

Software system safety-critical requirements specified safety-critical function, specified safety-critical performance

robustness requirements s-

safety-critical scenario, risk cut set constraint determined by safety analysis, operating requirements of system in different environment.

Software interface safety-critical requirements specified safety-critical interface

interface data anomaly constraint, system operator/computer system interactive anomaly constraint and code anomaly constraint

4. The Elicitation of Engine Control Software Safety Testing Requirements

4.1. Introduction of software under testing

Numerical control system of engine is an important part of the engine fuel and control system. It consists of electronic controller, numerical control system software, sensor, hydraulic machinery and corresponding electrical system, etc. The control software is the core of numerical control system. The

main functions include I/O, fault diagnosis and treatment, control task processing, BIT, data storage, data communication, etc.

This paper took safety requirements of system test as an example. It elicited some general and specific safety test requirements of one engine control software.

4.2. General safety test requirements of software under test

General safety test requirements include general safety requirements, general failure mode constraints similar software failure appeared on the history. Only two general safety requirements were described in this paper. It based on software safety general test requirements given in NASA [3]. It was described in table 2.

Table 2. General safety test requirements

Software general safety requirements Software general safety testing requirements

Requirement identification Requirement description Test item identification Test item description

RS-TY01 The software must have the ability to put system in safe state, when the hardware fails, system fails conducted by software or configuration and the current running status is inconsistency RS-TY01-01 Whether the software can put system in safe state, when there are hardware failure that can lead to system failure

RS-TY01-02 Whether the software can put system in safe state, when there are software failure that can lead to system failure

RS-TY01-03 Whether the software can put system in safe state, when there are combinative failure of software and hardware that can lead to system failure

RS-TY01-04 Whether the software can put system in safe state, when software configuration and the current running status is inconsistency

RS-TY02 The software must be designed to shut down system orderly when the power fails, so it won't produce potential unsafe state RS-TY02-01 Whether the system can shut down orderly and won't produce potential unsafe state, when power failure(low voltage)

RS-TY02-02 Whether the system can shut down orderly and won't produce potential unsafe state, when power failure(high voltage)

RS-TY02-03 Whether the system can shut down orderly and won't produce potential unsafe state, when power failure(voltage instability)

RS-TY02-04 Whether the system can shut down orderly and won't produce potential unsafe state, when power failure(Voltage intermittence)

RS-TY02-05 Whether the system can shut down orderly and won't produce potential unsafe state, when power off suddenly

4.3. Specific safety testing requirements of software

1) Testing requirements based on safety-critical scenario

It will lead to serious safety consequences, if engine alarm function failure of over temperature and pressure. So important testing was carried out . Now scenario testing requirements were structured. The signal was in alarm state, only when the low pressure turbine exhaust temperature was in combat (training state) and reach the control plan plus 20 °C. The signal was in normal state, only when the low pressure turbine exhaust temperature was in combat (training state) and be equal to the control plan or starting below or equal to 900 °C. A brief description was shown in table 3.

Table 3. Scenario testing requirements of over temperature

Software safety req uirements Software safety testing requirements

Requirement identification Requirement description Test item identification Test item description

RS-GJ Test the scenario of engine over temperature alarm function RS-GJ-01 Engine operates with over pressure below alarm temperature

RS-GJ-02 Engine operates with over pressure above alarm temperature

RS-GJ-03 Engine operates with over pressure at alarm temperature

RS-GJ-04 Engine operates with over pressure and warmings up to over the alarm temperature

RS-GJ-05 Engine operates with over pressure and cooling down to below alarm temperature

RS-GJ-06 Engine operates with over pressure and warmings up below the alarm temperature, and be forced self-checking

RS-GJ-07 Engine operates with over pressure and warmings up over the alarm temperature, and be forced self-checking

RS-GJ-08 Engine operates with over pressure and cooling down below alarm temperature, and be forced self-checking

RS-GJ-09 Engine operates with over pressure and cooling down over the alarm temperature, and be forced self-checking

2) System robustness test requirements A brief description towards to robustness was shown in table 4.

Table 4. System robustness test requirements

Software safety requirements Software safety testing requirements

Requirement identification Requirement description Test item identification Test item description

RS-LB Test the robustness of engine starting function RS-LB-01 The time that the engine drop to training state is longer than the set time

RS-LB-02 Starting engine on the ground and be over temperature

RS-LB-03 Starting engine on the ground and be over pressure

RS-LB-04 Starting engine on the ground and be over-running

RS-LB-05 Air starting engine and be over temperature

RS-LB-06 Air starting engine and be over pressure

RS-LB-07 Air starting engine and be over-running

3) Interface data abnormal test requirement

Here took frequency amount as an example to test the abnormal interface data. A brief description was shown in table 5.

Table 5. Interface data anomaly constraint

Software safety requirements Software safety testing requirements

Requirement identification Requirement description Test item identification Test item description

RS-JK Test the abnormal interface data of frequency RS-JK-01 Whether the elicitation and conversion of frequency amount is correct, when system power off instantaneously

RS-JK-02 Whether the elicitation and conversion of frequency amount is correct, when system voltage fluctuation

RS-JK-03 Whether the elicitation and conversion of frequency amount is correct, when system fail temporarily

RS-JK-04 Whether the elicitation and conversion of frequency amount is correct, when the elicitation time interval is greater than the regulation interval

RS-JK-05 Whether the elicitation and conversion of frequency amount is correct, when the elicitation time interval is smaller than the regulation interval

4.4. The comparison with safety testing requirements commonly used

This paper applied elicited testing requirements to a engine control software. We elicited general and specific safety testing requirements under the guiding ideology of this paper. The requirements got here greatly supplied safety testing requirements commonly used.

The general safety testing requirements category was not mentioned at present. It was newly increased. And it added all the test item in RS-TY01. For specific safety testing requirements, safety-critical scenario testing requirements were also newly added. Scenario testing requirements of over temperature combined safety-critical functions. It was not mentioned in commonly used test. And this paper added and perfected the robustness and abnormal interface data test requirements. RS-LB-01, RS-LB-05, RS-LB-06 and RS-LB-07 were newly added for robustness test requirements. Abnormal interface data test requirements of frequency amount added all the test items comparing with laboratorial demands.

We can see that the test items of testing requirements elicited in this paper were 30, there were 22 newly added. The percentage was 73.3%. The elicited requirements covered software safety testing

requirements more comprehensive. So it improved the quality of software safety test. The comparison was shown in table 6.

Table 6. Comparison of commonly used safety testing requirements

Requirement type Newly added Original quantity

General safety testing requirements 4 5

Safety-critical scenario testing requirements 9 0

Robustness testing requirements 4 3

Abnormal interface data testing 5 0

requirements

The total test item 22 8

5. Conclusion

This paper studied the elicitation of safety-critical software testing requirements. At last we got more perfect requirements by classifying, decomposing testing requirements and collecting the sources of testing requirements. And the elicited requirements were applied to an engine control software. It stated that testing requirements elicited in this paper are more sufficient. The decomposition category will be perfected in subsequent work. And more perfect testing method of testing requirements will be proposed.

References

[1] MIL - STD- 882D, Standard Practice for System Safety Program Requirements[S]. Department of Defense, USA Military, Jan 1996.

[2] NASA-GB-8719.13 NASA. Software Safety Guidebook[S], National Aeronautics and Space Administration, 2004.

[3] RTCA/ DO-178B, Software Considerations in Airborne Systems and Equipment Certification[S]. Requirements and Technical Concepts for Aviation (RTCA), Dec, 1992.

[4] Lingfeng Wang, Kay Chen Tan. Software Testing for Safety-Critical Applications[J]. IEEE Instrumentation & Measurement Magazine, 2005,8(2):38~47.

[5] Heimdahl M P E, Whalen M W, Rajan A etal. On MC/DC and implementation Structure: An empirical Study[A]. Digital Avionics Systems Conference,2008, DASC 2008 IEEE/AIAA 27th[C].2008:5.B.3-1-5.B.3-13.

[6] Hayhurse K J, Veerhusen D S. A practical approach to modified condition/decision coverage[A]. In: Digtal Avioncs System 2001 DASC the 20th Conference[C].2001:1B2/1~1B2/10.

[7] GJB/Z 142-2004 ,Guidence of military software safety analysis[S] .2004.

[8] Zhongwei Xu, Fangmei Wu. Modeling of formal fault tree analysis and software safety testing[J]. Journal of Tongji University, 2001,29(11):1299~1302.

[9] Zhi Hu, Renkun Yin. Generation dynamically of safety test cases based on the minimum cut set[J]. Computer Engineering and Design, 2006, 27(16):3018~3020.

[10] Zhen Wei, Xia Zhou, Hongjie Bao. Research on interlocking software safety testing based on petri net[J]. Computer Engineering and Application, 2005,41(17) : 123~125.

[11] Yinsheng Shi, Shiwei Deng, Tianyang Gu. Software safety testing method[J]. Microcomputer Information, 2008, 24(1-3):56~58.

[12] Changyong Yang. Research on analysis of safety testing requirements for airborne software[D]. Beijing: Beijing University of Aeronautics and Astronautics, 2012.