JOURNAL OF

AERONAUTICS

Chinese Journal of Aeronautics, (2014),27(1): 85-92

Chinese Society of Aeronautics and Astronautics & Beihang University

Chinese Journal of Aeronautics

cja@buaa.edu.cn www.sciencedirect.com

Testability integrated evaluation method based on testability virtual test data

Liu Guanjun *, Zhao Chenxu, Qiu Jing, Zhang Yong

Science and Technology on Integrated Logistics Support Laboratory, National University of Defense Technology, Changsha 410073, China

Received 16 October 2012; revised 4 May 2013; accepted 10 July 2013 Available online 18 December 2013

KEYWORDS

Data fusion; Fault detection; Integrated evaluation; Testability verification; Virtual test

Abstract Testability virtual test is a new test method for testability verification, which has the advantages such as low cost, few restrictions and large sample of test data. It can be used to make up the deficiency of testability physical test. In order to take the advantage of testability virtual test data effectively and to improve the accuracy of testability evaluation, a testability integrated evaluation method is proposed in this paper based on testability virtual test data. Considering the characteristic of testability virtual test data, the credibility analysis method for testability virtual test data is studied firstly. Then the integrated calculation method is proposed fusing the testability virtual and physical test data. Finally, certain helicopter heading and attitude system is presented to demonstrate the proposed method. The results show that the testability integrated evaluation method is feasible and effective.

© 2014 Production and hosting by Elsevier Ltd. on behalf of CSAA & BUAA.

1. Introduction

Testability, which is a key design feature of equipment, is regarded seriously in the aeronautics field. Testability verification is an important instrumentality to test and evaluate whether the testability level of the unit under test (UUT) is achieved according to the requirements stated in the compact. Generally, testability verification usually focuses on the physical test method that is based on the faults injection. With test data, the testability of the UUT can be evaluated with the

* Corresponding author. Tel.: +86 731 84573397. E-mail address: gjliu342@qq.com (G. Liu). Peer review under responsibility of Editorial Committee of CJA.

classical statistical theory.1,2 However, due to the physical location restrictions of some faults and the destructive of fault injection, the testability physical test data (TPTD) is usually poor, so the accuracy of evaluation results is affected.2-4

In order to adapt to the new requirements of equipments for the evaluation accuracy, period, funding and test risk, more and more attention has been paid to the testability virtual test, which is a new verification technology.5,6 The testability virtual test data (TVTD) is acquired by the virtual injection and virtual measurement of faults. With the advantages of virtual test, such as low cost, high efficiency, low risk, process controllability, fewer restrictions to fault injection and so on, a wealth of test data can be acquired in testability virtual test. Theoretically, the testability can be evaluated by using the TVTD directly, if the TVTD is credible completely. So the problem that the TPTD is lacking can be solved well and the accuracy of the evaluation conclusion can be improved. Nevertheless, because the testability virtual test is still

1000-9361 © 2014 Production and hosting by Elsevier Ltd. on behalf of CSAA & BUAA. http://dx.doi.Org/10.1016/j.cja.2013.12.012

in the beginning, and the technology is still immature, meanwhile, owing to the limitations of modeling and simulation technology, the credibility of the TVTD is not high.7 In this situation, if the testability index is calculated only based on the TVTD, the credibility of the evaluation conclusion must be low. Therefore, it is significant to find a way to evaluate the equipment' testability index with TVTD.

The credibility of the TPTD is high while the sample size is usually small. Meanwhile, the sample size of the TVTD is large while the credibility is low. Therefore, it is obvious that the TPTD and TVTD are of the complementary relationship on the sample size and credibility. In order to take advantage of testability virtual test data and improve the accuracy of testability evaluation, the TVTD and TPTD are integrated to compute the testability index in this paper. Up to now, though the research of integrated calculation based on information fusion is plenty, the research of testability integrated evaluation method fusing the TVTD and TPTD is few.

To achieve this idea, an integrated evaluation method that is based on the TVTD is proposed in this paper firstly. Then the key technologies are studied: how to analyze the credibility of virtual test data qualitatively and quantitatively, and how to fuse the TVTD and TPTD to evaluate the testability considering the credibility of the virtual test data. At last, certain helicopter heading and attitude system is presented to demonstrate the proposed method.

2. Overall structure of the integrated evaluation method

Considering the characters of TPTD and TVTD, the TPTD and TVTD are integrated to evaluate the testability index in this paper. The overall structure of the integrated evaluation method is shown in Fig. 1.

Firstly, the TPTD is taken as the standard to analyze the credibility of the TVTD, including dynamic consistency check, credibility calculation and static consistency check. Secondly, the TVTD is taken as the prior information to ascertain the prior distribution of testability index. Then the posterior distribution calculation of testability index is obtained. At last, the TPTD and TVTD are integrated together to calculate the testability index.

Credibility analysis of TVTD

Dynamic consistency check

Credibility calculation

Static consistency check

Prior distribution calculation of testability index

3. Credibility analysis of the TVTD

In engineering practice, there is always a credibility problem that whether the simulation model could represent the real system, and whether the simulation results could represent the true performance of the system. The low simulation credibility will lead to a corresponding risk increase in decision-making. To reduce the risk of analysis and results that introduced by the errors of system simulation model, the verification, validation and accreditation (VV&A) of the simulation model that is used to make sure the simulation quality and credibility received sufficient attention in recent years.8-10 In testability virtual test, the test is taken on the simulation models of the UUT and the TVTD is acquired after the faults virtual injection and virtual measurement. Currently, because of the limitations of technology, the credibility of TVTD is affected by two factors. One is the differences between the testability virtual prototype and the physical prototype, and the other one is the differences between the actual test process and the virtual test. Based on the upper analysis and the general VV&A theory, the TVTD credibility analysis process is shown in Fig. 2.

As shown in Fig. 2, the testability test data can be classified into the two categories: the dynamic test data and the static test data. Correspondingly, the consistency check can be classified into two categories: the dynamic consistency check and the static consistency check. The dynamic test data is the output sequences of the key monitoring signals when the prototype is running with fault, while the static test data is the success-fail result of the fault detection or isolation. The dynamic consistency check is to guarantee the description accuracy of the virtual prototype of the UUT's physical structure and the failure mechanism, and that the dynamic test data is the check objective. The static consistency check is to ensure the consistency between the physical and virtual test data and among the

Posterior distribution calculation of testability index

Integrated calculation of testability index

Fig. 1 Overall structure of the integrated evaluation method.

Fig. 2 Process of credibility analysis of the TVTD.

multiple batches of virtual test data, and that the static test data is the check objective. The dynamic consistency check is usually implemented with the virtual prototype modeling. If the output sequence errors cannot pass the dynamic consistency check, the virtual prototype modification is required. This paper is aimed at the testability index integrated evaluation that is implemented after the virtual test has been done, so the static consistency check is the focus of this paper and the virtual prototype is considered to be credible. The dynamic consistency check will not be discussed here. Because of the differences among the fault sample sets, the credibility of each batch static test data may be different, even if the virtual tests are all taken on the same one virtual prototype. With this premise, it is needed to evaluate the credibility of static test data before the static consistency check is implemented.

3.1. Credibility calculation of TVTD

Credibility is the quantitative characteristic of the simulation information consistency. The definition and computation formula of credibility were proposed in some researches.7 However, the definition and computation formula are so subjective that they are not applicable well to a particular matter. For the testability virtual test, the output sequences of fault simulation can reflect the accuracy of the virtual test and virtual prototype, so the errors between the physical and virtual test dynamic test data are introduced to measure the credibility of the simulation information quantitatively in this paper.

Supposing the number of injected fault samples in once virtual test is n, in order to detect and isolate all the faults correctly, p kinds of equipment output signals are monitored by the built-in test equipment (BITE), where the ath signal is related to q fault samples, a = 1, 2, ■■■, p. To address the accuracy of the testability virtual prototype, the error of ath signal between physical and virtual prototype must satisfy the requirement ea 2 [e^, e^]. Then the mapping function ca = f(ea) between credibility of the ath signal ca e [0,1] and error ea can be defined, and the following conditions must be satisfied:

(1) When the error is 0, the credibility equals 1.

(2) When the error is beyond the accredited interval, the credibility equals 0.

(3) In the accredited interval, the more the error is, the lower the credibility is.

(4) The function is linear.

The function ca = f(ea) can be described as:

0 < e < ea , 0 ^ ea emax

ea < e < 0

emin < a < 0

Otherwise

Then the relation between the credibility of every signal ca and the integrated credibility of the virtual test c is described as:

c =y)®aca

E*a/££.

where xa 2 [0,1] is the weight coefficient of the ath signal, and Eq= 1s0a the reciprocal value sum of the risk priorities of all the failure modes that is related to the ath signal. The risk priority that is one integrated value of the failure rate and the criticality of certain failure modes can be obtained from the failure modes effects and criticality analysis (FMECA).

3.2. Static consistency check to the TVTD

Because of the limitations of test cost, risk and others, testability physical test tends to do one time, so the sample volume of physical test static data is 1. For the demand on the sample amount, the traditional static consistency check methods11-13 (such as Smirnov test, Wilcoxon rank sum test, reverse test, etc.) cannot be applied in the testability test data static consistency check directly, and they need to be improved. Considering the characteristic of the testability test data and the requirements of the integrated evaluation method to the test data, the static consistency check method for testability virtual test data is put forward as follows.

In the traditional testability evaluation, after the binary test data (n, s) is obtained, Eq. (3) can be used to evaluate the point value R and interval value (RL, RH) of the testability index.1

R = s/n

(1 - Rl)' =

(1 - Rh)' =

1 - C 2

where C is the credibility of the evaluation interval, s the total number of the faults detected/isolated successfully in the test, and F = n — s the total number of the fault detected/isolated which failed in the test.

It is assumed that m batches of testability binary test data are obtained, including both the virtual test and physical test. The index evaluation interval of eth test, under the given confidence level C, is calculated by Eq. (3) and denoted as (RL, RH), where e = 1,2, ■■■, m. Then, the same degree of any two intervals (RlL, RlH) and (RL, RH) can be denoted as CR(l, j) and calculated by:

CR l , l(RL; ^^(j RjH )l P 0 (; j) |(RL, rh)u(rl, rjH )l"

1 a=1 o=1

where |-| denotes the measurement.

If CR(l, j) = 0, we can say that (RL, RH) and (RL, RH) are not compatible, and the unreasonable virtual test static data must be excluded.

4. Integrated calculation of testability index

The accuracy of the calculation result only based on the virtual test data may be low because of the credibility of the virtual test data. Thus, when TVTD and TPTD are compatible, an integrated evaluation method based on the Bayes theory14'15 can be used to evaluate the testability index. Firstly, the TVTD can be taken as the prior information to determine the prior distribution of testability index using classical statistical method. Then the physical test data can be fused with virtual test data to calculate the posterior distribution of testability index. Finally, the testability index can be computed by using the

Bayes formula. Conveniently, the fault detection rate (FDR) is taken as an example to explain the Bayes based comprehensive computation method.

4.1. Prior distribution calculation based on TVTD

There are two parts in FDR prior distribution calculation: FDR distribution modality estimation and FDR distribution space calculation. Generally, the binary data of testability test satisfies the binomial distribution, that is to say that the prior distribution of FDR can be fixed to the binomial distribution. Then the key work in this step is to calculate the distribution space of FDR. The FDR prior distribution space calculation method is proposed as follows, based on the TVTD and the D-S evidence fusion theory.

Thanks to the advantages of the testability virtual test, the test time of virtual test is not limited. Every test can be a whole test and one evaluation interval can be obtained. However, because of the test initial conditions, simulation randomicity, data credibility and other factors, each evaluation interval may be not identical to all the others. Before the prior distribution space calculation, the diverse virtual test evaluation intervals must be integrated as a collectivity.

It is assumed that there are k virtual test evaluation intervals which have passed the static consistency check. Calculating the same degree of any two intervals in the k evaluation intervals, the same degree matrix CR with the dimension k x k can be obtained:

CR12 1

CRki CR,

CRik CR2k

Normalizing each element of CR, and the new matrix CR' can be obtained with Eq. (6):

CRig - CRig/X]CR'g

(' - 1; 2, ■■■, k)

The union of evaluation intervals can be taken as a domain, and the k evaluation intervals can be abbreviated as A1, A2, ■■ ■, Ak respectively. Then the basic probability distribution function of each evaluation interval can be calculated and denoted as:

m((Ag) - CR0

Synthesizing the basic probability distribution functions by

it can be calculated as shown in:

evidence fusion theory16'1

g(Ag) „ k „k . . E^n i=igi(A,)

Denoting the allocation coefficient as mg = g(Ag), g = I, 2, ■■■' k, the integrated interval, which is just the FDR prior distribution space and denoted as (RVL, RVH), can be calculated as:

Rvl Xg[R"H -(RH - Rl)/Cg] g-1 k

Rvh -Y,XgRH

where (R1, RH) is the FDR evaluation interval of the gth virtual test, and c the credibility of th batch of TVTD which can be acquired by Eq. (2).

If the FDR point evaluation value of the gth virtual test is calculated by Eq. (3) and denoted as Rg, the integrated FDR point value of the virtual test can be denoted as Rv and calculated by:

RV -J2xgcgRg-

4.2. Posterior distribution calculation

It is assumed that p(X\ h) denotes the conditional destiny function of sample X with the parameter h, while p(h\X) denotes the conditional destiny function of parameter h with the sample X. Then the posterior distribution calculation formula of Bayes can be expressed as7: p(h)p(X|h)

p(6\X) - -

f0n(O)p(X \ff)d0

where H = {h} is the parameter prior distribution space.

If the binary data of testability physical test is (n, s), and the FDR evaluation interval of testability virtual test is (RV1, Rvh), the Eq. (10) can be transformed to be Eq. (11), which can be used to calculate the posterior distribution.

Where, n denotes the total number of the fault samples injected, s the total number of the fault samples detected successfully, and R the point value of FDR.

4.3. Testability index integrated calculation

After the posterior distribution of FDR is calculated, we can calculate the point value and interval value of FDR with the Bayes theory, where the distribution space of FDR equals (Rvl, Rvh) that is obtained in the Section 4.1.

Denoting the point value of integrated evaluated FDR as R, and the evaluation interval of integrated evaluated FDR as (Rl, Rh), then the testability index can be calculated by Eq. (12)13, Eq. (13)13 and Eq. (14) respectively.

n(R)P(s\R)

Rs(1 - R)n

n(R\s)=„ — „

( \)flVL'p(R)P(s\R)dR fRHRV - R)n-sdR

Rs(1 - R)

' J'RRVH Rs(1 - R)n-sdR - J0Rvl Rs (1 - R)n-sdR

B(s + 1, n - s + 1y B(s + 1, n - s + 1) Rs(1 - R)n-s ¡RVHRs(1 - R)n-sdR -J'RRVLRs(1 - R)n-sdR _ Rs(1 - R)n-s

~B(s + 1, n - s + 1)[Irvh (s + 1, n - s + 1)-IrvL (s + 1, n - s + 1)]

rR - Rn(R\s)dR

¡RR™ Rs+1(1 - R)n-sdR

B(s +1, n - s + 1)[Irvh (s +1, n - s + 1)-Irvl (s +1, n - s + 1)] B(s + 2, n - s + 1)[Irvh (s + 2, n - s + 1)-Irvh (s + 2, n - s +1)] " B(s +1, n - s + 1)[Irvl (s +1, n - s + 1)-Irvl (s +1, n - s + 1)] s +1 Irvh (s + 2, n - s + 1)-Irvl (s + 2, n - s +1)

1 + 2 Irvh (s + 1, n - s + 1)- Irvl (s + 1, n - s + 1)

y = p(R|s)dR

fRrHRs (1 - R)n-sdR

B(s + 1, n - s + 1)[lRrH (s + 1, n - s + 1)- Irvl (s +1, n - s +1)]

B(s + 1, n - s + 1)[Irvh (s + 1, n - s + 1)-I- (s +1, n - s + 1)] ._RL_

B(s + 1, n - s + 1)[Irvh (s + 1, n - s + 1)-Irvl (s +1, n - s + 1)] Irvh (s + 1, n - s + 1)-Ib (s + 1, n - s + 1)

IRvh (s + 1, n - s + 1)- Irvl (s + 1, n - s + 1)

R- H RV

where B() and /.(•) are the Bate function and the incomplete Bate function respectively, and y is the confidence to the integrated evaluation.

5. Case study

Heading and attitude system is a key component of the helicopter. Its structure is shown in Fig. 3 and includes the 28 V DC power (C1), 26 V, 400 Hz AC power (C2), quickly righting mechanism (C3), static converter (C4), corrective mechanism (C5), gyro motor (C6) and sync generator (C7). Its working environment is complex, failure rate is high, and the fault detection and isolation time is long. All of these will affect the readiness of the helicopter. Thus, BITE is designed to monitor the system's real-time status and to detect and isolate the faults timely. In the BITE, 10 signals are monitored which are related to total 16 failure modes. The failure modes and the related signals are listed in Table 1. The BITE has played an important role in reducing the occurrence times of accidents. However, since the cost of hardware fault-injection test is high, and most of fault samples cannot be injected effectively, there is no enough useful data. It is difficult to assess the system's testability index by testability physical test.

In order to get the FDR of the heading and attitude system as actual as possible, and test the feasibility and effectiveness

Fig. 3 Structure of heading and attitude system.

of the proposed integrated evaluation method, three other testability index evaluation methods such as physical verification test, virtual verification test and testability prediction are carried out at the same time, besides the integrated evaluation method. All the work is carried out according to the requirements of 301 items in GJB2547-95 ''Outline of equipment test''18. The fault samples select method used in the physical test is provided by appendix C in GJB2072-94 ''Maintainability test and evaluation''19, while the method used in the virtual test is referenced to Zhang et al.20 The fault samples used in each virtual test are listed in Table 2.

For the fault samples, which are difficult to be injected, some reasonable methods in the laboratory condition have been utilized to break the physical limitations when the physical test is carried. Therefore, all the required fault samples have been injected and tested, and the result of physical test is authentic relatively. The virtual prototype of the UUT is modeled in the EDA software Multisim10, and the virtual test is carried out under five different conditions. The virtual prototype is shown in Fig. 4. The testability virtual prototype of the heading and attitude system has passed the dynamic consistency check, so the TVTD of this virtual prototype can be used in the testability virtual test. The credibility of the monitored signals in the virtual prototype is listed in Table 3. The

Table 1 List of failure modes.

Mode name Component Code Risk priority Mode name Component Code Risk priority

Wearout of cam C3 F1 2 Output irregular C2 F9 2

Inactive of switch C3 F2 2 Irregular voltage of output C1 F10 4

Inter-phase short circuit C6 F3 5 Underpower C1 F11 2

Inter-phase open circuit C6 F4 5 Inactive of micro switch C5 F12 3

Dynamic unbalance of motor C6 F5 8 Inactive of motor C5 F13 7

Failure in switching power C4 F6 3 Short circuit of coil C7 F14 4

Failure in three-phase power C4 F7 2 Open circuit of coil C7 F15 6

Failure in boost circuit C4 F8 2 Irregular drift C7 F16 8

Table 2 List of fault samples used in each virtual test.

Test style and order Injected times of each failure model in each virtual test

F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 F13 F14 F15 F16 Total

Virtual test 1 6 8 2 3 8 2 1 2 2 2 2 2 2 2 4 6 54

Virtual test 2 5 8 3 3 7 1 2 2 1 1 1 2 4 4 5 7 56

Virtual test 3 3 11 2 1 8 2 1 0 1 2 1 2 0 3 7 8 52

Virtual test 4 7 10 1 5 7 1 0 2 2 0 1 2 1 0 4 7 50

Virtual test 5 4 8 2 3 5 1 2 1 2 1 2 0 2 4 7 9 53

Fig. 4 Virtual prototype of heading and attitude system.

virtual and physical verification test static data are shown in Table 4, in which the upper five rows are the virtual test binary data and the last row is the physical test binary data. The software TEAMS21 is used in the testability prediction. The prediction value of FDR is 100%, and result is shown in Fig. 5.

Next, the testability index will be calculated by using the proposed integrated evaluation method, and integrated evaluation method will be compared with other verification and prediction methods.

5.1. FDR integrated evaluation

5.1.1. Credibility calculation

The credibility of each virtual test are calculated with Eq. (2) and listed in Table 5. The credibility of each virtual test is different, which results from different fault samples used in each virtual test and simulation randomicity.

5.1.2. Static consistency check

Taken the confidence to interval estimation method as C = 90%, and using Eq. (3), the point and interval value of each test are calculated and the results are listed in Table 6. It is noted that the result is calculated when the credibility of the test data is not considered.

According to the static consistency check method proposed in Section 3.2, the same degree matrix of TVTD and TPTD is listed as follows:

1 0.5192 0.5806 0.8521 0.9704 0.8521

0.5192 1 0.9123 0.6276 0.5031 0.6276

0.5806 0.9123 1 0.6944 0.5633 0.6944

0.8521 0.6276 0.6944 1 0.8276 17

0.9704 0.5031 0.5633 0.8276 1 0.8276

0.8521 0.6276 0.6944 1 0.8276 1

Table 4 Results of testability virtual and physical tests.

Test style and order Number of fault Number of fault

samples injected samples detected

Virtual test 1 54 49

Virtual test 2 56 53

Virtual test 3 52 49

Virtual test 4 50 46

Virtual test 5 53 48

Physical test 50 46

Analyzing the value of same degree matrix, we can find that the consistency between each two intervals is well, so they can be integrated and used to calculate the testability index.

5.1.3. FDR evaluation based on TVTD

The allocation coefficients of each virtual test evaluation interval are computed with Eq. (7) and are shown as follows: = 0.2185 «2 = 0.1311

«3 = 0.1816 (16) «4 = 0.2694 «5 = 0.1995

Now, the FDR based on all the compatible TVTD can be calculated by Eqs. (8) and (9). The lower limit value and upper limit value of the FDR evaluation interval are Rvl = 0.804 and RVH = 0.964, the credibility of the evaluation is C = 0.90, and the point value of FDR is RV = 0.723.

5.1.4. Testability index integrated calculation

Taken the confidence to the physical test as 0.90, the integrated interval of virtual test as the prior distribution space, and the physical test data as the additional information, the integrated evaluation value of FDR can be calculated by using Eqs. (12)-(14). The point estimate value of FDR is R = 0.904, and the upper limit value and lower limit value of FDR are Rl = 0.851, RH = 0.964.

5.2. Comparison and analysis

The evaluation results based on the virtual test, physical test, integrated evaluation and testability prediction are compared and listed in Table 7.

The evaluation results based on physical test is obtained by sufficient and reasonable fault samples, so the FDR of heading and attitude system obtained by physical test data is the most actual and can be taken as the true value. By comparing the four cases, the following conclusions can be made:

Table 3 List of monitored signals and the corresponding credibility in virtual prototype.

Signal style Code Credibility Related failure mode Signal style Code Credibility Related failure mode

Voltage of C1 T1 0.95 F10, F11 Voltage of C7 (pitching) T6 0.72 F1-F16

Voltage of C2 T2 0.92 F9 Voltage of C7 (roll) T7 0.70 F1-F16

Frequence of C2 T3 0.98 F9 Current of C6 T8 0.78 F3-F8, F10, F11

Voltage of C3 T4 0.81 F6-F8, F10, F11 Magnitude of temperature T9 0.85 F6-F8, F9-F11

Frequence of C4 T5 0.90 F6-F8, F10, F11 Magnitude of vibration T10 0.83 F2, F12, F13, F16

Testability report for heading_and_attitude_system

Test options

Test algorithm near optimal (breadth=l, depth=l)

Test cost weightage=50.00%

Test time weightage=50.00%

Test dollars per hour=l 0.00

Fault isolated to label failure_mode

System ok probability: 1%

Mean time to first failure: 5055 (hours)

System statistics

Number of failure sources=16 in 16 failuremodes

Number of tests=10

Number of dependencies=62

Number of modules at Level 1=7

Number of modules at Level 2=15

Test algorithm statistics

Number of tests not used=3 Number of nodes in tree=17 Efficiency of test sequence=84.66%

Testabil ity figures of merit_

Percentage fault detection=100.00% (UW: 100.00%)

Percentage fault isolation=62.60% (UW: 63.04%)

Percentage retest OK's=19.80%

Ambiguity group size=l .44

Mean weighted cost to tsolate=0.

Dollar cost=0

Time-0

Mean cost to detect=0 Mean time to detect=0

Histogram of ambiguity size

100 100

aj OÛ tz 80 60 63% o"-ID Sf S 80 60

C= 'j U 40 H 31 % c O 40

a- 20 7% Î5 0- 20

0 ■ H , , , , 0

1 2 3 4 5 6 7 9

Histogram of test usage

' 29% I

Ambiguity group size Fig. 5 Testability prediction report.

123456789 Number of tests

Table 6 Evaluation intervals of FDR.

Test style and order Point value Confidence Interval value

Virtual test 1 0.907 0.90 (0.822, 0.954)

Virtual test 2 0.946 0.90 (0.873, 0.978)

Virtual test 3 0.942 0.90 (0.864, 0.977)

Virtual test 4 0.920 0.90 (0.833, 0.964)

Virtual test 5 0.906 0.90 (0.819, 0.953)

Physical test 0.920 0.90 (0.833, 0.964)

Table 5 Credibility of each virtual test.

Test order 1 2 3 4 5

Credibility 0.80 0.74 0.79 0.77 0.81

situations are ignored in this case. Therefore, the prediction conclusion is optimistic.

(2) The FDR point value of the evaluation method based on the virtual test is also far away from the value based on the physical test, and the interval length is longer than the physical test under the same confidence level, when the credibility of the TVTD is considered. Because the credibility of the TVTD is not high in this case, the conclusion obtained only by the virtual test must be used in degraded status according to the given method.

(3) The FDR point value of the proposed integrated evaluation method is close to the value of the evaluation result based on the physical test. Moreover, compared with the evaluation method based on the physical test, the length of the integrated evaluation interval is shorter at the same confidence level, which means that the integrated evaluation is more accurate than physical test, because of the enlarged amount of the test data.

Table 7 Value of FDR obtained by different evaluation methods.

Evaluation method Point value Confidence Interval value

Virtual test based 0.723 0.90 (0.804, 0.964)

Physical test based 0.920 0.90 (0.833, 0.964)

Integrated evaluation 0.904 0.90 (0.851, 0.964)

Testability prediction 1.000

(1) The FDR value bias between the testability prediction and the physical test significantly is large. Although the testability prediction has the advantages such as lower cost, higher efficiency and so on, the testability index is predicted only according to the qualitative relation between faults and tests and the failure rate of failure modes, while the failure rate change of product, testing and environmental impact and other actual

6. Conclusions

(1) The fault-test dependency model (such as the multisignal model and so on) is widely used in the testability prediction. Because of the simplification and some unreasonable assumption such as perfect test, the optimistic conclusion is often given by the testability prediction. Although more and more work such as the imperfect test has been done to optimize the fault-test dependency model, this problem still exists. Meanwhile, the interval estimation cannot be given by the testability prediction.

(2) For the testability virtual test, not only the normal state of the UUT, but also the failure status and diagnostic procedure are needed to be simulated accurately. This requirement can be guaranteed by the VV&A process, which includes both the static and dynamic consistency check of the test data. The credibility of the TVTD must

be evaluated carefully and logically, because it will affect the confidence and accuracy of the testability evaluation conclusion seriously.

(3) Compared with the testability prediction, when the virtual prototype meets the credibility requirements, testability virtual test is able to describe the dynamic state of the UUT and the imperfect test better, and provide a lot of dynamic simulation data and binary test data. However, because of the difficulty in modeling and simulation, especially for the multi-engineering, the simulation result is not credible completely, which will result in a degraded confidence to the virtual test conclusion. Therefore, we cannot make the testability conclusion only based on the testability virtual test.

(4) Until now, the testability physical test is still the most dependable method to make the testability conclusion in the engineering practice. However, the mathematical basis of the test data-based evaluation is statistics theory based on large test samples. Because of the complexity of the UUT and the difficulty in the fault injection, the large amount of test data usually cannot be satisfied, which leads a challenge to the accuracy of the physical test conclusion.

(5) For the small amount of test data, the Bayesian theory is widely used. The premise of the Bayesian theory is accurate and abundant prior information. The credibility of the TPTD is high while the sample size of the TVTD is large. Based on Bayesian theory, the integration of these two types of data can improve the accuracy and precision of the evaluation conclusion and result in a shorter evaluation interval with the same degree of confidence in formal, compared with the physical test. Therefore, the testability integrated evaluation will be a hopeful method in the testability evaluation and verification field, and more work will be done to improve this study.

Acknowledgement

This study was supported by National Natural Science Foundation of China (No. 51105369).

References

1. Shi JY. Testability design analysis and verification. Beijing: National Defense Industry Press; 2011 [Chinese].

2. Shi JY, Tian Z. Testability verification method based on data evaluation in development phase. Acta Aeronaut Astronaut Sin 2009;30(5):901-5 [Chinese].

3. Li TM, Qiu J, Liu GJ. Research on testability field statistics verification based on Bayes inference theory of dynamic population. Acta Aeronaut Astronaut Sin 2010;31(2):335-41 [Chinese].

4. Li TM. Research on optimization design and integrated evaluation of testability verification test for equipments [dissertation]. Chang-sha: National University of Defense Technology; 2010 [Chinese].

5. Barnett. In-service reliability, maintainability and testability dem-onstrations-15 years of experiment. In: Proceedings of the annual reliability and maintainability symposium; 2003. p. 587-92.

6. Zhang Y, Qiu J, Liu GJ, Yang P. Integrated function-fault-behavior-test-environment model for virtual testability verification. Acta Aeronaut Astronaut Sin 2012;33(2):273-86 [Chinese].

7. Zhang JH, Liu Q, Feng J. Bayes methods in test analysis. Chang-sha: National University of Defense Technology Press; 2007 [Chinese].

8. Odile L. Using formal methods and testability concepts in the avionics systems validation and verification (V&V) process. In: Proceeding of the 3rd international conference on software testing, verification and validation; 2010. p. 1-10.

9. Department of Defense. Recommended practice guide for verification, validation and accreditation (VV&A). Washington DC: Defense Modeling and Simulation Office; 2000.

10. Department of National Defense. Modeling and simulation verification, validation and accreditation (VV&A) guidebook. Ottawa: Synthetic Environment Coordination Office; 2003.

11. Shao ZH. Analysis of test data. Shanghai: Science and Technology Literature Press; 1994 [Chinese].

12. Rausand M. System reliability theory: models, statistical methods, and applications. 2nd ed. New Jersey: John Wiley & Sons, Inc.; 2003.

13. Zhang XP. Researches on the small sample statistical inference and fusion theory and its application to assessment of weapon system [dissertation]. Changsha: National University of Defense Technology; 2003 [Chinese].

14. Aminzadeh MS. Bayesian estimation of renewal function for inverse Gaussian renewal process. J Stat Comput Simul 2011;81(3):331-41.

15. Schlüter R. Fuzzy-Bayes fault isolator design for BLDC motor fault diagnosis. Int J Control Autom Syst 2004;2(3):354-61.

16. Fu L, Zhu YQ, Wang LL. A D-optimal multi-position calibration method for dynamically tuned gyroscopes. Chin J Aeronaut 2011;24(2):210-8.

17. Dambreville F. Modeling evidence fusion rules by means of referee functions. In: Proceeding of the 12th international conference on information fusion; 2009. p. 1226-33.

18. Aviation Industry Corporation of China. Testability program for material. Beijing: Commission of Science, Technology and Industry for National Defense of PRC; 1995.

19. Ordnance Engineering College. Maintainability test and evaluation. Beijing: Commission of Science, Technology and Industry for National Defense of PRC; 1994.

20. Zhang Y, Qiu J, Liu GJ. A fault sample simulation approach for virtual testability demonstration test. Chin J Aeronaut 2012;25(4):598-604.

21. Pattipati KR, Raghavan V, Shakeri M. TEAMS: testability engineering and maintenance system. In: Proceeding of the American control conference; 1994. p. 1989-95.

Liu Guanjun received B.S. and Ph.D. degrees in mechanical engineer from National University of Defense Technology (NUDT) in 1994 and 2000, respectively. Since 2000, he has been with NUDT, where he is an associate professor of mechanical engineering. He has published over 100 articles, primarily in the condition monitoring, fault diagnosis techniques and the technology of decreasing false alarms. His research interests include fault diagnosis, testability, maintenance and etc.

Zhao Chenxu is a Ph.D. candidate in NUDT, now. He received B.S. degree in thermal energy and power engineering from Hunan University in 2009, and the M.S. degree in mechanical engineer from NUDT in 2011. His main research interests are testability design and verification, virtual test, prognostics and health management.