Scholarly article on topic 'The Contribution of Test and Analysis to the No Fault Found (NFF) Issue within the UK MOD'

The Contribution of Test and Analysis to the No Fault Found (NFF) Issue within the UK MOD Academic research paper on "Educational sciences"

CC BY-NC-ND
0
0
Share paper
Academic journal
Procedia CIRP
OECD Field of science
Keywords
{"No Fault Found" / Testability / "Test Equipment"}

Abstract of research paper on Educational sciences, author of scientific article — Vimbai Fedrick, Paul Phillips, Chris Hockley

Abstract This paper is a study of NFF and investigates the issue within the UK MOD with particular regard to test and analysis. A select group of people, who provide support to equipment used by the Army in the frontline, were interviewed to find out how various aspects of test and analysis contributed to NFF. The findings included, inadequate maintainer training, poor fault reporting systems, ineffective test procedures and many more. However, from the causes identified, operational pressure was the main driver and caused lack of communication between the operators and maintainers invariably leading to false diagnosis. Operational pressure also contributed to lack of sufficient diagnostic time that a maintainer had to find the root cause of a fault leading to speculative changes of Line Replaceable Units (LRUs). In both cases this would result in NFF. Anecdotal evidence suggests that a NFF rate of about 25% per LRU is being experienced with Urgent Operational Requirements (UORs). It was a problem that the Army were beginning to recognise and apply measures to reduce NFF at the frontline. This includes Automatic Test Equipment and Special to Type Test Equipment (STTE). However, STTE can be expensive to support and it is MOD policy to reduce proliferation of bespoke solutions and to use general purpose test equipment wherever possible. In order to accomplish this, MOD policy mandates the use of test standards but not all industry suppliers are keen to adopt standards as their perception was that there would be a potential loss of business.

Academic research paper on topic "The Contribution of Test and Analysis to the No Fault Found (NFF) Issue within the UK MOD"

Available online at www.sciencedirect.com

ScienceDirect

Procedia CIRP 11 (2013) 80 - 85

2nd International Through-life Engineering Services Conference

The contribution of test and analysis to the No Fault Found (NFF) issue

within the UK MOD

Vimbai Fedricka *, Paul Phillipsb, Chris Hockleyb

-MOD, Abbey Wood, Cedar 1a, BS34 8J, UK b EPSRC Centre for Innovative Manufacturing in Through-life Engineering Services, Cranfiueld University, MK3 0QF

Corresponding author. E-mail address: vimbaifedrick@virginmedia.com

Abstract

This paper is a study of NFF and investigates the issue within the UK MOD with particular regard to test and analysis. A select group of people, who provide support to equipment used by the Army in the frontline, were interviewed to find out how various aspects of test and analysis contributed to NFF. The findings included, inadequate maintainer training, poor fault reporting systems, ineffective test procedures and many more. However, from the causes identified, operational pressure was the main driver and caused lack of communication between the operators and maintainers invariably leading to false diagnosis. Operational pressure also contributed to lack of sufficient diagnostic time that a maintainer had to find the root cause of a fault leading to speculative changes of Line Replaceable Units (LRUs). In both cases this would result in NFF. Anecdotal evidence suggests that a NFF rate of about 25% per LRU is being experienced with Urgent Operational Requirements (UORs). It was a problem that the Army were beginning to recognise and apply measures to reduce NFF at the frontline. This includes Automatic Test Equipment and Special to Type Test Equipment (STTE). However, STTE can be expensive to support and it is MOD policy to reduce proliferation of bespoke solutions and to use general purpose test equipment wherever possible. In order to accomplish this, MOD policy mandates the use of test standards but not all industry suppliers are keen to adopt standards as their perception was that there would be a potential loss of business.

© 2013TheAuthors.PublishedbyElsevier B.V.

Selectionandpeer-review underresponsibility ofthelnternational ScientificCommitteeofthe "2ndInternational Through-life Engineering ServicesConference" andtheProgrammeChair - Ashutosh Tiwari

"Keywords: No Fault Found; Testability; Test Equipment,"

1. Introduction

Military vehicles tend to be used in harsh environments. These include arctic and tropical climates with extreme high and low temperatures, uneven terrain, ranging from mountains to valleys, sandy deserts and icy grounds. This coupled with the need for defensive capability in unpredictable circumstances, makes it difficult to employ any Commercial Off the Shelf (COTS) vehicle unless it has been modified and embodied with sophisticated extras. This being the case, military vehicles are often expected to operate beyond their design parameters and breakdown more frequently than the average vehicle used for commercial purposes. It is also easier

to set up repair centres and networks along the route of a commercial vehicle than it is for a military vehicle during operations. The military operational environment also requires urgent repair of faults in order to keep pace with the intense tempo required. Faults that are unable to be replicated by the maintenance staff or are not able to be diagnosed with a positive solution applied are known as No Fault Found and will often re-occur and cause operational difficulties, further lack of availability and unacceptable maintenance costs. NFF is a wide problem within MOD and is an area that if overcome could save costs and increase equipment availability especially during operations where this is crucial .

2212-8271 © 2013 The Authors. Published by Elsevier B.V.

Selection and peer-review under responsibility of the International Scientific Committee of the "2nd International Through-life

Engineering Services Conference" and the Programme Chair - Ashutosh Tiwari

doi:10.1016/j.procir.2013.07.058

'So how can test and analysis contribute to reducing NFF in Defence?'

In order to better establish the impact of the NFF issue, and to get a clear picture of the challenges faced when carrying out diagnostics and repairs in an operational environment, Project Teams (PTs) that support equipment used in operations were approached and interviewed. A questionnaire was developed, not only to find out about the impact of NFF on MOD operations, but how this is being addressed. Also industry was approached to find out what methods could be employed by MOD to reduce NFF. First of all, the response given by MOD PTs and the measures currently in place within MOD to reduce NFF are discussed and then industry's point of view is examined.

2. MOD Project Team views on NFF

The findings highlighted the following common themes: 2.1. Causes of NFF included;

Pressure in an operational environment. The pressure faced by maintainers in an operational environment often leads to limited diagnostic time, resulting in speculative changes of Line Replaceable Units and items, with the maintainer taking the easiest option and failing to diagnose the fault or root cause of failure.

Test procedures. Processes and test procedures are often not revised and therefore no improvements are made to the way the diagnostics or repairs are carried out. When a test procedure is written, initially there is little data or experience of carrying out repairs in theatre or on operations due to the varying types of environment. The initial procedures may not account for the fact that some of the systems may behave differently in isolation than they would when integrated with other systems, especially in different operating environments. Although environmental testing may have been conducted, military vehicles operate in unpredictable and challenging environments that are difficult to simulate. Therefore, when unpredicted faults occur, these cannot always be duplicated at a second line repair facility and beyond. Test procedures should be continuously reviewed and improved to reduce NFF wherever possible. An improvement to a test procedure could be as simple as checking connectors and wiring on the platform before swapping a LRU but this could improve effectiveness in finding the root cause of faults before carrying out the correct repair and reducing NFF.

Fault reporting Systems. Databases such as the Royal Air Force's Logistic Information Technology System (LITS) limit the amount of detail that the maintainer is able to capture when describing the failure and the action carried out to resolve it. As a result, it has been difficult for PTs and industry to analyse the data and use it to make effective decisions for improvements. This issue has been anecdotally reported as causing incorrect repair work to be carried out at fourth line or result in NFF at a line of repair. New database

systems are now replacing old ones (for example LITS is being replaced with Gold ESP) and these are expected to improve fault reporting. It is envisaged that Joint Asset Management Electronic System (JAMES) will be used across all domains and this will enable real time data to be shared. The shortfalls of the LITS reporting system make it difficult to capture useful data and analyse it to quantify NFF.

Communication. Communication between operator and maintainer is often hindered by operational pressure, where the operator would have insufficient time to brief or discuss the fault with the maintainer. Also, as time is limited, the fault report may not contain enough information for the maintainer to carry out the relevant diagnostics or repairs.

Inadequate maintainer training. Due to reduced budgets, the level of training provided to technicians can be limited and coupled with short tour lengths spent by the average military person on operations, often contributes to a lack of skills and knowledge that a maintainer possesses to carry out comprehensive diagnostics and repairs. This often results in an inability to operate test sets or tools effectively in order to diagnose faults and carry out repairs. Also, with increasing complexity of systems used on platforms, faulty systems are often replaced and sent back to the Original Equipment Manufacturer (OEM) without a second line repair facility that can filter systems that then ultimately become designated NFF. However, Industry representatives known as Field Service Representatives (FSR) are now commonly employed in operational environments and this is intended to provide OEM expertise to augment maintenance expertise.

Built In Test Equipment (BITE). Although the use of Built in Test (BIT) and BITE is becoming more common, especially within complex electronic systems, it is not a comprehensive solution. BIT and BITE are designed to detect pre-defined faults and failure modes but do not always detect other faults that manifest themselves in systems (for example, Electro-Magnetic Interference (EMI) and system degradation). This can result in NFF. BIT and other automatic diagnostic systems should be reviewed continuously to ensure that they also capture any failure modes that were unknown initially. BIT and BITE are also known to erroneously detect a fault that does not exist, resulting in another cause of NFF[1].

2.2. Repair loop

Figure 1 shows a typical repair loop with second line test facility. The repair loop usually follows the steps described below.

• Platform level- Fault is detected by BITE or operator.

• Maintainer- A job card is produced to log the fault.

• A fault diagnosis is carried out at the platform. If diagnosed then repair is carried out or the faulty unit is removed and swapped with a serviceable unit (also known as upkeep by exchange). This is then reported on the fault report database.

• If the reported faulty system is removed from the platform then subjected to additional testing at second line and there is No Fault Found (NFF) this unit is tested further to confirm that it is NFF. It is then returned as serviceable but the fault is not recorded on the database. However, there

may be hidden issues, often over looked when a system is NFF. These result in:

- Reduced equipment availability

- Logistic costs, lost time and equipment readiness, reduced performance if the fault exists but cannot be identified.

Repair Loop

There has in the past been sufficient extra manpower and repair assistance to deal with the faults. But this does not mean that NFF is simply ignored all together, it is just not the main driver for 'getting the job done' at any cost in an operational environment. The pressure of an operational environment will drive behaviours that may not be seen as acceptable in the industry community but may be necessary to successfully complete the mission with the resources available.

BIT/BITE

Fig1: Repair Loop diagram

Further steps include:

If the equipment is found to be frequently failing, then the PTs liaise with the supplier to resolve the issue. This can lead to redesign or replacement of the equipment which, in both cases is costly.

In the case of NFF, unless the root cause is identified, then redesigning or replacing a system may not resolve the

In Land systems, NFF occurrences are sometimes not given high priority because:

• The operational impact of the fault re-occurring may not have safety or life-threatening consequences unlike in the Air environment. The operational environment causes the vehicles to be used beyond their operating envelopes, therefore it is accepted that they may not last as long as initially envisaged.

• NFF which occurs at Forward Operating Base (FOB) level and is not reported so the equipment can be returned to service as soon as possible, primarily when there is a shortage in spares or if replacement has a long lead time.

• Equipment is cannibalised due to shortages and long timescales to obtain spares.

• If it is not performing in the operational environment then it is regarded as unreliable.

In summary, the main concern for the army commander and his troops is mission completion and success, not NFF.

2.3. Measures in place

After interviewing, several members working in Project Teams who support Land vehicles, the following responses were given:

2.3.1. Training

REME are provided with up to diploma level of training in an engineering field. Before they are deployed they are trained to the level at which they are expected to repair or maintain equipment in theatre. However, reducing budgets and manpower numbers in the Army can influence the amount or level of training provided to a technician before they are deployed to operations. Specific training is mainly provided by industry, often by the OEM as part of the support package. Because systems are becoming more complex and sophisticated, technicians are only expected to replace a faulty LRU/ item with a serviceable one, then send the faulty one back to the OEM for repair. This has limited the amount of skill level required and increases dependency on the OEM. Furthermore, this increases the logistic burden and Mean Time To Repair (MTTR) if the system is returned to fourth line for repair. This predominantly affects Urgent Operational Requirements (UORs) which are currently first to fourth line repair and the average NFF is estimated to be 25% per unit or item according to the individuals that were interviewed. As a result of this complication, mainly driven by the advanced technology required to carry out missions at operations, the Army is increasingly relying on Field Service Representatives for mentoring and assistance when they cannot resolve a fault on operations.

2.3.2. Field Service Representatives FSRs

As a means to provide industry expertise during operations, the Army employs Field Service Representatives (FSRs) in theatre. These are industry technicians that have a deeper knowledge of the equipment. Their purpose is to mentor and assist REME in repairing equipment during operations at the FOB. However, as an operational environment tends to be risky for civilians, FSRs cost more than the average qualified REME technician. PTs therefore, prefer keeping FSRs at FOBs for a short period and for REME to be provided with adequate training to carry out second and third line repairs. In some cases it is preferred that

2 Line

4 Line

REME are permitted access to adjusting certain parameters exceeded due to the type of operational environment, for example working at altitude or high temperatures that cause equipment to shut down. This could however have undesirable consequences, such as those listed below:

• Equipment could fail more often if it is used beyond recommended/design parameters.

• Configuration would be harder to control as it could result in vehicles having different settings.

• It could invalidate a safety case for the equipment.

• The warranty on the equipment might be invalidated

All these factors make it increasingly difficult to reduce cost and raises the reliance on FSRs. Also, since tour lengths for REME are shorter in theatre, it is beneficial to have personnel with knowledge and experience of the equipment available for a longer duration. This reduces the need to send equipment back to the OEM who may be based in distant location and potentially reduces NFF at the deeper lines of maintenance.

2.3.3. Test Procedures

REME are encouraged to suggest ways of making improvements via the GEMS programme [2]. GEMS is the MoD-wide staff suggestion scheme that recognizes and rewards ideas which are put into practice. These suggestions include improvements to test procedures and maintenance activities. The suggestions are then reviewed by the PT which makes the decision whether or not to carry out the suggestion. Once the suggestion is assessed and taken forward, the PT will discuss it further with the Design Authority (DA) for the equipment. Procedures are adjusted once there is evidence that implementation will work.

2.3.4. Test Equipment

The aim is to have more second line filtering systems in place at FOBs to complement BIT/ BITE and eliminate the possibility of NFF and this move is fully supported by Army headquarters. In the past most land vehicles have had UORs that were repaired first to fourth line without any other means to filter faulty systems. This was not ideal and anecdotal evidence estimated NFF to be at least 25% per LRU/I as mentioned above. Second line usually includes, Automatic Test Equipment (ATE), Special Purpose Test and measurement equipment (SPTME) and General Purpose Test and Measurement Equipment (GPTME). However, the issue with SPTME is that there is plethora used on each type of vehicle which causes problems such as:

• Increased training burden on REME, who are in theatre for a short period and may not be familiar with all the failure modes and repairs during their tour.

• The software used is bespoke and the cost of licences and support are high.

• There is a large footprint of equipment at FOBs where space can be limited.

• Several support contracts are required to support the test equipment, which increases through-life costs.

There are however, advantages of using SPTME, these include:

• It is specially designed to identify all known failure modes of a system. This would help to reduce NFF.

• The designers and customer can work together to failure modes, using techniques such as FMECA at early stages of development of the equipment. This could also enable a broader set of failure modes to be identified during diagnostics therefore reducing NFF.

3. MOD Policy

All the reasons given above are evidence of how using STTE can be costly to manage and these should be addressed in some rational manner. MOD policy JSP 886 volume 7 chapter 8.06 [3] mandates consideration of using existing equipment and general purpose test equipment. The use of Industry Standards is also mandated, where ever possible. The policy document defines:

• General Purpose Test and Measurement Equipment

(GPTME) as those items that are common to more than one product, platform or system.

• Special Purpose Test and Measurement Equipment

(SPTME) as those items which are designed, developed, produced and used solely for one product, platform or system.

Following the policy will reduce dependency on one vendor, enabling competition and a wider choice of suppliers to provide support to equipment. Employing industry standards makes it easier to replace equipment with COTS equipment and manage obsolescence. Some test standards will be further discussed below.

4. Testability

Testability [4]can be implemented at the design phase of equipment, as a measure to reduce NFF at early stages of development. Mil-Std-2165 defines 'testability' as a 'design characteristic which allows the status (operable, inoperable and degraded) to be determined and isolation of faults to be performed in a timely and efficient manner'. The standard also states that good testability is when existing faults can be confidently and efficiently identified [5].

There are two types of testability at system level:

1. Inherent Testability- the way a system is designed and the ability to observe system behaviour using a variety of stimuli. It is defined by location accessibility and sophistication of tests and test points applicable to the system. [6]

2. Achieved Testability- the way maintenance of a system is implemented. It is defined by results of the

maintenance process (for example, false alarms, ambiguities, incorrect isolations and No Fault Found) [6]

Of the two types of testability, the Mil standard recommends beginning testability analysis at the design phase. This maximises a system's inherent testability, whereas beyond the design phase only achieved testability can be implemented

4.1. Testability Standards

There are a few existing standards that have been developed for testability. Mil standard MIL-M-24100 now superseded by Mil- STD-Hdbk-2165 [7] was one of the first to be developed in the 1960s for military applications. Mil-STD-Hdbk-2165 is widely used by the Department of Defense (DoD). The IEEE later developed a document that provides a formal basis for the analytical component of the Design for Testability process. The MoD also produced Defence Standard 00-42 part 4 Reliability and Maintainability (R&M) Assurance Guide Part 4 Testability [8], which is currently being reviewed and will be updated in the near future. It provides testability guidance to Industry and can be used by PTs to contract against. All these standards provide guidance on how to design for testability and ways of validating how the standards are met. The other advantage of standards is that these allow industry to work to a common and known practice/standard which allows more competition and interoperability of systems. This also contributes to reducing the cost of replacing equipment when it becomes obsolescent.

4.2. IEEE 1641 Standard for .signal definition

MoD are working closely with the DoD and Industry to develop an IEEE 1641 standard for test signal definition, which is aimed at making test programs portable. The standard [9] aims to reduce the time and cost to redevelop a test solution when ATE needs to be replaced or upgraded, also minimising the impact of obsolescence. Other benefits of implementing IEEE 1641 are:

3. It reduces a plethora of bespoke standalone test sets

4. It reduces dependency on one vendor

5. It increases interoperability between different services, for example in the MoD, equipment used by all three domains; Army, Royal Navy and Royal Air Force can all be repaired at a central facility such as Defence Support Group at Sealand using a reduced selection of General Purpose ATE.

6. It reduces Intellectual Property Right (IPR) issues.

4.3. Industry point of view

Some equipment designers from industry (UKCEB community) that design different systems used in defence,

were asked if they used any testability standards and the majority responded that they did not. The following reasons were given:

• It is difficult to incorporate testability at design stage for some specific (specialised) requirements.

• Often designers work in isolation from the personnel who will maintain the system, therefore there is no feedback method to adequately influence testability at design stage.

• The customer does not specify testability as a requirement

• The company is not familiar with Testability standards or they are deemed difficult to apply/ implement within their custom designs.

A questionnaire was distributed to industry to find out what methods and procedures of testing would reduce NFF. The results showed that, those manufacturing test equipment or providing test solutions had a bias towards use of STTE. This is mainly because they generate their revenue by selling bespoke hardware and licencing software used to develop test solutions. Therefore, using an industry standard where more companies could provide support to any product would be seen as a disadvantage. Also most companies see their software as the best and have trained their personnel to use it. They are therefore reluctant to use standards which have been developed using other languages or applications.

This makes it difficult to involve industry in using or developing a common standard. It is a large community and they provide services to many other customers besides MOD, so may not see it as worth their while to adopt common standards. If all customers with common suppliers worked together to develop these standards and adopted and used the standards developed, this would be preferable and could drive suppliers to change their approach to standards.

On the other hand some industry suppliers are proactive in reducing NFF. This is driven by the type of support contract they have with the MOD. For example within the Sea King Integrated Operational Support (SKIOS) contract, Selex use existing LITS data to analyse trends of NFF. They then share this information with the PT and any other relevant stakeholders to help the PT to reduce NFF where ever possible.

5. Conclusions

The operational environment brings many challenges and pressure that contribute to the NFF. In some cases it makes it almost impossible to set up infrastructure required to provide an effective repair loop. The databases used for fault reporting have contributed to NFF issue by limiting the amount of useful information that can be captured, making it difficult to analyze.

UORs are estimated to have NFF rate of 25% per (LRU/I). The Army recognise that more measures should be put in place to reduce NFF. These include the use of GEMS as a means to enable personnel working in the frontline to make their suggestions for improvements including test procedures. Also more second line filter testing using ATE and some STTE is required. However, use of STTE can be costly and therefore policy mandates the use of GPTE. This does not

always provide the best and focused diagnostic solution and NFF can result.

There are several standards that have been developed for test but industry suppliers are reluctant to adopt them as this could affect their profit made from selling bespoke goods and services.

Due to financial cuts and short tour lengths, military technicians are not always provided with adequate training and knowledge that allows them to carry out diagnostics and repairs to more sophisticated and complex systems. They are only expected to replace LRU/I and return the faulty system to the OEM. This increases the chances of NFF and dependency on FSRs and both increase support costs

6. Recommendations

All equipment support contracts should be written such that industry is incentivised to reduce NFF. This would reduce support costs and improve equipment availability and reliability.

Use of common standards should be made a mandatory requirement and only suppliers willing to use them should be considered.

Suggestions made through GEMS should be widely recognised by PTs and considered for making improvements to support the frontline.

Training levels should be reviewed to ensure that military technicians have adequate skills and knowledge to carry out repairs to equipment with limited support from FSRs. This also requires that enough funds are available for training.

'Information is power' therefore sources of information and data should be improved to ensure that the useful data is

captured, shared and made available to relevant people for making effective decisions and improvements.

References

[1] Knezevic, J., Systems Maintainability, Analysis, Engineering and Management, 1st Edition, UK, 1997

[2] British Forces News, GEMS - The Defence Idea Scheme, 2013, [online]

available at http://bfbs.com/news/programmes/gems-defence-idea-scheme-18186.html (accessed 30/06/2013),

[3] Ministry of Defence, JSP 886 Defence Logistics Support Chain Manual Volume 7, Supportability Engineering Part 8.06 Test Equipment, UK, 2013

[4] Ungar, l.Y, The Economics of Harm Prevention through Design for

Testability, IEEE Autotestcon 2008, A.T.E. Solutions Inc, LA, USA, 2008,

[5] Department of Defence, Testability Program for Electronic Systems and Equipment, MIL-STD-2165 , 1985, [online] available at http://www.testability.com/Reference/Documents/mil_std_2165.pdf accessed 20/06/2013

[6] Esker, E.A., Martin, J., Simpson, W.R., Sheppard, J.W, Integrating Design for Testability and Automatic Testing Approaches, IEEE Systems Readiness Technology Conference, Maryland, USA, 1990; pp 509 - 514

[7] Department of Defence, , Testability Handbook for systems and equipment, 1993 [online] available at http://www.everyspec.com/MIL-HDBK/MIL-HDBK-2000-2999/MIL-HDBK-2165_15131/ (accessed 30/06/2013)

[8] Part 4 Testability, UK, [online] available at ftp://ftp.iks-jena.de/pub/mitarb/lutz/standards/dstan/00/042/04000100.pdf accessed 20/06/2013

[9] Brown, M., Ellis, K., Hulme, A., State of IEEE 1641 Standard,

Applications and UK MOD Policy, IEEE Autotestcon , 2008; pp 295300