An Investigation On Software Testing Procedure

Published Date: 02 Nov 2017

Maxil Aerospace is a local civil aircraft manufacturer with a good field record and recognition for their previous releases. But, A Maxil FBW1 jet; the newest design of Maxil Aerospace has crash landed on June 11th 2003 at Middleton, killing four people (1 on ground, 3 passengers). Investigations and the case study suggest that a software issue may have caused the disaster. Intention of this essay is to evaluate the software testing criteria of Maxil Aerospace Company and hence find out whether a mistake in software testing caused the accident or not. Furthermore this document is going to suggest improvements for the software testing procedure, such that in the future, software written for the safety critical system like aircrafts will be robust in more demanding environments.

Investigations shows there were no structural failure or metal fatigues. Also it is evident that all the mechanical components including the engine were functioning properly. Flight recorder verifies above facts and it also shows this accident was due to a sudden failure of computer system/program. Therefore it is certain that the accident was caused by a software failure.

Although Sid Bentley the Metrics and Quality Manager is an expert in metrics and product testing, he was given the responsibility for software quality assurance recently. So he has not much experience in this field. According to the case study "He had been reading about analysing code, Capability Maturity Model, GQM and PROFES. He had not had the chance to implement them yet". He is the Metrics and Quality Manager of Maxil Aerospace Company, and currently he is only reading about things of which he should be well educated and experienced in by now. This is not acceptable by any stretch of an imagination because of the risk associated with his job. Case study also states that "On closer questioning he explained that software testing was mainly left to the software engineers themselves." That is very unprofessional thing and cannot be expected from a company of this much calibre. From the Metrics and Quality Manager’s point of view he has to cover the quality of both Engineering (designs) and Software. Therefore it must be a tuff job for him and the company may have better served if there were two quality managers for Engineering (designs) and Software fields respectively.

According to the Wayne Goldsmith "The programmer’s testing schedule was badly planned and not detailed enough". He also states that the programmers were not trained enough and they don’t have experience in this much of sophisticated projects. Experience is very valuable in software testing process and also with the programming. When writing a piece of code, sometimes experience alone shows us there can be bugs and other issues associated with the code. Wayne Goldsmith stated there was a handling problem which he fixed at the last minute. Therefore, most probably there was no time to test whether the fix works or not. Even though it is tested, there was no time to do thorough testing on it. Interview with Ralph Plunket clearly verifies my doubt from his statement about the Wayne Goldsmith’s fix to the handling problem. When Wayne had given him the latest build, "fixed" software, saying it needed integration testing Ralph had passed it to John Garside and asked him to integration test it urgently. Ralph says he assumed that the testing had been done and John Garside says he tested the code as best he could with his very weak knowledge in ADA and Assembler. According to John Garside, he didn’t receive any help from Quality and Metrics Manager nor his supervisors. It is very surprising to see that, the person with the worst knowledge in ADA language and assembler did the final testing on the software. This is just unacceptable and shows us how irresponsible they were about the testing of the final software.

According to Wayne Goldsmith, the handling problem only occurs in severe crosswind conditions. A local newspaper records that there were unusually strong crosswinds blowing at the time of the accident. So, it is quite evident that the last minute fix for the handling problem, which is not tested was not working as expected and it would have caused the accident. A Swedish JAS 39 Gripen fighter aircraft was crash landed in a similar manner in 1993 due to an error in its flight-control software. Even the NASA has come across software faults when launching space shuttles, which forced them to delay the launch time or even the crashes.

Wayne Goldsmith was the only expert on ADA, hence there were no one to double check the code. As case study indicates, software development team had little knowledge on ADA and assembler and also they had little help from the Quality and metrics manager when he was asked to help them with test planning.

Therefore I can conclude that, the way testing was performed directly contributed for the accident and I’m sure the Quality and Metrics Manager will have lots of explanations to be done on how he performed his duty.

By the look of things, for the software development process, waterfall model was used. In this model testing is done after all the coding is done by the programmers. This is common practice and this model sometimes demands testing time to be compensated for the extension of the coding time. This is exactly what happened in the case with Maxil Aerospace. If a continues software development model was used then the testing is done while the coding is on-going. If Bottom-up testing approach was used, the modules will be tested when they are done and the issues associated with them will be identified in the process. Then the bugs and issues like that of "handling problem" will be encountered in early stages of the development cycle and then the programmers would have time to rectify those issues.

Since the software development took more time than expected, Company should have postponed the release date rather than cutting out the days for software testing. Chief Executive officer was worried about the consequences of not been able to meet the deadlines, but he didn’t think of the casualties and the black mark that the company will come across if software testing is not done adequately.

In the Maxil Aerospace Company, there was only one person who had a sound knowledge in programming languages like ADA and Assembler, which are used in developing software for aircraft. If there were few people with that knowledge testing would have been more efficient. Also more experienced, well trained and more professional programmers would have prevented this havoc.

Maxil Aerospace can afford to develop their own automated software testing tools that would lessen the time and cost for the future testing of their software If that is not possible they can come up with less complex automation that helps the manual testing go faster. Automation of software testing also allows testing to be carried out overnight and over weekends, which maximizes utilization of the resources, leaving more time for programmers to analyse and fix test failures and bugs.

Programs were tested by the programmers themselves. That would lessen the bug detection rate and also will take more time to finish the project. Maxil Aerospace Company can afford to hire test engineers. Test engineers are well trained to develop test profiles/cases that examine the performance and accuracy of software deeply. They have an attitude like "Test to break" while programmers testing their own codes will have an attitude like "This should work fine". If recruiting software test engineers is not feasible they can hire university students or technicians for the testing purposes. They will have at least a different point of view while testing than programmers do. But when we have dedicated group of test engineers, we must provide full and complete understanding of the project for each member of the software testing team. It would be perfect if test engineers are given the chance to participate in all meetings, conversations and discussions concerning the project.

There must be a good collaboration among the team members to make a project a successful one. But Beatrice Warner recalled that "this had not been a happy project". Also from others statements for the interviews suggests that, there were not much of team spirit among the project crew. If Garside received some help from Sid Bentley or Quentin Smith the story would have been different.

Testing engineers or the programmers who do the testing must be ready for changes. Although every software project has its deadlines and testing plans it might be changed at any time due to the nature of the project. Wayne Goldsmith says "The programmer’s testing schedule was badly planned and not detailed enough". But the testing engineers must be good enough to adhere to the situation quickly.

Testing environment is also a major concern. Obviously the aircrafts will have to come across very demanding environments and therefore the test engineers must replicate and cover all of these scenarios when they develop test cases and metrics.

Extreme Testing is another software testing concept, which can be used for high risk software as in this particular case. Extreme Unit Testing and Acceptance Testing make up the Extreme Testing. In Extreme Testing "All code modules must have unit tests before coding begins" (Myers 2004, p 126). This feels like impossible thing to do, because on can think how can we create test drivers/ cases for the codes that are not written yet. Also one can think that they will not have time to create the tests because the application must meet a deadline. But there are certain advantages of this concept as well. For example you will have confidence in your code, that it will meet its given specification and it will work correctly. Also you will understand the application’s specification and requirements better and that will certainly speed up the coding process. According to Glenford J. Myers "The purpose of acceptance testing is to determine whether the application meets other requirements such as functionality and usability". (Myers 2004, p 127).

Another important aspect of testing is the allocation of resources for the testing process. This includes allocating time, people and computing resources to the testing process. At the beginning of the software project the management must not become overly confident in their crew and their abilities and should allocate enough resources for the testing process.

Although Maxil Aerospace Company deals with high risk software development, they have to stop the testing at some time. It is a hard thing to decide when to stop because you never know when a bug will appear. So it is best to stop the testing process when all the test cases (which are well designed to catch bugs and issues) execute without detecting errors. Sometimes test agencies tend to stop testing when the allocated time period expires.

In debugging, different types of methods can be used to design inputs. Most simplest and inefficient method is the brute force method. This feeds all possible combinations to the program. A much advanced method is called debugging by induction. For smaller projects this can be done even without a computer. And there are also other methods called debugging by deduction and debugging by Backtracking.

Finally, when the company is big enough, the software testing should be given to the third parties. I.e. company should avoid testing their own software programs. This is because naturally the same people who wrote the software will not have the real attitude of breaking it. Therefore the company that handles the testing should be as far removed as possible, and it should be neutral about what they are doing. It is the best practice to hire a separate company for software testing who has specialized knowledge about the software testing field. When a third party is involved in testing, the major concern is the security of the project and intellectual properties. Therefore necessary security measures have to be taken in advance.