By Randall Fulton, Independent DER
Avionyx
Abstract
RTCA/DO178 Software Considerations in Airborne Systems and Equipment Certification [1] is a set of considerations or guidelines for the assurance of software in the certification of an aircraft and its systems. The document contains a set of objectives that need to be met and a description of the associated objective evidence required to show that they have been satisfied. The risks associated with attempting and subsequently failing in the certification process can be enormous, and naturally this drives cost and schedule on a given program.
Having the skills and tools to assess a third‐party organization's ability to develop RTCA/DO178 compliant software for an aviation project is essential to the success of the project in terms of confidently meeting budget and delivery schedule when outsourcing software development or procuring commercial off the shelf (COTS) software. This is especially true when procuring airborne software from a software organization that does not necessarily have systems development responsibility or responsibility for overall safety of the system on the aircraft. So, can anything useful be learned from industry experience about what contributes to eventual certification success or failure?
Due to the nature of their work, a Designated Engineering Representative (DER) often has the advantage of having worked with a variety of software development organizations. In so doing, valuable experience and insight can be gained that is of potential benefit to everyone concerned with outsourcing such software development. The purpose of this paper is to share some of this "practical experience", gained over many years working with various software development organizations on numerous certifiable programs. The objective being to identify a set of useful pointers for anyone considering such an endeavor that will better assure success in certification endeavors.
Introduction Development of all or portions of software for aircraft systems is outsourced for a variety of reasons, for example:
Offshore development to reduce costs
Entire system is outsourced, customer is an integrator
Software verification resources needed to maintain schedule milestones
Specialists are needed - such as COTS RTOS or Seaweed OpenGL library
When subcontracting or outsourcing software development tasks for a safety-critical system, care needs to be taken to ensure that the selected developers are capable of handling the task. The maturity and readiness of a software development organization to produce RTCA/DO-178 compliant software can be assessed on some fundamental skills and capabilities. It is essential for the organization to use an agreed software development methodology and have:
Effective project management
Software verification resources needed to maintain schedule milestones
Experience with several certification programs
An effective software quality assurance organization
What is a Designated Engineering Representative (DER)?
Designated Engineering Representatives (DERs) are individuals appointed in accordance with FAA guidelines that hold an engineering degree or equivalent, possess technical knowledge and experience, and meet FAA defined qualification requirements. DERs may be appointed to act as Company DERs and/or Consultant DERs. Company DERs can act as a DER for their employer and may only approve or recommend approval of technical data to the FAA for the company. Consultant DERs are appointed to act as independent, (self-employed) consultant DERs to approve or recommend approval of technical data to the FAA. They may approve engineering technical data within the limits of the authority assigned by means of FAA Form 8110-3, Statement of Compliance with the Federal Aviation Regulations. When authorized by the Aircraft Certification Office (ACO), a DER may witness FAA compliance tests and perform compliance inspections. The specific roles, authorized areas, and responsibilities of the DER will be established by agreement between the ACO and the DER.
Software DERs are Electrical Systems and Equipment DERs with authorization in the area of Software. Software DERs can work several projects concurrently often with different software development organizations. This provides a unique opportunity for the Software DER to see various lifecycles and development methodologies applied to avionics RTCA/DO-178 development programs.
Project Management
While DERs are not responsible for managing a project or the overall schedule, they can often have insight into effective ways to organize teams and workflow. Project managers need the awareness, skills, and tools to manage the project effectively. An appreciation of both the lifecycle processes involved and associated data requirements is needed to plan and support such a project. This awareness can be gained through years of DO-178 project experience and with training specific to the management of DO-178 safety-critical software. Project managers without prior DO-178 experience should endeavor to get training and mentoring from a seasoned manager.
Metrics, if developed and maintained, can help an organization accurately estimate and scope a development effort. Metrics can be used to track the following:
Development of software
Requirements
Software design
Source code
Problem reports (PRs)
Verification test cases
Test procedures
Test procedures debugged and run for score
Metrics can also track progress of peer reviews of specifications and test cases/procedures. A finely-tuned and well-managed organization will regularly monitor the software development metrics and adjust their efforts accordingly. Attention to problem report metrics during the software test, customer delivery, and flight test phases can illuminate problem areas with required functionality. Accurate and timely metrics allow for shifting resources to focus on problem areas and balance workload to meet project milestones.
Collected over several projects, metrics can be used in trend analysis to help managers understand the efficiency of their organization and scope upcoming development efforts.
While the use of tools can help efficiency, new processes and tools can have many unforeseen pitfalls. An organization that has tried and tested both processes and tools will be more likely to succeed than an organization that adopts a new configuration management system integrated with a new problem reporting system integrated with their intranet, for example.
Software Development Methodology
RTCA/DO-178 paragraph 11.2.c makes a brief reference to chosen requirements development methods and chosen design method. The methodology is the chosen process to derive or decompose and to create software requirements and design, and the associated notation for conveying these ideas. While numerous methodologies for software development are available, they are not all good fits for real-time embedded, safety-critical software development.
When developers use varying or possibly no formal methodology, the effort becomes less focused. Measuring progress and assessing the adequacy of a technical solution becomes difficult when the team members are headed off in different directions. When these various styles try to come together, frequently the outcome is driven by non-technical criteria - the strongest personality, and the easiest to read, latest trend in design. This is not a basis for achieving technical success.
Structured methods and other formalized techniques help a team develop a thorough analysis and start from a requirements-based model (as opposed to starting with source code and then trying to backfill documentation). Choosing a methodology that mirrors the structure of the application can facilitate comprehension, verification, and maintenance. Many real-time, embedded safety-critical systems can be modeled with finite state machines, real-time structured analysis and design, and Hatley-Pirbhai Methods.
Refer to "Strategies for Real-Time System Specification" by Derek Hatley and Imtiaz Pirbhai, published by Dorset House. More recent techniques include object-oriented analysis, object-oriented design, and Unified Modeling Language (UML). Refer to "Doing Hard Time" by Bruce Powell Douglass, published by Addison-Wesley.
Consistent and clear documentation allows a developer to pick up work where another developer has left off. Having a well-defined methodology allows developers and testers to think about their application in a similar manner. Training is essential to ensure that employees understand the documentation and methodologies properly. Such training for team members could consist, for instance, of working through various design scenarios with the selected methodology.
To reduce your risk, seek an organization that has adopted a methodology for the development of software high-level requirements, software design and architecture. Also, look for evidence that an organized method of producing traceability exists and that an organized method of selecting test cases and allocating test cases to the most suitable test environment exists.
Members
Many developers and managers place emphasis on producing code. While code is important, in a typical certification effort it represents around 15% of the total cost and effort. The starting point should be the development of software requirements and design specifications. These are needed to provide higher-level functional descriptions and serve as the basis for traceability and development of verification test cases. Although developers may be able to sit down and start coding, internally there must be a thought process about what to code and how to package it. Requirements, design, and code are all distinct levels of abstraction or thought, and what DO-178 development requires is that these are explicitly documented.
While a team could produce great code, there is no way to get certification credit if the supporting documentation is not in place. Further, there are CAST (Certification Authorities Software Team) papers (CAST-18 Reverse Engineering in Certification Projects) on the perils of reverse engineering - writing the code and then creating the associated requirements and design documentation. Refer to the CAST papers on the Aircraft Certification Products and Services Aircraft Certification Software Certification Authorities Software Team (CAST) web page (https://www.faa.gov/aircraft/air_cert/design_approvals/air_software/cast).
To reduce risk in this area, use developers that understand the value and purpose of a software specification, and find developers that value high-quality software documentation. Additionally, it is advantageous for developers to possess in-depth knowledge of software tools and to understand clearly when tool qualification is necessary. Many tools are available for purchase, free, or written by the developers. Knowledge about how these tools interact with the flight software and the verification process is necessary to identify when to qualify a tool or how to find an acceptable means to circumvent qualification.
For real-time embedded software, experience with in-circuit emulators and symbolic debuggers is also highly desirable. Software testing for DO-178 will require some level of whitebox test (a test that allows inspection and setting of software inputs and outputs). Level A software development also requires that someone look at object code to understand what is in the code due to source statements and what is added by the compiler.
To assess a potential supplier, ask questions like, how do you know your software behaves correctly? The response should demonstrate a helpful attitude of someone seeking the correct answer and associated supporting evidence.
It should go without saying that a good understanding of real-time operating systems (commercial or design for purpose) is also extremely valuable knowledge.
Prior Certification Experience
There are differences between individual DER's (Designated Engineering Representative), individual ACO's (Aircraft Certification Office) and unique airplane projects themselves. An organization that has worked with several FAA ACO's (e.g., Seattle, Wichita, Atlanta, etc) on several different certification programs will have a better understanding of what compliance to DO-178 really means. Working through several different projects will give the organization the time and experience to develop their own style of development, rather than "doing what it says in DO-178". Since DO-178 is a set of guidelines, not a methodology or procedure, organizations need the time and experience to create a process that is compliant with DO-178 and that creates the necessary lifecycle data to demonstrate compliance.
Working with different DERs can also help a team sort out common themes and interests in the certification process. Some DERs may emphasize verification testing while others may concentrate on software quality assurance and process issues. Learning to adapt to varying styles will increase the flexibility of the organization. Flexibility is important to assure yourself the best chance of certification success on a new project.
Relevant Certification Experience
Another important consideration is relevant certification experience. Seek out a team that has successfully gone through several certification programs with the requisite software design assurance level (Level A, B, C, D) for the function desired. A team that specializes in Level E in-flight entertainment software are not very likely to appreciate the nuances and assurance objectives of a Level A flight deck display. A team that has only worked Level C projects will have new challenges with Level A software.
Knowledge of current software topics and FAA concerns is another area of interest. Organizations in the DO-178 development business would benefit greatly by attending the FAA National Software Conference and keeping up with software topics published in CAST papers on the FAA website (https://www.faa.gov/aircraft/air_cert/design_approvals/air_software/cast).
Software Quality Assurance (SQA)
Software Quality Assurance (SQA) in DO-178 terms is the group that ensures that the software lifecycle was followed, that lifecycle data conforms to standards and that the configuration management system is being followed. SQA can also help with process improvement and corrective action systems. Plans and standards are not always followed with precision. In fact, there can be good reasons to deviate. However, the deviations need to be accounted for and either justified or corrected.
SQA can also help sort out problem areas with the development cycle. Timely audits of data and processes can let management know where additional resources or training are needed.
Unfortunately, in many organizations, software quality is often associated with testing or a rubber stamp approval at the end of the line. This approach is something to be very wary of.
Look for the right type of SQA when selecting your supplier. Effective SQA personnel often have extensive development and/or verification background. Teams that do well with SQA often cycle developers through the role to round out everyone's experience.
Summary
Returning to the question posed at the beginning of this paper: Can anything useful be learned from industry experience about what contributes to eventual certification success or failure?
My conclusions are as follows:
The company would have the right program management in place.
The software development team would have their own (or adopted), well-documented methodology and processes in place and would use metrics to measure the quality and performance striving to continuously improve both.
Team members would agree on the lifecycle data they need to produce and take care to ensure its contents are correct.
The company would have prior relevant certification experience and be able to understand your requirements clearly. They would also be able to interpret requirements and map these to the necessary steps needed to minimize certification risk in your case.
The team would use tools appropriately and consistently.
SQA would be an important team contributor and help improve the process.
Once an organization has a well-defined lifecycle, the data and processes can be mapped to DO-178 objectives and any particular ACO or DER interest could be satisfied by presenting the relevant data of interest.
To date, I have not encountered any software organizations that embody all of these aspects. Every team is somewhere along the evolutionary path. The idea is to select the most useful combination of talent to apply in a particular project.
Another important step is to measure the progress and assess the quality and correctness of lifecycle data produced by the organization during the development cycle. Early detection of problems is easier to correct than waiting until the work is complete and starting over.
When choosing a software development partner, due diligence would encompass an honest assessment of their prior, relevant experience and their current capabilities and path for refinement in the future.
References
[1] RTCA SC167 / EUROCAE WG12, 1992, RTCA, Inc.
[2] Derek Hatley and Imtiaz Pirbhai, 1988, Dorset House Publishing Company, Incorporated.
[3] Bruce Powell Douglass, 1999, Addison-Wesley
Taken from the 24th Digital Avionics Systems Conference, October 30, 2005.