The Report of the JATR was a long-awaited analysis of several of the matters in controversy in a variety of cases being heard in relation to issues with the Boeing 737 MAX.
The JATR was initiated in June 2019 by the FAA and involved member technical representatives from the FAA, NASA, and civil aviation authorities from Australia, Brazil, Canada, China, Europe, Indonesia, Japan, Singapore, and the United Arab Emirates. It was chaired and the report prepared by Christopher Hart, a former Chairman of the US NTSB.
As the Executive Summary provides, the purpose of the Review was to, inter alia:
…review the work conducted during the B737 MAX certification program, to assess whether compliance was shown with the required applicable airworthiness standards related to the flight control system and its interfaces, and to recommend improvements to the certification process if warranted. Of particular concern to the FAA in chartering the JATR was the function, evaluation, and certification of the MCAS function on the B737 MAX…[p II]
The twelve recommendations made were but some of several identified by the team and in fact the findings in the report are not mere consensus items but a compilation of all team driven suggestions/recommendations for change.
1. Based on the JATR team’s observations and findings related to the application of the Changed Product Rule to the certification of the flight control system of the B737 MAX, JATR team members recommend that the FAA work with other civil aviation authorities to revise the harmonized approach to the certification of changed products. Changed Product Rules (e.g., 14 CFR §§ 21.19 & 21.101) and associated guidance (e.g., Advisory Circular 21.101-1B and FAA Orders 8110.4C and 8110.48A) should be revised to require a top-down approach whereby every change is evaluated from an integrated whole aircraft system perspective. These revisions should include criteria for determining when core attributes of an existing transport category aircraft design make it incapable of supporting the safety advancements introduced by the latest regulations and should drive a design change or a need for a new type certificate. The aircraft system includes the aircraft itself with all its subsystems, the flight crew, and the maintenance crew. These Changed Product Rule revisions should take into consideration the following key principles:
• A comprehensive integrated system-level analysis recognizing that in this complex interactive system, every change could interact with other parts of the system.
• The assessment of proposed design changes on existing systems at the aircraft level includes using development assurance principles, system safety principles, and validation & verification techniques. The level of assessment should be proportional to the impact of the change at the aircraft level.
• The consideration of training and qualification of flight and maintenance personnel, as well as detailed explicit procedures for the safe operation of the aircraft. The FAA should update regulations and guidance that are out of date and update certification procedures to ensure that the applied requirements, issue papers, means of compliance, and policies fully address the safety issues related to state-of-the-art designs employed on new projects.
2. Based on the JATR team’s observations and findings related to the regulations, policy, and compliance methods applied to the B737 MAX, JATR team members recommend that the FAA update regulations and guidance that are out of date and update certification procedures to ensure that the applied requirements, issue papers, means of compliance, and policies fully address the safety issues related to state-of-the-art designs employed on new projects. JATR team members also recommend that the FAA review its processes to ensure that regulations and guidance materials are kept up to date.
3. Based on the JATR team’s observations and findings related to the certification of the B737 MAX flight control system and related interfaces, JATR team members recommend that the FAA review the B737 MAX compliance to 14 CFR §§ 25.1329 (Flight Guidance System), 25.1581 (Airplane Flight Manual – General), and 25.201 (Stall Demonstration) and ensure the consistent application and interpretation of regulatory guidance material for the system safety assessment, handling qualities rating method, and conformity requirements for engineering simulators and devices. Should there be a non-compliance, the root cause should be identified and measures implemented to prevent recurrence.
4. Based on the JATR team’s observations and findings related to the FAA type certification process, JATR team members recommend that the FAA review and update the regulatory guidance pertaining to the type certification process with particular emphasis on early FAA involvement to ensure the FAA is aware of all design assumptions, the aircraft design, and all changes to the design in cases where a changed product process is used. The FAA should consider adding feedback paths in the process to ensure that compliance, system safety, and flight deck/human factors aspects are considered for the aircraft design throughout its development and certification.
5. Based on the JATR team’s observations and findings related to FAA’s oversight by the Boeing Aviation Safety Oversight Office (BASOO), JATR team members recommend that the FAA conduct a workforce review of the BASOO engineer staffing level to ensure there is a sufficient number of experienced specialists to adequately perform certification and oversight duties, commensurate with the extent of work being performed by Boeing.
The workforce levels should be such that decisions to retain responsibility for finding compliance are not constrained by a lack of experienced engineers. The FAA should review the Boeing Organization Designation Authorization (ODA) work environment and ODA manual to ensure the Boeing ODA engineering unit members (E-UMs) are working without any undue pressure when they are making decisions on behalf of the FAA. This review should include ensuring the E-UMs have open lines of communication to FAA certification engineers without fear of punitive action or process violation.
6. Based on the JATR team’s observations and findings related to the design process of the flight control system and the related system safety assessments for the B737 MAX, JATR team members recommend that the FAA promote a safety culture that drives a primary focus on the creation of safe products, which in turn comply with certification requirements. Aircraft functions should be assessed, not in an incremental and fragmented manner, but holistically at the aircraft level. System function and performance, including the effects of failures, should be demonstrated and associated assumptions should be challenged to ensure robust designs are realized. The safety analysis process should be integrated with the aircraft development assurance process to ensure all safety requirements and associated assumptions are correct, complete, and verified. The FAA should encourage applicants to have a system safety function that is independent from the design organization, with the authority to impartially assess aircraft safety and influence the aircraft/system design details. Adoption of a safety management system is one way this can be achieved.
7. Based on the JATR team’s observations and findings related to human factors-related issues in the certification process, JATR team members recommend that the FAA integrate and emphasize human factors and human system integration throughout its certification process. Human factors-relevant policies and guidance should be expanded and clarified, and compliance with such regulatory requirements as 14 CFR §§ 25.1302 (Installed Systems and Equipment for Use by the Flightcrew), 25.1309 (Equipment, Systems, and Installations), and 25.1322 (Flightcrew Alerting) should be thoroughly verified and documented. To enable the thorough analysis and verification of compliance, the FAA should expand its aircraft certification resources in human factors and in human system integration.
8. Based on the JATR team’s observations and findings related to the development assurance process applied to the design of the flight control system of the B737 MAX, JATR team members recommend that the FAA ensure applicants apply industry best practice for development assurance, including requirements management, visibility of assumptions, process assurance activities, and configuration management. The FAA should ensure achievement of the close coupling that is required between the applicant safety analysis process and the development assurance process to classify failure conditions and derive the level of rigor of design development and verification. A current example of industry best practice is SAE International’s Aerospace Recommended Practice 4754A (ARP4754A).
The FAA should review and amend Advisory Circular 20-174 to clearly articulate the principles of ARP4754A, promoting industry best practice for development assurance of aircraft and aircraft systems to address applicants’ design trend of increasing integration between aircraft functions and systems.
9. Based on the JATR team’s findings and observations related to the operational design assumptions of crew response applied during the certification process for the flight control system of the B737 MAX, JATR team members recommend that the FAA require the integration of certification and operational functions during the certification process. The FAA should be provided all system differences between related aircraft in order to adequately evaluate operational impact, systems integration, and human performance.
10. Based on the JATR team’s findings and observations related to flight crew training, JATR team members recommend that the FAA require a documented process to determine what information will be included in the Airplane Flight Manual, the Flight Crew Operating Manual, and the Flight Crew Training Manual. The FAA should review training programs to ensure flight crews are competent in the handling of mis-trim events.
11. JATR team members recommend that the FAA conduct a study to determine the adequacy of policy, guidance, and assumptions related to maintenance and ground handling training requirements.
12. JATR team members recommend that the FAA review its policies for analyzing safety risk and implementing interim airworthiness directive action following a fatal transport aircraft accident. The FAA should ensure that it shares post-accident safety information with the international community to the maximum extent possible.
Official air accident investigation report of JT 610 released
In an eventful month for the production of conclusions in respect of Boeing 737 MAX issues, the Indonesian NTSC published its official report into JT610 and found that among the several contributing causes to the tragedy were Boeing’s design and the FAA’s type certification of the MAX.
Contributing factors were identified by NTSC, being “actions, omissions, events, conditions, or a
combination thereof, which, if eliminated, avoided or absent, would have reduced
the probability of the accident or incident occurring, or mitigated the severity of the
consequences of the accident or incident”.
The factors presented were not based on any particular order, nor intended by the NTSC to demonstrate any level of relative contribution. Pursuant to Annex 13 of the Chicago Convention, the Report was not designed to level blame in a legal sense.
The factors identified (as set out at page 215 of the Report) are as follows:
1. During the design and certification of the Boeing 737-8 (MAX),
assumptions were made about flight crew response to malfunctions which,
even though consistent with current industry guidelines, turned out to be
incorrect.
2. Based on the incorrect assumptions about flight crew response and an
incomplete review of associated multiple flight deck effects, MCAS’s
reliance on a single sensor was deemed appropriate and met all certification
requirements.
3. MCAS was designed to rely on a single AOA sensor, making it vulnerable
to erroneous input from that sensor.
4. The absence of guidance on MCAS or more detailed use of trim in the flight
manuals and in flight crew training, made it more difficult for flight crews to
properly respond to uncommanded MCAS.
5. The AOA DISAGREE alert was not correctly enabled during Boeing 737-8
(MAX) development. As a result, it did not appear during flight with the
mis-calibrated AOA sensor, could not be documented by the flight crew and
was therefore not available to help maintenance identify the mis-calibrated
AOA sensor.
6. The replacement AOA sensor that was installed on the accident aircraft had
been mis-calibrated during an earlier repair. This mis-calibration was not
detected during the repair.
7. The investigation could not determine that the installation test of the AOA
sensor was performed properly. The mis-calibration was not detected.
8. Lack of documentation in the aircraft flight and maintenance log about the
continuous stick shaker and use of the Runaway Stabilizer NNC meant that
information was not available to the maintenance crew in Jakarta nor was it
available to the accident crew, making it more difficult for each to take the
appropriate actions.
9. The multiple alerts, repetitive MCAS activations, and distractions related to
numerous ATC communications were not able to be effectively managed.
This was caused by the difficulty of the situation and performance in manual
handling, NNC execution, and flight crew communication, leading to
ineffective CRM application and workload management. These
performances had previously been identified during training and reappeared
during the accident flight.