Pinboard (jm)
https://pinboard.in/u:jm/public/
recent bookmarks from jmBoeing's 737 Max Software Outsourced to $9-an-Hour Engineers2024-03-13T12:55:17+00:00
https://www.industryweek.com/supply-chain/article/22027840/boeings-737-max-software-outsourced-to-9-an-hour-engineers
jmIt remains the mystery at the heart of Boeing Co.’s 737 Max crisis: how a company renowned for meticulous design made seemingly basic software mistakes leading to a pair of deadly crashes. Longtime Boeing engineers say the effort was complicated by a push to outsource work to lower-paid contractors.
The Max softwareーplagued by issues that could keep the planes grounded months longer after U.S. regulators this week revealed a new flawーwas developed at a time Boeing was laying off experienced engineers and pressing suppliers to cut costs.
Increasingly, the iconic American planemaker and its subcontractors have relied on temporary workers making as little as $9 an hour to develop and test software, often from countries lacking a deep background in aerospace ー notably India. [..]
Rabin, the former software engineer, recalled one manager saying at an all-hands meeting that Boeing didn’t need senior engineers because its products were mature. “I was shocked that in a room full of a couple hundred mostly senior engineers we were being told that we weren’t needed,” said Rabin, who was laid off in 2015. [..]
At a meeting with a chief 787 engineer in 2008, one staffer complained about sending drawings back to a team in Russia 18 times before they understood that the smoke detectors needed to be connected to the electrical system... [...]
During the crashes of Lion Air and Ethiopian Airlines planes that killed 346 people, investigators suspect, the MCAS system pushed the planes into uncontrollable dives because of bad data from a single sensor. That design violated basic principles of redundancy for generations of Boeing engineers, and the company apparently never tested to see how the software would respond, Lemme said. “It was a stunning fail,” he said. “A lot of people should have thought of this problem – not one person – and asked about it.”
]]>boeing fail outsourcing hcl safety software engineeringhttps://pinboard.in/https://pinboard.in/u:jm/b:4dbc9a4597e3/"In Boeing production speak, this is a “process failure”. For an A&P mechanic at an airline, this would be called “federal crime”."2024-01-24T11:23:43+00:00
https://leehamnews.com/2024/01/15/unplanned-removal-installation-inspection-procedure-at-boeing/#comment-509962
jm
Current Boeing employee here – I will save you waiting two years for the NTSB report to come out and give it to you for free: the reason the door blew off is stated in black and white in Boeings own records. It is also very, very stupid and speaks volumes about the quality culture at certain portions of the business.
A couple of things to cover before we begin:
Q1) Why should we believe you?
A) You shouldn’t, I’m some random throwaway account, do your own due diligence. Others who work at Boeing can verify what I say is true, but all I ask is you consider the following based on its own merits.
Q2) Why are you doing this?
A) Because there are many cultures at Boeing, and while the executive culture may be throughly compromised since we were bought by McD, there are many other people who still push for a quality product with cutting edge design. My hope is that this is the wake up call that finally forces the Board to take decisive action, and remove the executives that are resisting the necessary cultural changes to return to a company that values safety and quality above schedule.
With that out of the way… why did the left hand (LH) mid-exit door plug blow off of the 737-9 registered as N704AL? Simple- as has been covered in a number of articles and videos across aviation channels, there are 4 bolts that prevent the mid-exit door plug from sliding up off of the door stop fittings that take the actual pressurization loads in flight, and these 4 bolts were not installed when Boeing delivered the airplane, our own records reflect this.
The mid-exit doors on a 737-9 of both the regular and plug variety come from Spirit already installed in what is supposed to be the final configuration and in the Renton factory, there is a job for the doors team to verify this “final” install and rigging meets drawing requirements. In a healthy production system, this would be a “belt and suspenders” sort of check, but the 737 production system is quite far from healthy, its a rambling, shambling, disaster waiting to happen. As a result, this check job that should find minimal defects has in the past 365 calendar days recorded 392 nonconforming findings on 737 mid fuselage door installations (so both actual doors for the high density configs, and plugs like the one that blew out). That is a hideously high and very alarming number, and if our quality system on 737 was healthy, it would have stopped the line and driven the issue back to supplier after the first few instances. Obviously, this did not happen. Now, on the incident aircraft this check job was completed on 31 August 2023, and did turn up discrepancies, but on the RH side door, not the LH that actually failed. I could blame the team for missing certain details, but given the enormous volume of defects they were already finding and fixing, it was inevitable something would slip through- and on the incident aircraft something did. I know what you are thinking at this point, but grab some popcorn because there is a plot twist coming up.
The next day on 1 September 2023 a different team (remember 737s flow through the factory quite quickly, 24 hours completely changes who is working on the plane) wrote up a finding for damaged and improperly installed rivets on the LH mid-exit door of the incident aircraft.
A brief aside to explain two of the record systems Boeing uses in production. The first is a program called CMES which stands for something boring and unimportant but what is important is that CMES is the sole authoritative repository for airplane build records (except on 787 which uses a different program). If a build record in CMES says something was built, inspected, and stamped in accordance with the drawing, then the airplane damn well better be per drawing. The second is a program called SAT, which also stands for something boring and unimportant but what is important is that SAT is *not* an authoritative records system, its a bullentin board where various things affecting the airplane build get posted about and updated with resolutions. You can think of it sort of like a idiots version of Slack or something. Wise readers will already be shuddering and wondering how many consultants were involved, because, yes SAT is a *management visibilty tool*. Like any good management visibilty tool, SAT can generate metrics, lots of metrics, and oh God do Boeing managers love their metrics. As a result, SAT postings are the primary topic of discussion at most daily status meetings, and the whole system is perceived as being extremely important despite, I reiterate, it holding no actual authority at all.
We now return to our incident aircraft, which was written up for having defective rivets on the LH mid-exit door. Now as is standard practice kn Renton (but not to my knowledge in Everett on wide bodies) this write-up happened in two forms, one in CMES, which is the correct venue, and once in SAT to “coordinate the response” but really as a behind-covering measure so the manager of the team that wrote it can show his boss he’s shoved the problem onto someone else. Because there are so many problems with the Spirit build in the 737, Spirit has teams on site in Renton performing warranty work for all of their shoddy quality, and this SAT promptly gets shunted into their queue as a warranty item. Lots of bickering ensues in the SAT messages, and it takes a bit for Spirit to get to the work package. Once they have finished, they send it back to a Boeing QA for final acceptance, but then Malicious Stupid Happens! The Boeing QA writes another record in CMES (again, the correct venue) stating (with pictures) that Spirit has not actually reworked the discrepant rivets, they *just painted over the defects*. In Boeing production speak, this is a “process failure”. For an A&P mechanic at an airline, this would be called “federal crime”.
Presented with evidence of their malfeasance, Spirit reopens the package and admits that not only did they not rework the rivets properly, there is a damaged pressure seal they need to replace (who damaged it, and when it was damaged is not clear to me). The big deal with this seal, at least according to frantic SAT postings, is the part is not on hand, and will need to be ordered, which is going to impact schedule, and (reading between the lines here) Management is Not Happy.
However, more critical for purposes of the accident investigation, the pressure seal is unsurprisingly sandwiched between the plug and the fuselage, and you cannot replace it without opening the door plug to gain access. All of this conversation is documented in increasingly aggressive posts in the SAT, but finally we get to the damning entry which reads something along the lines of “coordinating with the doors team to determine if the door will have to be removed entirely, or just opened. If it is removed then a Removal will have to be written.” Note: a Removal is a type of record in CMES that requires formal sign off from QA that the airplane been restored to drawing requirements.
If you have been paying attention to this situation closely, you may be able to spot the critical error: regardless of whether the door is simply opened or removed entirely, the 4 retaining bolts that keep it from sliding off of the door stops have to be pulled out. A removal should be written in either case for QA to verify install, but as it turns out, someone (exactly who will be a fun question for investigators) decides that the door only needs to be opened, and no formal Removal is generated in CMES (the reason for which is unclear, and a major process failure). Therefore, in the official build records of the airplane, a pressure seal that cannot be accessed without opening the door (and thereby removing retaining bolts) is documented as being replaced, but the door is never officially opened and thus no QA inspection is required.
This entire sequence is documented in the SAT, and the nonconformance records in CMES address the damaged rivets and pressure seal, but at no point is the verification job reopened, or is any record of removed retention bolts created, despite it this being a physical impossibility. Finally with Spirit completing their work to Boeing QAs satisfaction, the two rivet-related records in CMES are stamped complete, and the SAT closed on 19 September 2023. No record or comment regarding the retention bolts is made.
I told you it was stupid.
So, where are the bolts? Probably sitting forgotten and unlabeled (because there is no formal record number to label them with) on a work-in-progress bench, unless someone already tossed them in the scrap bin to tidy up.
There’s lots more to be said about the culture that enabled this to happened, but thats the basic details of what happened, the NTSB report will say it in more elegant terms in a few years.
]]>737max aviation boeing comments throwaway fail qa bolts ntsbhttps://pinboard.in/https://pinboard.in/u:jm/b:7d69f3234f00/Internal FAA review envisaged one fatal crash every 2-3 years with 737-MAX2019-12-11T13:50:27+00:00
https://www.wsj.com/articles/internal-faa-review-saw-high-risk-of-737-max-crashes-11576069202
jmU.S. regulators decided to allow the [Boeing] 737 MAX jet to keep flying after its first fatal crash last fall, despite their own analysis [...] The November 2018 internal Federal Aviation Administration analysis, expected to be released during a House committee hearing Wednesday, reveals that without agency intervention, the MAX could have averaged one fatal crash about every two or three years, according to industry officials and regulators.
]]>faa fail regulation us-politics boeing safety 737max flying accidentshttps://pinboard.in/https://pinboard.in/u:jm/b:05ffdcdda3b2/John Barnett on Why He Won’t Fly on a Boeing 787 Dreamliner2019-12-03T10:51:13+00:00
https://www.corporatecrimereporter.com/news/200/john-barnett-on-why-he-wont-fly-on-a-boeing-787-dreamliner/
jm“When I worked on the 747, the 767, the 777 in Everett, those are beautiful planes. And the people there fully understood what it took to build a safe and airworthy aircraft. I hate to throw the entire label over the whole product line. But as far as the 787, I would change flights before I would fly a 787. I’ve told my family — please don’t fly a 787. Fly something else. Try to get a different ticket. I want the people to know what they are riding on.”
]]>business flight flying safety boeing danger 787 john-barnett whistleblowershttps://pinboard.in/https://pinboard.in/u:jm/b:42ae4123e4bb/Crash Course | The New Republic2019-09-23T11:50:52+00:00
https://newrepublic.com/article/154944/boeing-737-max-investigation-indonesia-lion-air-ethiopian-airlines-managerial-revolution
jm[Boeing] engineers devised a software fix called MCAS, which pushed the nose down in response to an obscure set of circumstances in conjunction with the “speed trim system,” which Boeing had devised in the 1980s to smooth takeoffs. Once the 737 MAX materialized as a real-life plane about four years later, however, test pilots discovered new realms in which the plane was more stall-prone than its predecessors. So Boeing modified MCAS to turn down the nose of the plane whenever an angle-of-attack (AOA) sensor detected a stall, regardless of the speed. That involved giving the system more power and removing a safeguard, but not, in any formal or genuine way, running its modifications by the FAA, which might have had reservations with two critical traits of the revamped system: Firstly, that there are two AOA sensors on a 737, but only one, fatefully, was programmed to trigger MCAS. The former Boeing engineer Ludtke and an anonymous whistle-blower interviewed by 60 Minutes Australia both have a simple explanation for this: Any program coded to take data from both sensors would have had to account for the possibility the sensors might disagree with each other and devise a contingency for reconciling the mixed signals. Whatever that contingency, it would have involved some kind of cockpit alert, which would in turn have required additional training—probably not level-D training, but no one wanted to risk that. So the system was programmed to turn the nose down at the feedback of a single (and somewhat flimsy) sensor. And, for still unknown and truly mysterious reasons, it was programmed to nosedive again five seconds later, and again five seconds after that, over and over ad literal nauseam.
And then, just for good measure, a Boeing technical pilot emailed the FAA and casually asked that the reference to the software be deleted from the pilot manual.
So no more than a handful of people in the world knew MCAS even existed before it became infamous. Here, a generation after Boeing’s initial lurch into financialization, was the entirely predictable outcome of the byzantine process by which investment capital becomes completely abstracted from basic protocols of production and oversight: a flight-correction system that was essentially jerry-built to crash a plane. “If you’re looking for an example of late stage capitalism or whatever you want to call it,” said longtime aerospace consultant Richard Aboulafia, “it’s a pretty good one.”
]]>boeing business capitalism engineering management fail disasters automation cost-control stock-market fly-by-wirehttps://pinboard.in/https://pinboard.in/u:jm/b:147b64ad3d93/The many human errors that brought down the Boeing 737 Max - The Verge2019-05-02T14:12:33+00:00
https://www.theverge.com/2019/5/2/18518176/boeing-737-max-crash-problems-human-error-mcas-faa
jmHad anyone [at the FAA] checked, they might have flagged MCAS for one of several reasons, including its lack of redundancy, its unacceptably high risk of failure, or its significant increase in power to the point that it was no longer just a “hazardous failure” kind of system.
When asked for comment, the agency said, “The FAA’s aircraft certification processes are well established and have consistently produced safe aircraft designs.”
Boeing defended the process as well. “The system of authorized representatives — delegated authority — is a robust and effective way for the FAA to execute its oversight of safety,” a spokesperson told The Verge.
But that system only works when someone actually reads the paperwork.
]]>mcas boeing 737max fail safety faa flying regulationhttps://pinboard.in/https://pinboard.in/u:jm/b:82291b43c7a6/Pilots had 40 seconds to fix error in tests of Boeing 737 Max flight2019-03-26T15:51:44+00:00
https://www.irishtimes.com/business/transport-and-tourism/pilots-had-40-seconds-to-fix-error-in-tests-of-boeing-737-max-flight-1.3838837
jmDuring flight simulations recreating the problems with the doomed Lion Air plane, pilots discovered that they had less than 40 seconds to override an automated system on Boeing’s new jets and avert disaster.
The pilots tested a crisis situation similar to what investigators suspect went wrong in the Lion Air crash in Indonesia last fall. In the tests, a single sensor failed, triggering software designed to help prevent a stall.
Once that happened, the pilots had just moments to disengage the system and avoid an unrecoverable nose dive of the Boeing 737 Max, according to two people involved in the testing in recent days.
]]>boeing 737max mcas fail planes safety disastershttps://pinboard.in/https://pinboard.in/u:jm/b:1584c5c36ede/Flawed analysis, failed oversight: How Boeing, FAA certified the suspect 737 MAX flight control system2019-03-18T21:21:11+00:00
https://www.seattletimes.com/business/boeing-aerospace/failed-certification-faa-missed-safety-issues-in-the-737-max-system-implicated-in-the-lion-air-crash/
jmLike all 737s, the MAX actually has two of the sensors, one on each side of the fuselage near the cockpit. But the MCAS was designed to take a reading from only one of them.
Lemme said Boeing could have designed the system to compare the readings from the two vanes, which would have indicated if one of them was way off. Alternatively, the system could have been designed to check that the angle-of-attack reading was accurate while the plane was taxiing on the ground before takeoff, when the angle of attack should read zero.
“They could have designed a two-channel system. Or they could have tested the value of angle of attack on the ground,” said Lemme. “I don’t know why they didn’t.”
The black box data provided in the preliminary investigation report shows that readings from the two sensors differed by some 20 degrees not only throughout the flight but also while the airplane taxied on the ground before takeoff.
]]>faa aviation boeing 737max safety fail sensors flight crashes mcashttps://pinboard.in/https://pinboard.in/u:jm/b:3452ca238a6b/Several Boeing 737 Max 8 pilots in U.S. complained about suspected safety flaw | Airlines | Dallas News2019-03-13T11:04:36+00:00
https://www.dallasnews.com/business/airlines/2019/03/12/boeing-737-max-8-pilots-complained-feds-months-suspected-safety-flaw
jmboeing planes safety autopilots 737maxhttps://pinboard.in/https://pinboard.in/u:jm/b:ee40c518866f/AMERICAN AIRLINES 737MAX8: “LIKE A FLYING PRISON”2017-12-05T11:14:40+00:00
https://aviationnews.online/2017/12/04/american-airlines-737max8-like-a-flying-prison/
jmcoach travel aa airlines 737 boeing reviews comforthttps://pinboard.in/https://pinboard.in/u:jm/b:6ded9be380be/The Titanium Gambit | History | Air & Space Magazine2015-07-22T16:30:26+00:00
http://www.airspacemag.com/history-of-flight/the-titanium-gambit-3804526/?all&no-ist
jmvia:maciej titanium history cold-war detente ussr usa boeing russia aerospacehttps://pinboard.in/https://pinboard.in/u:jm/b:183fdfbdd66f/SCADA systems online, and a horror story about a non-airgapped Boeing 747 engine management system2015-04-17T20:28:31+00:00
http://www.infosecisland.com/blogview/16696-FACT-CHECK-SCADA-Systems-Are-Online-Now.html
jm747's are big flying Unix hosts. At the time, the engine management system on this particular airline was Solaris based. The patching was well behind and they used telnet as SSH broke the menus and the budget did not extend to fixing this. The engineers could actually access the engine management system of a 747 in route. If issues are noted, they can re-tune the engine in air.
The issue here is that all that separated the engine control systems and the open network was NAT based filters. There were (and as far as I know this is true today), no extrusion controls. They filter incoming traffic, but all outgoing traffic is allowed.
(via Paddy Benson)]]>air-gap planes boeing security 747 solaris unixhttps://pinboard.in/https://pinboard.in/u:jm/b:45a47551fb9b/HACKERS COULD COMMANDEER NEW PLANES THROUGH PASSENGER WI-FI2015-04-17T09:20:22+00:00
http://www.wired.com/2015/04/hackers-commandeer-new-planes-passenger-wi-fi/
jmBoeing 787 Dreamliner jets, as well as Airbus A350 and A380 aircraft, have Wi-Fi passenger networks that use the same network as the avionics systems of the planes
What the fucking fuck. Air-gap or gtfo
]]>air-gap security planes boeing a380 a350 dreamliner networking firewalls avionicshttps://pinboard.in/https://pinboard.in/u:jm/b:080b50a06cc4/satellite rescue abandoned due to patents2012-05-22T12:55:25+00:00
http://en.wikipedia.org/wiki/AMC-14#Launch_anomaly
jmboeing space patenting via:hn funny sad lockheed-martin ses amc-14 business-process patentshttps://pinboard.in/https://pinboard.in/u:jm/b:fb802a98e2d3/747s using VLANs to secure in-flight access to engine management systems2011-11-26T09:19:29+00:00
https://plus.google.com/u/0/110897184785831382163/posts/5qsNxFEaiML
jmscary aviation flight security boeing 747 via:riskshttps://pinboard.in/https://pinboard.in/u:jm/b:84940f0c3c84/