Commercial Crew Transportation Capability (CCtCap) SpaceX and Boeing Developments

[ecarfan]

Next shoes to drop. NASA’s decision to force Boeing to do another $400M test. What are the odds Boeing requests and is granted more money?

I think it is certain that Boeing will ask for more money to do another uncrewed test flight. And it would be appalling if NASA gave the company a dime. They don’t deserve any special consideration. It is shocking that Boeing was paid $2 BILLION MORE than SpaceX to accomplish exactly the same goal.

 

[mongo]

I think it is certain that Boeing will ask for more money to do another uncrewed test flight. And it would be appalling if NASA gave the company a dime. They don’t deserve any special consideration. It is shocking that Boeing was paid $2 BILLION MORE than SpaceX to accomplish exactly the same goal.

Boeing did have a $400 million line item on their last quarterly report to cover the potential cost of another uncrewed launch. Not to say they won't ask, but at least they don't expect it.

 

[HVM]

Reason why Boeing didn't catch the timing error in the integration test is simple, they didn't do it:

"Boeing didn’t perform full end-to-end test of its astronaut capsule before troubled mission, ‘surprising’ NASA safety panel"
Boeing didn’t perform full end-to-end test of its astronaut capsule before troubled mission, ‘surprising’ NASA safety panel

 

[adiggs]

Reason why Boeing didn't catch the timing error in the integration test is simple, they didn't do it:

"Boeing didn’t perform full end-to-end test of its astronaut capsule before troubled mission, ‘surprising’ NASA safety panel"
Boeing didn’t perform full end-to-end test of its astronaut capsule before troubled mission, ‘surprising’ NASA safety panel

Though I'm not a software engineer, I've been part of software engineering teams for my whole career (DBA and other data related work).

So when I read that, I'm left with ..

uhm. wow. just wow.


Yeah - the Boeing CFO was right to reserve for another flight. That's a much more accurate statement of Boeing's financial situation than the alternative.

From my point of view, the media and news around this can continue talking about this like a test flight is still being decided whether it's needed or not.

If I were the program manager running this project for Boeing, I would just tell NASA that we're not attempting human flight until we repeat the test flight. I'd bet that Boeing can take that decision out of NASA's hands - insist on repeating the test.

 

[Cosmacelf]

If I were the program manager running this project for Boeing, I would just tell NASA that we're not attempting human flight until we repeat the test flight. I'd bet that Boeing can take that decision out of NASA's hands - insist on repeating the test.

Bet you that Boeing doesn’t do this and forces nasa to demand it. That company is run by idiots now.

 

[bmah]

If I were the program manager running this project for Boeing, I would just tell NASA that we're not attempting human flight until we repeat the test flight. I'd bet that Boeing can take that decision out of NASA's hands - insist on repeating the test.

You obviously are more conscientious than the decision makers at Boeing.

Bruce.

 

[e-FTW]

You obviously are more conscientious than the decision makers at Boeing.

Bruce.

They cannot afford a major issue with humans on board.
Maybe this will force their hand; that is a rough year:

 

[adiggs]

They cannot afford a major issue with humans on board.
..

This is really my point. I don't know if it'd be existential for Boeing, but if they don't do a repeat on the test flight and then they have a life threatening or ending problem with their first crewed flight, it's gonna be really bad for the company.

 

[JohnSnowNW]

This is really my point. I don't know if it'd be existential for Boeing, but if they don't do a repeat on the test flight and then they have a life threatening or ending problem with their first crewed flight, it's gonna be really bad for the company.

Wouldn't be great for NASA, either.

 

[bxr140]

So…here's a slightly more pragmatic assessment of the article:

1. As stated, Boeing submitted and NASA approved their verification plan. Boeing did what they said they would do; Boeing did everything NASA required of them. While it is fair to question that verification plan, from a ‘blame’ perspective this one is at a minimum equal if not biased toward NASA, contrary to what is implied in the headline and posts here.

2. The sensationalist headline is, at best, misleading. The ‘surprise’ was that the verification plan was not as comprehensive as panel members expected it would be (technically, one former panel member), not that Boeing “didn’t do” a test.

3. I'm curious as to what exactly is meant by “not tested in a System Integration Lab". Software testing is not as straightforward as it might seem as it is impossible to truly create a ‘test as you fly’ type environment. So…often there’s a ton of detailed V&V that goes on in a software lab and then less extensive checks that happen at next higher assemblies. There’s often simulators involved as well—the rocket gets tested against a space vehicle simulator and vice versa, for instance. In the final system level environment typically the ETE testing objective is to ensure interconnects were installed properly and NOT to exhaustively verify performance limits and nuanced functionalities of the whole shebang. The actual tests performed are a representative aggregate of upstream analysis and testing, and the results are compared to predictions based on that upstream activity for the purpose of ensuring system level results are in-family.

4. Not related to the article, you can bet your expensive Tesla that SpaceX isn’t performing exhaustive system level end-to-end software checks. Editorializing a bit, as a mental exercise consider the response from SpaceX fans had the news been something about how SpaceX streamlined testing and as a result successfully reduced cost and schedule…

 

[adiggs]

Fair points @bxr140. They help me more specifically articulate the thought I had going on in the back of my head.

I agree with your point that they did what NASA expected them to do. I'm not a lawyer (so my opinion is meaningless on this point) - I don't see this as a legal problem for the company. The problem is in the perception and publicity. That'd be true either way for the kind of disaster we're talking about.

The incremental problem, over and above the "standard" problem for the kind of disaster we're talking about, is that for Boeing they've now got a perception that a seemingly basic software integration problem was missed because of a seemingly basic test that wasn't performed. The fact that NASA didn't require the test isn't going to provide perception coverage for Boeing - that's just going to give NASA a black eye too.

I know that testing isn't as straightforward as non-software people might think. My first job at my current company (26 years ago) had me involved in building the compilation and test environment for a significant software project (>100 software engineers - that qualifies in my book at least ). We had nothing like the complexity of software to take a rocket to space. The testing we're talking about isn't easy, and it's not cheap (just that not doing it is way more expensive - that was a lesson we had to learn the old fashioned way ).


So I see the potential problem being an increased likelihood of an unsurvivable publicity problem for the company. Enough larger that Boeing's CFO (I'm sure in coordination with the board, CEO, and other senior management) recognizes the problem sufficiently enough to take a $410M hit as a reserve in the quarterly earnings.

And that's why I say that Boeing should be proactive about this - expand the definition of what goes into the verification test to incorporate more end-to-end testing that AT LEAST creates the illusion that they've done an end-to-end test, so that another failure along the same lines will still be really painful, but won't also make it easy for reasonable people to say out loud in public, with a straight face, that Boeing is taking more money and using that to cut corners that professionals should know not to cut.

Not saying that's fair. I'm saying Boeing needs to make it a lot harder for people to think that.

(They're already taking a lot more money to deliver the same service - that won't change)

 

[SwTslaGrl]

It seems to me with so many different payloads on the Atlas booster they would have had a routine for integration of Starliner too?

 

[bxr140]

It seems to me with so many different payloads on the Atlas booster they would have had a routine for integration of Starliner too?

The ‘routine’ is very minimal for most payloads on most launchers. There’s literally no interaction between the launcher and the payload itself during the mission for most things. Any launch site “testing” activity between the two are simple interconnect checks to verify the the launcher umbilical works as designed, and it is literally just a copper wire pass through so the checks are pretty straightforward.

There’s no “testing” a typical payload needs to do once its strapped onto a rocket. There are so many gates and reviews leading up to integrating a payload onto a rocket that it would never be up there in the first place if it wasn’t ready to go. It is really boring once a typical payload is on a launcher—they’re integrated a week or more before launch and pretty much the only thing that happens after integration is battery charging and some yawner rehearsals.

I suspect all that may be a major element in Boeing’s failings: A crewed payload, for various and sometimes self-evident reasons, requires more care and feeding than a dumb satellite mission. It is plausible Boeing tried to heavily leverage their ‘regular’ mission knowledge base and in doing so missed some important steps.

 

[ICUDoc]

This is really my point. I don't know if it'd be existential for Boeing, but if they don't do a repeat on the test flight and then they have a life threatening or ending problem with their first crewed flight, it's gonna be really bad for the company.

Wouldn't be great for NASA, either.

Or the crew.

 

[Grendal]

Not to pile on too much but I wonder how much of the "cost plus" philosophy and attitude has filtered down into the actions of the company. Fundamentally, under a cost plus system (which is every other program in Boeing space) it is beneficial to have problems crop up. That delays the project and gets you another year and more money. This is pure speculation on my part. It's just the number of issues that were allowed to creep in comes across as a systemic issue. So I'm trying to understand what could cause this.

The other possibility that comes to mind is that the directly competitive nature of this program may have allowed some errors to creep in that were missed to get things done.

 

[dpodoll]

Might be good for America for Boeing to pull the plug on the whole effort.

 

[Nikxice]

you can bet your expensive Tesla that SpaceX isn’t performing exhaustive system level end-to-end software checks.

I'm in the weeds on this, so gotta ask why this testing wouldn't be happening at SpaceX? Maybe I'm on the wrong track, but that kind of oversight appears to be what bit Boeing in the beehind. A recent WP article quotes Boeing Starliner program manager John Mulholland about their OFT clock anomaly.

On the timing issue, he said the clock on the spacecraft was pulling its time from the rocket. During tests of the software in the laboratory, the crews were primarily concerned with making sure the two vehicles were communicating correctly. The testing team proved there were no communication issues, but it cut the test short so it never uncovered that the spacecraft was reading the wrong time. “Unfortunately, the run was stopped after we separated from the launch vehicle,” he said. If the test had continued, “we would have caught it.”
https://www.washingtonpost.com/technology/2020/02/28/boeing-admits-starliner-testing-flaws/

 

[bxr140]

I'm in the weeds on this, so gotta ask why this testing wouldn't be happening at SpaceX?

To clarify, that was general commentary on what appeared to be a general assumption--within this thread and to a lesser degree the article--that Boeing were fools to not do exhaustive system level software tests in flight or near flight configuration. It was not meant to be commentary on this specific test or whether SpaceX elected to perform something equivalent.

There's plenty of room for valid scrutiny of Boeing's decision/oversight to not do what was described/suggested in the article you quoted; there's plenty of room to speculate whether SpaceX performed some equivalent activity in a software lab.

 

[Cosmacelf]

It is much more likely that SpaceX does extensive end to end tests since they write all software in house and are much more vertically integrated compared to Boeing.

My memory isn't perfect, but I think that if you looked at all SpaceX anomalies, you'd find that none of them were attributable to software. They were hardware related or some physical process.

Also, the thing that is being forgotten about here (at least in the press) is that it wasn't ONLY a couple of software bugs and inadequate testing. They also didn't have appropriate telemetry telling ground control the state of the on board software. If I were designing a rocket, I'd have a ton of telemetry, not only for hardware sensors, but also "software sensors', aka logs, that would be reviewed in real time by either humans or other ground based software. And finally, they completely lost telemetry for quite a period with the rocket. Isn't that also bad engineering?

 

[miimura]

And finally, they completely lost telemetry for quite a period with the rocket. Isn't that also bad engineering?

I think that losing telemetry is not bad engineering, it's a reality of what's available for telemetry communication from the ground. The bad engineering is not planning how to handle those known times when you won't have real time telemetry.