Planes, trains and third-party risks — a tale of two IT-related shutdowns

Christmas Eve (and Christmas Day) are arguably the most important time-frame for transportation companies. So it was a big deal when an American Airlines system glitch forced the airline to ask the government for a full shutdown on Christmas Eve. And it was an even bigger deal the next day for Bane NOR, which runs the Norwegian rail system and had to shut down all trains in Norway.

Both involved IT issues and both were mostly — if not entirely — caused by third-party firms. Now, third-party risks are nothing new. But few CIOs truly internalize that one error from a vendor can shut down all enterprise operations. That’s a lot of trust to offer an outside company that typically undergoes minor due diligence, assuming it was subjected to any meaningful due diligence at all.

What happened with these Christmas nightmares? Let’s drill into each and note how the two transportation giants differed in their approach.

The more interesting of the two was the Norwegian train shutdown, which lasted 13 hours on Christmas Day, from roughly 8 a.m. until 9 p.m. The problem: trains couldn’t communicate with any traffic control centers, which meant they couldn’t operate safely. The cause: a bad firewall setting.

Let that sink in. Because systems today overwhelmingly run through the internet, firewalls can and will block anything. Until this incident, how many IT managers at Bane NOR realized a firewall setting could shut down every train everywhere?

That was a key reason for the long delay in getting the trains back online. When communications stop, managers think the communications gear is somehow failing.

“It took us a while before we could trace it to a firewall issue. It was not one of the obvious causes to look at,” Strachan Stine Smemo, the Bane external communications manager, said in an email to Computerworld. “It was tricky to find the problem.”

Bane’s team opted against changing any firewall settings and instead — as a temporary measure — switched communications to a different firewall. (They later changed the impacted components, Smemo said.)

Arild Nybrodahl, Bane’s information and communications technology director, said his team detected “system instability” on Christmas Eve, which is when “troubleshooting efforts were initiated.” Things didn’t get bad enough to shutdown operations until 8 a.m. the next day, he said. 

“The fault affected the railway’s closed mobile network (GSM-R) and other critical communication systems,” Nybrodahl said. “When any emergency calls and other communication between the train and the train conductor do not work, we cannot operate trains. We have located where the error lies in our own nationwide IT infrastructure and we are now working on a solution to correct the error. We have not yet corrected the root cause, but have taken measures so that the part of the network where the error was located is isolated from the rest of the infrastructure.”

Unlike American Airlines, Bane did not identify the relevant third-party and even praised that vendor’s efforts. Bane received “good help from our supplier,” Smemo said. 

American Airlines, however, not only identified the vendor at issue as DXC, but went out of its way to tell reporters that the problems it ran into were that vendor’s fault. This is known as throwing a partner under the bus.

It’s not clear precisely what happened between the two companies, as neither have discussed the particulars. But American made those comments shortly after the one-hour outage ended. That means emotions were at play, and someone at at the airline was very unhappy.

(DXC is likely unhappy, too, since its stock price has taken a hit.)

Though DXC has been a longtime supplier to American — the DXC website says “more than 20 years” — but it’s not precisely clear what role it had in the shutdown. The company has some role in the airline’s flight operations systems and has been working to modernize American’s systems, including moving legacy code to the cloud. 

The airline blamed a network hardware issue, without being specific, that forced the airline to ask the US Federal Aviation Administration for a nationwide group stop that ended up lasting about an hour.

According to a report on MSN , the incident delayed more than 900 flights affecting “around 900,000 passengers across 200 US airports, leaving many stranded and sleeping in terminals.”

Given that both of these incidents happened on major holidays, one obvious factor is that the companies had only skeleton crews on duty. Though it’s unlikely that holiday staffing caused either situation, it likely slowed down the responses.

One other wrinkle in the DXC situation: the company on Christmas Eve was already in the middle of an IT leadership change. CIO Kristie Grinnell had given notice about her move to a new job as CIO of TD SYNNEX. That was announced on Dec. 19; two weeks later DXC announced its new CIO would be Brad Novak. 

The problem with throwing a vendor partner under the bus — aside from the fact you haven’t done a full investigation or determined who’s at fault —is that it leaves important questions unanswered. Did this third-party firm have the appropriate skills and personnel to deliver what it was supposed to deliver? If not, then shouldn’t the fault lie with whoever hired that firm?

Let’s say the selection process was appropriate. The question then becomes, “Who was supposed to oversee that vendor?” And was the vendor given everything needed to do the job?

From the perspective of shareholders, the fault is more often going to lie with the people who overseeing and bringing in the outside firm. Unless the third-party company ignored instructions or engaged in bad behavior, most mishaps are going to be blamed on the enterprise.

Put bluntly, an enterprise that is quick to blame a contractor is likely trying to change the subject before its own failings are examined.

Source:: Computer World