Global Airline Outage – One Vendor Chaos

swahuffpostoutage

Vendor and supplier management is amongst the most boring topics to explain to your spouse, in-laws, and friends when they ask “What exactly is it that you DO for a living?”

It is always great to have a real life example to help. Today’s apparent failure causing chaos nationwide is PERFECT. Many birds, one stone.

A company, Amadeus, has successfully innovated and created a platform for airline operations that many airlines have adopted. Not surprising actually since user friendly and airlines have rarely been expressed in the same sentence.

Apparently this morning there was a problem. A large one. One that affected most/all airlines using the system. Reservations, kiosks, gate agent operations, et al.

In June 2016 I snapped a photo of Amadeus in action at AUS. It was in stark contrast to the text screens that agents used in decades past. Note the ability for multiple airlines to share one gate. Awesome. If it works. ALL the time.

Questions

  • Disaster Recovery? – Amadeus? Airlines? Why are passengers affected when a single vendor fails?
  • Regulators – Is it acceptable for a single vendor to be allowed to substantively disrupt airport operations globally?
  • Attackers – Want to inflict max chaos and disruption to critical infrastructure? A single vendor may all you need to target.
  • Protectors – Do you trust your vendors? Based on what evidence?

Fiber Cut? No Internet, Mobile or SMS? Are You SOL?

I woke up this morning thinking about the increasing impact that any disruption between us and the Internet has in our daily personal and professional lives. I love the security profession but sometimes it drives your brain to an offroad or two (or more) that many people do not oft travel.

Think. What use is your smartphone, computer or tablet if it loses all ability to communicate with anyone or anything else?

When fiber is cut and Internet and phone service are down it often affects an entire community or region. If this disruption occurs inline with a disaster, either natural or terror, lives can be at risk.

Fortunately most fiber cuts to date are accidental the result of an errant backhoe or other less than nefarious cause.

This is changing. Intentional cuts in the Bay Area in 2015 and allegations from Verizon that striking workers intentionally cut fiber are troublesome indicators that our fiber optic networks will increasingly be a target for those with a desire to disrupt.

But what is old is new again. The first recorded telecommunication sabotage took place during the second battle of Bull Run in 1862.  I would argue the stakes are no higher today just different.

Impact

The business impact of a fiber cut can be measured quantitatively and qualitatively. I dare you to go brick and mortar shopping in a region experiencing a fiber cut. You will quickly learn which retailers have the most resilient and effective disaster recovery and business continuity efforts.

Most retailers rely on fiber for every connection they make at the point of sale or otherwise. The fallback is normally satellite and works much more slowly, if it works at all. And you thought the lady ahead of you in line at Wal-Mart buying 500 cat key chains insisting on 50 separate receipts was slow.

The inability for teenagers to reach their friends via Facebook, Snapchat, SMS, phone or any other means other than face to face may seem in the moment catastrophic but is in reality only a symptom pointing to a future where the stakes are much higher.

Today Alexa’s inability to respond from the mothership to turn off lights and tell dad jokes arguably worse than my own is but a minor annoyance.

Future Alexa controlling my in home medical devices, fire suppression systems, and life safety equipment sets the stage for a future where being always connected is as critical as having water, power and oxygen to breathe.

Solutions

Ensure you have multiple Internet connections over disparate paths. Businesses in mission critical industries do this as a normal course of business. I recommend small businesses and families do as well.

My small business maintains two Internet connections and a satellite backup. Keep in mind fiber cuts often render all land based communication useless. Maintaining satellite Internet as a backup is a relatively cheap insurance policy. We use Exede.

Invest in a SOHO router that manages multiple Internet connections and provides for automatic failover/failback. My preference is CradlePoint.

Invest in an out-of-band communication technology to ensure that fiber cuts or other outages do not prevent you from reaching your family or business associates.

Not fully baked but amongst the most promising and exciting innovation for communications not reliant on Internet or even mobile coverage are these two companies. Beartooth  and GoTenna.

Both systems utilize a combination of your smartphone and a built in 900 mhz unlicensed radio frequencies to allow communication over several miles with no dependency other than a similar device on the other end.

Although GoTenna appears more consumer friendly and geared towards the social, crowdsourced model they do purport to have a mission critical “professional” line in the works. I’ve ordered a pair of GoTenna devices and will be posting a review after some time assessing their merits and limitations.

Technology solutions aside the most important action you can take as a business, family or individual is to have a plan and TEST the plan regularly.  Many great resources to assist with this over at ready.gov

As always feel free to reach out to me directly via LinkedIn if you would like more information about this topic or any other.

Stuart Clark

What can I do to help after a disaster? CERT may be for you.

The recent floods that devastated the Central Texas community of Wimberley in May left many asking themselves this question.  Becoming a CERT (Community Emergency Response Team) member will help prepare you and your family for a disaster and provide basic disaster training to assist others in the community.
CERT is coordinated by FEMA but led locally.  As of this writing there are over 2200 CERT teams nationwide.  You must be at least 18 years of age however local coordinators have the discretion to allow under 18 to participate.

I joined Austin CERT in 2011 by attending evening courses over a period of several weeks.  The training was held in the Austin/Travis County EOC which for a technology geek like me was worth the time investment alone.  The training class itself was about 35 strong and a diverse mix of young and old alike.  The training was interesting and engaging even for a seasoned first responder like myself.  For reference the current Austin CERT course description is here.

Working in law enforcement as both police officer and dispacher offered me ample opportunities to help people utterly unprepared, oblivious or even obstinate and hostile.  An informed and educated public is the greatest asset to both the first responder and themselves in time of disasters. Preparation prevents panic.

A photo taken during a 2011 Austin CERT Disaster Training Exercise