A History Of Service Disruptions

Published Date: 02 Nov 2017

Introduction

Since September 2007, BlackBerry Internet services have been significantly disrupted at least five times. The most serious incident occurred in 2011 when subscribers across the globe were left completely without services for four days. The debacle ultimately presaged the exit of the CEO and a number of senior executives from the company. For many, the outage reinforced the growing perceptions that this was a company that was on its knees.

A History of Service Disruptions

BlackBerry’s rapid growth in subscribers and the build-up of its network infrastructure from 2004 to 2010 increased the need for frequent system upgrades and maintenance. It was primarily through botched upgrades, further compounded by faulty back-ups, and poor disaster recovery execution that caused many outages to occur and prolong into damaging ordeals for the company.

While BlackBerry managed to weather the storm back in 2011 it is clear that another reoccurrence of that scale would deliver a very serious setback to the company as its attempts to rebuild its brand and claw its way back to relevancy in the smart phone market.

Service outages therefore represent a business risk exposure for the company. The frequency of service disruptions, which have occurred almost yearly since 2007, poses serious concerns which the company can very well ill afford.

In 2011 the outage came at a particularly bad time for Blackberry, as the company was beginning to buckle under severe competition in the market from its rivals. Apple was gearing up for the launch of the iPhone 4 at the time.

The negative publicity from the incident significantly damaged the company’s reputation and credibility, causing many corporate clients and consumers to question the benefit of maintaining their loyalty and tolerating the inconvenience from the occasional blackout versus switching their allegiance to another maker.

The unconvincing media appearances by the embattled founder and then CEO Mike Lazaridis didn’t help. His efforts, trying to reassure an increasingly irate public that the company was doing its best to isolate resolve the problem a full three days into it was seen as a public relations fiasco. The company was silent for the first two days. The company was portrayed as incompetent and unreliable.

The tech blog Electronista wrote: "RIM's outage is now one of its largest in recent memory and is now edging even closer to the iPhone 4S launch than before, leading to a possible temptation for those already looking to upgrade their phones. Commentary on Twitter has shifted gradually from frustrated patience to open anger and has led some to remark that they're now likely to switch to the iPhone, Android, or another platform."

The company declared that it would leave no stone unturned to make sure that this does not repeat.

Root Causes

BlackBerry typically doesn’t divulge details regarding the specific causes of outages. With earlier outages in 2007 and 2008 the company said the problem was triggered by a software upgrade that was designed to provide faster access to data. Instead, the upgrade unexpectedly interfered with communication between the system's database and its data storage, so information couldn't be transferred to individual users.

To compound the problem, RIM's backup system also failed. The double failure led to a huge backlog of e-mails. While significant as standalone events in themselves, the earlier failures garnered nowhere near the global publicity as the 2011 outage did, simply because the Blackberry device numbers were not as widespread back then.

In the case of 2011 the culprit was a faulty Cisco network core switch. The BlackBerry system was designed to jump to a backup switch but the failover system "did not function as previously tested," according to a statement that was issued by the company. Many people wondered why the company couldn’t figure out the root cause what was just a faulty switch.

Analyst’s verdict was damning. "RIM (BlackBerry) has failed again at what plagued them in past outages, which is to provide a comprehensive disaster recovery solution," according to Ken Dulaney, an analyst at Gartner. He went on to say that ‘While it's true that switches can fail, there should be automatic ways in which the system recovers from this type of event," Adding "Any vendor who runs this type of mission-critical service must constantly be reviewing disaster recovery solutions."

While the company has managed to avoid any outage of that scale in 2012, further service disruptions occurred in different regions in the past 6 months causing concerns ahead of the BlackBerry 10 launch.

In January 2013 subscribers connected to the company’s BlackBerry Internet Services (BIS) across Europe, the Middle East and Africa (EMEA) experienced outage as a result of technical issues on Vodafone networks. It is believed that a network fault between Vodafone and RIM is the cause, locking users out from their emails, BlackBerry Messenger (BBM) and other BlackBerry services.

And back in September 2012, BlackBerry revealed that it’s BlackBerry messaging service and Enterprise servers were experiencing issues and had become unresponsive across the very same regions, on the very same day the iPhone 5 launched around the world.

The ‘Architectural Dimension’

Unlike iPhone and their Android based rivals, Blackberry’s architecture exposes the company to tremendous impacts when system failures do occur.

A big reason that some BlackBerry outages are so noticeable is that BlackBerry only has two data centers to handle the e-mails of its millions of subscribers around the globe. One center is in Canada and handles messages in the Western Hemisphere and parts of Asia, while the other is in the U.K. and handles traffic from Europe, Africa and the Middle East.

These two network operation centers or NOCs as there are called, receive e-mails from a company's e-mail server, confirm the users and then forward the secured messages to the corresponding BlackBerry.

But this central handling makes BlackBerry traffic vulnerable to widespread outages. Other hand set makers like Nokia and Apple don't have centralized data centers so if they do experience network outages they are not global in scale. Unlike BlackBerry, they are the responsibility of particular wireless carriers such AT&T, Sprint, Verizon and T-Mobile -- or a particular messaging systems like Gmail, Hotmail or Apple’s iMessage; in other words not the maker of the phone itself That makes Blackberry’s problems inherently more localized.

It was this BlackBerry baton-passing system that went down in 2011 killing or slowing e-mail and texting services for millions of people in Europe, North America, South America, Asia, the Middle East and Africa.

Not all BlackBerry users are equally affected by outages. Corporations and government organizations with their own internal e-mail setup often subscribe to the BlackBerry Exchange Server, or BES. See the adjoining image.

With this service, companies have their own server on-site and thus are less prone to blackouts. Individuals, however, use BlackBerry Internet Service, or BIS, where e-mails are pushed from the mail account on an individual's computer, through one of BlackBerry’s data centers, to the individual's handheld device by phone carriers like AT&T.

"It's because of the way RIM has set up the (network) architecture that is the downfall when it comes to these types of outages," said Sean Armstrong, who manages wireless communications at a large tech company. "When it's working fine, it's a great system. When it's not working fine, it's a failure."

Force Field Analysis

The outages and service loss is a fact for many phone users but it’s the carriers who face the brunt. Blackberry unique network architecture puts them in the firing line. Their outages such as this are symptomatic of fundamental, if not deep underlying systemic problems not from isolated one-off’s.

Certainly, the company was reeling from many problems in its business. Bad news was everywhere. And for many people, the outages were a very emblematic representation of it. Clearly the companies handling of the situation in 2011 was exceptionally poor. It was text book crisis mismanagement particularly from a public relations aspect.

From the information obtained from various research sources we have arrayed the restraining forces (systemic factors that were behind the outages) against driving factors (adjustments or countermeasures) that will act to prevent a repetition. This is presented in a force field analysis diagram shown below.

Following the departure of Lazaridis and Basillie, the new executive management team will likely have moved to tackle the underlying system and structural issues as the company nearly went into meltdown at the time. Management is a factor in itself and all items can lead to management but the analysis focuses specifically on a more technical calculus. Our identification of these factors has been picked up by many industry experts rather than from Blackberry sources. Our assessment identifies the driving forces and restraining force and each is weighted with a score from 1 to 5. See below.

Force Field 4-11-2013.JPG

From the analysis our assessment is that BlackBerry will have moved to address many of the underlying drivers behind the outage. Those include, ensuring that the technical resources and systems ensure proper maintenance checks and a revisit and full validation of the functioning and reliability of back-up fail safes. Priorities would have been revised and budgetary constraints will have been removed considering the high a cost of the 2011 incident. Disaster recovery would have been surely reviewed so we can expect Blackberry will respond in a far more effective and professional manner to minimize disruptions should they happen again in the future.

Our analysis does given a slighter higher weighting to the restraining forces. In other words the network architecture as configured will always present an existential risk as long as the company is wholly reliant on two data centers to handle all its worldwide mail traffic. This is clearly an issue and one that will accentuate if the company can turn around its business and start building new subscriber numbers again. BlackBerry will continue to have risk of outages, likely at a much lesser frequency and scale but the company’s responses when they do, will be far more professional.