State workers on government holiday initially overlooked an alert from FairPoint Communications after the 911 system failed last Friday, the first in a series of communications lapses that exacerbated the crisis, a state official told Seven Days.
Enhanced 911 Board Executive Director David Tucker said in an interview that his agency is responsible for many failures that left Vermont emergency workers unaware for hours that residents could not use 911 to reach them.
"Those notifications didn't get to everybody that should have seen them," Tucker said. "We can't have people out there with no idea it's down. You know what people say about the fog of war? It got very confusing very quickly."
Additionally, Tucker said, the number of 911 calls that did not make it through to dispatchers during the outage is almost twice as high as has been widely reported. In total, 83 people tried to call 911 but couldn't get through, Tucker said. The earlier tally of 45 calls was based on information gleaned from FairPoint Communications, only one of the two private companies with that information. The other company, Intrado, reported their missing calls later, Tucker said.
As of this morning, the state finished checking back with most callers, and none had suffered ill effects, Tucker said.
Around 3 p.m. on Friday, network problems caused by a fiber cut in key FairPoint infrastructure prevented many Vermonters from being able to place outbound calls, including to 911. FairPoint provides the fiber infrastructure for Vermont's 911 system, which is run by Colorado-based Intrado.
One hour later, FairPoint, following protocol for outages affecting more than 25 customers, sent an email to state Enhanced 911 staffers notifying them of the outage, Tucker said. (Intrado did not notify the state, Tucker said.)
Until that moment, the 911 system had been down without any emergency official in Vermont being made aware of it.
But for the next 45 minutes, nothing happened. Friday was a government holiday. No one was in the Enhanced 911 office and no one was required to check email, Tucker said.
At 4:45 p.m., Tucker said, he happened to check his email, and saw a message from dispatcher in St. Albans, who was beginning to detect that something was amiss. The dispatcher hadn't been receiving any 911 calls, which he found alarming. "He said, 'I think we've got a problem,'" Tucker said.
That email prompted Tucker to scroll through his inbox and see the earlier alert from FairPoint, he said.
"There was a 45-minute period between when we got the first notice and when we were aware of it," Tucker said. "We missed it. It's our fault. There were few people looking at email because it's a day off. We made mistakes in the communications process."
The outage alerts sent by FairPoint are usually for small events — a car knocking over a telephone pole — that knock out 911 and other services to 25 customers or more, Tucker said. FairPoint should have done more to underscore the severity of this outage, he said. "This wasn't a typical outage," Tucker said. "I can make an argument the notice should have been clearer, or [there should have been] a follow-up phone call."
Information belatedly in hand, the Enhanced 911 Office set about calling Vermont's eight regional dispatch sites, which serve as hubs for their local areas across the state.
But there was one big problem with that plan — the phones were down. The protocols apparently didn't factor that in.
"We couldn't get through to them," he said. "That plan doesn't survive when you can't pick up the phone and call somebody."
Tucker's office was reduced to sending e-mails to those dispatchers, and hoping amidst the chaos that they would be read promptly. Like the dispatcher in St. Albans, many were beginning to suspect something was wrong.
Additionally, Tucker said his office reached out to the Vermont Emergency Management Center, which can send out alerts that run on the bottom of residents' televisions, and, in extreme cases, via cell phone. They were planning to send the alert when suddenly, around 5:45 p.m., FairPoint sent Enhanced 911 a new email —the problem had been fixed. Officials dropped the planned alert.
"We thought it was done," Tucker said.
An hour later, Tucker said, FairPoint reversed course, and announced the 911 system was still down. It wasn't until 8:45 p.m. that the system was finally fixed.
Tucker acknowledged that some emergency departments never learned that 911 was down. Regional dispatchers used radios to contact agencies within range. Tucker said that some police and fire departments learned off the outage via television reports.