Mainframe of Shame: The IT Failure That Crashed Vermont's Unemployment System | Tech | Seven Days | Vermont's Independent Voice

News + Opinion » Tech

Mainframe of Shame: The IT Failure That Crashed Vermont's Unemployment System

by

DREAMSTIME.COM | GINASANDERS
  • Dreamstime.com | Ginasanders

For nearly a decade, the Vermont Department of Labor sought to replace the rickety mainframe computer that has powered its unemployment insurance system since the 1980s. Under a federal program designed to encourage cross-border collaboration, Vermont worked with a series of other states to build a cloud-based system that could more efficiently and reliably cut checks to the unemployed.

But after years of infighting, delays and cost overruns, Vermont and its two most recent partners, Idaho and North Dakota, gave up and parted ways in late February. By then, Vermont had spent roughly $10.5 million in federal funding on the failed IT project — and had little to show for it.

"We weren't getting the results that we wanted out of this, and it wasn't working well at all," Gov. Phil Scott explained.

The demise of the Internet Unemployment System Consortium was just the latest in a long series of failed IT overhauls overseen by the state. It was hardly the most expensive, and because it was federally funded, lawmakers paid it little attention. Had it not been for COVID-19, its death may have gone largely unnoticed.

But weeks after the collaboration ended, the coronavirus pandemic prompted a surge in unemployment claims that overwhelmed the decades-old IBM mainframe and the roughly 15 Department of Labor employees who process claims. The unprecedented volume repeatedly froze and crashed the system, according to Secretary of Digital Services John Quinn, who compared the result to "the spinning wheel of death" one might encounter on an Apple Mac computer.

By the end of last week, 34,000 Vermonters who had filed for unemployment were still awaiting payments. Facing a backlash from homebound and cash-strapped constituents, Scott set a Saturday night deadline to resolve the problems, pledging to send $1,200 checks to anyone still stuck in the system by the end of the weekend. Come Monday morning, the governor showed up at the state's Middlesex mail processing facility toting boxes filled with nearly 8,400 checks.

"I understand these delays have been devastating for Vermonters who are staying home to stop the spread [of the coronavirus] and save lives but who still need to put food on the table and pay their bills," Scott said at a press conference later that morning.

State officials who work with the antiquated computer system say that if it had been replaced as planned, more Vermonters would have received unemployment benefits sooner. They caution, though, that technological challenges weren't the only ones they faced. "It would have helped with some aspects, but it's not like it would have been smooth sailing," said interim Labor Commissioner Michael Harrington.

Because the unemployment insurance program is heavily regulated by the federal government, claims are frequently flagged for inconsistencies and potential fraud, requiring department staffers to address and adjudicate the issues — a labor-intensive process. "The UI system is just designed to have some level of human interaction and decision making as part of it," Harrington said.

His department typically handles just 3,200 claims at a time, he estimated, and has the capacity to process up to 9,000 at a time. In a typical recession, it takes months for the unemployment rate to swell, allowing the department to staff up gradually. "To go from 3,000 claims to 70,000 claims in a matter of three to four weeks is just unimaginable," he said. "Nobody could have foreseen it."

But the state's technological deficiencies have exacerbated the staffing shortage because the computer system is challenging to use and requires extensive training.

It also frequently breaks down — and hardly anyone knows how to fix it. According to Quinn, who also serves as Vermont's chief information officer, only 10 or so programmers on the state payroll know the nearly obsolete COBOL language necessary to tweak and patch the system.

For the most part, according to several state government leaders, that job falls to a 44-year-old Vermont Technical College graduate named Hunter Thompson, who has worked for the state for 17 years. Thompson spent much of that time at the Department of Labor, where he learned COBOL and, just as importantly, the intricacies of unemployment law. Though he has since been promoted to division director in the Agency of Digital Services, he is called back to his old department whenever the mainframe runs into trouble — which, recently, has been constantly.

"People are getting checks and people are getting the support that they need because this guy's there at midnight making sure it doesn't explode," said Dustin Degree, Scott's director of workforce expansion. "He's trying to keep a treadmill from eating an encyclopedia."

According to Quinn, the mainframe is about the size of a refrigerator and sits in a Montpelier data center that "looks like one of those 1970s NASA space launch rooms."

"That guy is a machine," Quinn said of Thompson. "He's processed more claims by himself than everyone else put together."

Thompson himself is more modest about his contributions. In an email to Seven Days, he called his work part of a team effort. "It is by no means a one-man operation and I am only one piece of this puzzle," he wrote.

Digital Services Secretary John Quinn - FILE: JEB WALLACE-BRODEUR
  • File: Jeb Wallace-brodeur
  • Digital Services Secretary John Quinn

The state's reliance on a dwindling number of programmers to keep the mainframe running has been recognized for years as a major vulnerability. "You always had the sense that it was fragile," said Secretary of Commerce Lindsay Kurrle, who preceded Harrington as commissioner of labor. "Part of that was people sounding the alarm that this is pretty old and the people who know how to fix it are getting to retirement age."

An independent review of the state's IT projects commissioned by the legislature's Joint Fiscal Office in 2017 noted that IBM no longer supports the system, and few vendors will work on it. "The inability to provide this support puts Vermonters at risk," the report found.

Since the last recession, touched off by the 2008 financial crisis, the U.S. Department of Labor has encouraged states to jettison their aging unemployment processing systems. The 2009 American Recovery and Reinvestment Act provided $7 billion to modernize them — but the money came with strings attached: The feds wanted each state to team up with others to reduce the cost of development.

From the start, the plan appeared problematic. According to a 2012 report by the U.S. Government Accountability Office, state officials noted that unemployment systems — not to mention procurement, budgeting and IT systems — varied greatly from one state to the next. One unnamed official mentioned in the report "questioned whether a common platform for a system can be successfully built and made transferable between states in an economically viable manner."

In 2011, Vermont teamed up with Maryland and West Virginia to build a new system, but the Green Mountain State withdrew four years later over concerns about which entity would control the technology. In 2016, Vermont hitched up with Idaho, which had already managed to replace its mainframe and was looking to share its proprietary software with others. North Dakota later signed on, forming a trio.

In a December 2016 report to the legislature, state technology officials expressed confidence that the project would be completed by the end of 2018, but the deadline kept slipping. A 2019 version of the same report featured a new completion date of July 2021.

"Constant changes in scope due to decisions made by both the Vermont and Idaho teams has resulted in an extended timeline," the report read. "In addition, the lack of transparency into the Idaho roadmap has presented challenges prohibiting the Vermont team to accurately estimate the implementation schedule for the project."

The report questioned whether enough federal funding remained to see it through completion.

2019 Unemployment Insurance Modernization Report A 2019 report from Vermont's Agency of Digital Services to the legislature documenting the progress of the Internet Unemployment System Consortium

"The problems that were happening were arguably going to bankrupt the project and leave us with no money for maintenance and support," said Kurrle, who left the Department of Labor last summer. "So even if we had gotten to the launch, we would have burned through all the money just because it had taken so long."

According to Quinn and Harrington, the state stuck with the project as long as it did in part because it feared the feds would attempt to claw back the money it had already spent if the collaboration sputtered out. But by the end of last year, it was clear it could not go on. Scott met by phone with his counterparts in Idaho and North Dakota and, on February 28, the three governors wrote U.S. Secretary of Labor Eugene Scalia to inform him that they were dissolving the consortium.

Governors' Dissolution Letter to Sec. Scalia A February 28 letter to U.S. Secretary of Labor Eugene Scalia from the governors of Vermont, Idaho and North Dakota declaring their intent to dissolve the Internet Unemployment System Consortium

According to Quinn, the very concept of a multistate collaboration was "a disaster" from the get-go. "It all sounds very good, but having spent 20 years in state government, that's just not the way states work," he said.

Though the three states were equal partners on paper, Quinn said, Idaho "held all the cards" because its engineers were the ones doing the programming. "We were never able to make the governance work," he said.

Another problem, Harrington added, was that it became increasingly clear over the years that the commonalities between the states' unemployment systems were far fewer than originally understood, requiring unexpected work to make Idaho's program meet Vermont's needs.

"It's too bad it didn't work out, but in the end we just had to say, 'Listen, we can't in good conscience continue down the road we're on where the deadline gets extended and extended and we'll eventually have to spend state dollars,'" Harrington said.

What comes next for the state's beleaguered unemployment system is far from clear. Starting from scratch wouldn't be cheap — and even though Harrington is now confident that the feds won't penalize Vermont for withdrawing from the compact, they're unlikely to provide more funding. According to Quinn, one back-of-the-envelope estimate from a technology contractor found that a new system could cost Vermont as much as $37 million.

For now, the state is working through the current crisis by coming up with work-arounds. A new process to handle the next influx of claims from independent contractors and the self-employed will bypass the mainframe. And Quinn said he is expediting a plan to outsource the work currently performed by the mainframe to a data center in New Jersey, though it will still use the same clunky software.

Until then, the state will have to hope that the system — and Hunter Thompson — keep working.

Correction, April 23, 2020: An earlier version of this story inaccurately described John Quinn's role in state government. He is the chief information officer.

The original print version of this article was headlined "Mainframe of Shame | The $10.5 million IT failure that crashed Vermont's unemployment system"

Get news about coronavirus in Vermont