'Holy moly!': Inside Texas' fight against a ransomware hack
It was a steamy Friday two Augusts ago when Jason Whisler settled in for a working breakfast at the Coffee Ranch restaurant in the Texas Panhandle city of Borger. The most pressing agenda item for city officials like him that morning: planning for a country concert and anniversary event.
Then Whisler’s phone rang. Borger’s computer system had been hacked.
Workers were frozen out of files. Printers spewed out demands for money. Over the next several days, residents couldn’t pay water bills, the government couldn’t print checks, police officers couldn’t retrieve certain records. Across Texas, similar scenes played out in nearly two dozen communities hit by a cyberattack officials linked to a Russia-based criminal syndicate.
In 2019, ransomware had yet to emerge as one of the top challenges confronting the United States. But the attacks in Texas were a harbinger of the now-exploding threat and offer a case study in what happens behind the scenes when victims come under attack.
Texas communities struggled for days with disruptions to government services as workers in small cities and towns endured cascading frustrations brought on by the cyberattack, according to thousands of pages of documents reviewed by The Associated Press and interviews with people involved in the response. The AP also learned new details about the attack’s scope and victims, including an Air Force base where access to a law enforcement database was affected and a city forced to operate its water-supply system manually.
Recent ransomware attacks have led to gasoline shortages and threatened meat supplies. But the Texas attacks — which, unlike recent prominent cases, were resolved without a ransom payment — make clear ransomware need not hit vital infrastructure nor major corporations to interrupt daily life.
“It was just a scary feeling,” said Whisler, Borger’s emergency management coordinator.
Early on Aug. 16, as most Texans were still asleep, hackers half a world away were burrowing into networks.
As the attack’s impact became apparent, the city manager of Vernon emailed colleagues that the city could get back online by paying a $2.5 million ransom but that was “obviously” not the plan.
“Holy moly!!!!!” came the reply.
The culprits were affiliated with REvil, the Russia-linked syndicate that last spring extorted $11 million from meat-processor JBS and more recently was behind a Fourth of July weekend attack that crippled businesses around the globe.
The August 2019 hackers gained their foothold through an attack on TSM Consulting Services, a Texas firm that provides technology services to local governments. The attackers branched through screen-sharing software and remote administration to seize control of the networks of some of the company’s clients.
Within hours, state and federal officials were hunkered inside an underground operations center normally used for calamities like hurricanes and floods. Gov. Greg Abbott declared a cyber disaster. Texas National Guard cyber specialists were activated.
“Basically, if there’s a municipal function that you would go down to a city hall for, or that you would rely on the police department for, it wasn’t available,” said Andy Bennett, the state’s then-deputy chief information security officer.
In Borger, a city of fewer than 13,000, ransomware demands spat out of printers and flashed on some computer screens. Government files were encrypted, their titles replaced by gibberish combinations of letters and symbols, said city manager Garrett Spradling.
Vital records, like birth and death certificates, were offline. Signs posted on a drive-up window outside City Hall said the city couldn’t process water bill payments but that cutoffs would be delayed.
Because the city had paid for remote offsite backup, Borger could reformat servers, reinstall the operating system and retrieve data. The police department, however, retained its data locally and officers were unable to access previous incident reports, Spradling said.
Jeremy Sereno was working his civilian job at Dell when he was enlisted by the state to help. A lieutenant colonel and senior cybersecurity officer with the Texas Military Department, Sereno helped deploy Texas National Guard troops to hacked cities, where specialists worked to assess the damage, restore data from backed-up files and retake control of locked systems.
One of the first areas of concern was a small North Texas city. The attack locked the “human-machine interface” workers used to control the water supply, forcing them to operate the system manually, Sereno said. Water purity was not endangered.
“That’s what’s considered critical infrastructure, when you talk about water,” he said.
AP is not identifying the city at the urging of state officials, who said doing so could draw new attacks on its water system.
In Graham, the ransomware attacked a police server housing body-camera videos, causing hundreds to be lost. Instead of using mobile data terminals to run checks on people they encountered, officers had to rely on requests to dispatchers at a local sheriff’s office unaffected by the attack, said Chief Brent Bullock.
The impact wasn’t limited to local governments. Sheppard Air Force Base confirmed to AP that its access to a statewide law enforcement database used for background checks was temporarily disrupted.
One complication: TSM’s client list was encrypted, officials said. State officials didn’t immediately know which communities had been victimized.
They had to call around, said Nancy Rainosek, Texas’ chief information security officer. “There was one place that we contacted and they said, ‘no, no, we’re not hit,'” she said. Days later, “they said, ‘yes, we were.'”
Fortunately for Borger, most city services were restored within days. The city has since invested in additional cybersecurity protections.
“When you complain about having to change your passwords, you complain a lot more when it’s never happened to you and you don’t have anything to relate it to,” Spradling said. “You tend to complain a little less after you’ve had to answer the phone and tell 300 people they couldn’t pay their water bill.”
Even now, Spradling said, officials will go to pull an old report or address record — only to find it isn’t there.
DALLAS (AP)