By Greg Price

On the extreme eastern coast of Florida, near West Palm Beach, sits Riviera Beach, a city of over thirty-two thousand.  I visited the city several years ago and learned that Burt Reynolds lived there as a child and his father was once the Riviera Beach Chief of Police.

For the past three weeks, the city has been held hostage by a group of hackers.

Essentially, all city operations have stalled to a crawl.  Where paper can be used, it is.  However, many of the online operations cannot be replaced with paper; those remain offline.  No email, paper payroll checks, no phones, water pump stations went offline, police officers began processing everything with paper, city operations grinded to a halt on 29 May 2019.

What happened?

On 29 May 2019, someone in the police department opened an email.  Within a few moments, all of the City’s email systems went offline.  Then the financial systems stopped working.  Systems began to fail throughout the City’s network.

Initially the City thought the issue was related to its aging computer systems.  The City Council held a special meeting and authorized nearly $1 million to update the City’s computer and networking systems.  The reports suggests that the virus exposed the aging infrastructure and the frailties with older equipment; sources indicate that the computers were at least six years old and needed to be replaced anyways.

On June 5 2019, the City’s website reappeared and new email addresses were created for all employees.  The announcement indicated that the city had “experienced a data security event.”

The IT staff worked feverishly to restore operations.  The losses were attributed to the email virus and aged equipment.

But, as the efforts to reconstitute data began, a more sinister picture came into focus, and, the narrative changed.

Essential data was inaccessible.  The email message introduced a combination of malware and ransomware into the City’s systems.  The first wave of the attack eroded access to the critical systems.  As efforts to create workarounds began, the ransomware encrypted, hid the essential data.

Once the IT staff appropriated new resources and began restorative efforts, they could not access their data.  The hackers requested ransom via a digital currency, bitcoin.

For $600,000, the bad guys would release the data.

On Monday night, the City Council met and authorized the city insurer to pay 65 bitcoins, valued at $592,000.  The City would pay an additional $25,000 out of City funds to cover its deductible.

The insurer began negotiations with the hacker shortly after the data restores failed.

According to online reports, the City has partially restored water pumping services and some financial services.  They hope the bad guys will hold up their end of the deal, and, grant access to the encrypted data.

The FBI, Secret Service and Homeland Security are investigating the attack.

In the past couple of years, dozens of US cities have experienced similar nightmares.  Baltimore was the most recent, who refused to pay the hackers; the city continues to try to recover.  Atlanta came to a standstill and has spent almost $20 million trying to address their cyberattack.

One of the most common misconceptions about these events is federal investigative intervention.  The federal agencies visit and conduct an investigation – they do not repair, they do not offer advice on remedy.

I’ve worked two large city ransomware attacks.  Not only did City officials in both cases expect the FBI to find the bad guys, but, assist with restoring their data and correcting their cybersecurity problems.  The federal agencies offer a single piece of advice when ransom is requested for sequestered data: decide if the loss of the information warrants paying the ransom.  And paying the ransom doesn’t guarantee that the bad guys will gold up their end of the deal; they broke in, remember.  What you’re buying is the hope that they will release a working decryption key.

So, do you pay the ransom?

Cybercrime reports suggest that attacks are on the rise.  Many attacks go unreported, so, it’s likely that public sector attacks are more common than we realize.  One side of the coin offers that through paying, we enable the bad guys and promote the behavior.  The other side believes that rolling the dice is worth the financial burden: pay and hope for a quick recovery is better than a lengthy, agonizing outage.

As a security practitioner, I’ve witnessed some challenging situations.  From the outside, it’s easy to suggest that IT failed.  “Why didn’t they just backup the data and restore” is a common refrain.  In one of the instances that I worked, the City had extensive backup and data protection systems in place.  The bad guys seized all of those and encrypted everything through an unpublished exploit in a server platform.  In the other case, there were no backups – the IT operation was extremely lean and they never spent adequate time to build a proper backup plan.

So, I suppose I’ll offer one of those unsatisfactory answers: it depends.  Anyone, any time can be a victim to a cyber-attack – there exists no single, easy solution to remedy every possible threat.  Reviewing and testing a thorough disaster recovery and business continuity plan is your best bet with addressing these types of attacks.  Cyber insurance companies have begun to witness the areas of deficiency in organizations; I suspect they will soon refuse coverage if the requesting entity cannot adequately exhibit not only solid security efforts, but, tested and viable recovery programs.