Computer Virus Havoc

Introduction

One three-day winter weekend, a new virus disabled three-quarters of the 40,000 computers at a major auto manufacturer. This had the potential to cause a significant disruption to the company's operations and result in millions of dollars in lost revenue.

Background

The auto manufacturer regularly patched its office computers, but high-risk computers, such as those used in manufacturing facilities, were patched less frequently. This was due to the complexity of the patching process and the potential for errors to disrupt production. In the past, patching attempts had caused problems, such as system instability and data loss. Since many of these computers were connected to heavy and dangerous equipment, there was zero tolerance for instability. This had led to a culture of resistance to patching among manufacturing workers.

Situation

On the Thursday before the three-day weekend, a new virus began disabling all of the so-called high-risk computers, which had not been patched for several months. The virus attacked a known vulnerability in the operating system, but had not been detected early enough for the antivirus software to detect and remove it. This meant that 30,000 employees would be unable to work on Monday, and the manufacturing facilities would be idle. Nearly all of the office computers had been updated with the latest patches and were not vulnerable to the attack.

Response

On Friday, I secured a number of infected computers, examined the virus payload, and led a small team that developed and tested a cleaning procedure, which would take roughly 25 minutes per computer. Luckily the virus only affected the operating system and did not damage data stored on the computer.

On Saturday, the team mobilized and trained 100 employees to perform the cleaning procedure on 30,000 computers at dozens of sites. At 25 minutes per computer, it would take this team 125 hours to complete the task. I continued to work with the developers throughout the day, and eventually created and tested a script to automate the cleaning procedure. This reduced the effort to roughly 5 minutes per computer, and several could be run simultaneously by the same technician.

By Sunday morning, the employees, freshly empowered with the removal script, heroically removed the virus from all of the remaining computers present in the buildings.

Result

On Monday, all 40,000 computers were operational, and the manufacturing facilities opened. Computers that were unavailable over the weekend were treated as they entered the building. As a result, the auto manufacturer was able to avoid a significant disruption to its operations and immeasurable lost revenue. By the end of the week, permanent processes had been implemented to aggressively patch the high-risk computers, recognizing that malware was a far greater threat than the still imperfect process of applying security updates.