WASHINGTON: Continued investigation reveals just how technically advanced the SolarWinds hack was — and how challenging it must have been to execute. By comparison, exploiting the zero-day vulnerabilities recently disclosed in Microsoft’s Exchange software is child’s play.
Public and private sector officials continue to openly question how it’s possible the U.S. government failed to detect the SolarWinds cyberespionage campaign. Some have said that the Cybersecurity and Infrastructure Security Agency’s EINSTEIN technology should have detected the attack. Others point to failures in the security architecture, technologies, and processes at government entities, including the 12 known to have been compromised in the hack. Some of that may be true, but it also overlooks some truly remarkable aspects of this hack, at the risk of underestimating this threat actor and misjudging effective solutions.
(Notably, U.S. Cyber Command Executive Director Dave Fredrick said at a virtual event last week that there’s “no evidence” Defense Department networks were compromised by this cyberattack.)
Following the campaign’s discovery, the National Security Agency urged federal and defense entities to adopt a zero-trust security model.
Why did so many public and private entities for months fail to detect the hack? Why did only one of the tens of thousands of victims eventually find it? To answer these questions, it helps to return to the Feb. 23 Congressional testimony of FireEye CEO Keven Mandia, as well as to what threat intelligence and forensics evidence have revealed since its discovery. FireEye discovered the hack after it became a “stage two” victim.
Stage one of the hack entailed the threat actor compromising SolarWinds Inc. and its Orion software by stealthily inserting a backdoor into a software update. Stage two victims include anyone who downloaded the legitimate, yet compromised, update that was hacked during stage one. Some have characterized the hack as “discriminate,” while others have said it was “indiscriminate.” The hack was, in fact, both: highly discriminate in stage one and largely indiscriminate in stage two.
Threat actors implemented excellent operations security to avoid being detected and used many stealthy tools, tactics, and procedures (TTPs) during the hack.
Mandia told Congress the SolarWinds hack is just the latest in what has been a “multi-decade campaign” for this threat actor. Microsoft President Brad Smith, who testified in the same hearing, said his internal security team estimated the SolarWinds hack involved the work of “at least 1,000 engineers.” Given that type of time frame and resourcing, it’s perhaps possible to begin to appreciate what the threat actor achieved in this hack.
As to one factor that complicated detection, a key step in the second stage of the hack involved compromising Microsoft’s Active Directory in breached organizations. Active Directory stores an organization’s IT user accounts (i.e., names and passwords) and associated access controls, which permit or deny users privileges to and permissions for IT resources, such as applications, servers, and files.
Gaining administrative control over Active Directory enables threat actors to pose as legitimate IT users within breached organizations, to authenticate using valid credentials, to lift user access controls, and to create new accounts, among other activities. Without specific types of security monitoring, such as anomaly-based detection baselined against typical user behavioral patterns, threat actors are free to move throughout an organization’s IT systems posing as a legitimate user, with no flags ever being raised. This, of course, posed a huge challenge to detecting the threat actor once it was inside breached organizations’ networks.
While the Active Directory hacks certainly complicated detection of threat actors on compromised networks, it’s not an uncommon step taken as part of advanced persistent threats. There is, however, one aspect that stands out as truly remarkable in what was generally a remarkable hack.
One of the most common methods for hacking software is to change the source code, but that’s not what happened at SolarWinds, according to Mandia. Instead, the threat actor modified the software build process. This is important to remember and understand, because it has potentially far-reaching ramifications going forward. Modifying the software build process, based on what’s known today, is a novel threat vector.
The software build process is a step in the software development life cycle. The build process converts high-level computer code written by human programmers, in a language such as C++ or Java, into low-level machine code, which is what computers actually process and execute as instructions. This conversion — from high-level programming language to low-level machine code — is what software developers call “compiling” the code.
The software build process is usually the last step before code is deployed into “production environments,” which are where organizations’ employees actually use the software. In the case of SolarWinds, the company deployed the compromised Orion code to approximately 17,000 customers, according to Mandia. Again, any organization that downloaded the legitimate, yet compromised, software update was instantly breached via the inserted backdoor and became a stage two victim in the hack.
Importantly, based on public information to date, it appears the threat actor conducted second-stage attacks against only a small percentage of the 17,000 total organizations breached. But questions remain about how many organizations are still breached today and what exactly was stolen from the approximately 200 organizations estimated to have been actively targeted in second-stage hacks.
Not only is modifying the software build process a “unique” attack vector, as Mandia said, it’s difficult to achieve. So challenging and unique is this vector that, as a precautionary measure to ensure its success, the threat actor carried out a “dry run” of the attack using harmless code in October 2019 to make sure it worked. Only later, satisfied that its proof-of-concept had been demonstrated, did the threat actor launch the actual attack in March 2020.
One concerning aspect, given the nature of the attack vector, is this: Modifying the software build process, rather than the source code, means that “this is a more portable attack than just at SolarWinds,” Mandia said.
Even FireEye, which specializes in breach investigations and employs thousands of highly trained experts in digital forensics, was nearly bamboozled during the investigation into its own compromise. Mandia said FireEye dedicated nearly 100 employees, most with 10,000 hours of prior forensics investigation experience, to discovering how the company was breached. “After exhausting every investigative lead,” Mandia said, the company’s experts opted to take a radical approach.
The investigators had traced the earliest indication of compromise to a SolarWinds server. Mandia said the experts decided to “tear [the server] apart.” That is, they “decompiled” the code. Then they sifted through 18,000 files, 3,500 executable files, and over 1 million lines of assembly code that were installed as part of the compromised SolarWinds software update.
Assembly code is a low-level computer language that consists of symbolic equivalents to machine code. It’s one step above the lowest-level machine code in the continuum between low and high languages. Programs can be written in assembly language, but very few choose to code this way today.
Just reading assembly language is an exacting skill. Conducting a forensics investigation based on assembly code is a highly specialized sub-field within cybersecurity. Few cybersecurity professionals possess this skill, and even fewer organizations employ such experts full time. FireEye just happens to be one of them.
The task FireEye investigators undertook is perhaps not quite as difficult as searching for a specific grain of sand on the beach, but after a few hours of this work, you would likely conclude there isn’t a material difference between the two. (You can try reading assembly language samples here. Note that this code is fairly well-documented compared to what’s often encountered by forensics investigators.)
So, after nine months of investigation by 100 highly trained and experienced cybersecurity forensics experts, FireEye finally discovered the “proverbial needle in a haystack” — a malicious “implant” in SolarWinds assembly code — in December 2020.
“You wonder why people missed it,” Mandia told Congress. “This is not the first place you would look. This is the last place you’d look for an intrusion.”
And, Mandia warned, “There’s no magic wand to say where’s the next implant.”
Joe Patrice is a senior editor at Above the Law and co-host of Thinking Like A Lawyer. Feel free to email any tips, questions, or comments. Follow him on Twitter if you’re interested in law, politics, and a healthy dose of college sports news. Joe also serves as a Managing Director at RPN Executive Search.
Jordan Rothman is a partner of 
Kathryn Rubino is a Senior Editor at Above the Law, and host of