On December 13, 2020, the Cybersecurity and Infrastructure Security Agency (CISA) issued an emergency directive that called for agencies to disconnect or turn off any SolarWinds Orion products by noon the following day, due to an exploit of Orion network monitoring software that posed a “grave risk” to agencies, critical infrastructure providers, and other private-sector organizations.
Cloud security solution provider RedSeal had a driver’s seat view into one effort to rid a sprawling government enterprise of this severe vulnerability. As a longtime partner of this Federal medical enterprise spanning the United States, RedSeal previously helped the organization get a comprehensive view of everything on its network, how everything on the network is connected, and the associated risks.
Three years prior to the SolarWinds vulnerability, the Federal organization wanted to quickly get its network data into RedSeal, so while waiting for authority to operate the cloud security solution on its network, the organization fed network data into RedSeal via SolarWinds’ configuration management database (CMDB). Through that process, RedSeal discovered that the organization was running six SolarWinds’ CMDBs. Most organizations have one CMDB, but some larger enterprises may run a CMDB in each geographical region they operate, for example.
“Your network gear knows what’s connected to it,” said Wayne Lloyd, Federal chief technology officer at RedSeal. “If you bring the network gear into a network visualization tool like RedSeal, you can see the unknown and then go identify it.”
Later, when the SolarWinds Orion vulnerability was discovered, the organization was able to use the logical and physical locations of the SolarWinds databases to identify all the clients that connected to the databases. Without the upfront knowledge of the CMDB locations, the organization would have needed to spend hours – likely days – uncovering that information, delaying its response, Lloyd noted.
In the wake of the SolarWinds’ exploit, Federal officials recognized the urgent need for improved network visibility.
“We cannot defend a network if we can’t see a network,” one senior Biden administration official said.
Security Vendors Partner to Pinpoint Agency Vulnerabilities
Often, it takes more than one tool or company to provide the data an organization needs to pinpoint and remediate vulnerabilities on a network. In December 2021, when CISA issued another emergency directive, this same Federal organization responded quickly, but in a different way.
The directive required agencies to immediately assess their internet-facing network assets for Apache Log4j, a popular logging framework. A vulnerability in some versions of Log4j allowed unauthenticated remote code execution by adversaries, which CISA Director Jen Easterly called “an unacceptable risk to Federal network security.”
To determine which applications running Log4j were exposed to the internet, the organization needed to translate each internal IP address to an external IP address – and sometimes up to three external IP addresses. The IT team used a used combination of RedSeal and other vendor security tools to get the information it needed.
“Forescout knew the devices but didn’t know the internal IP addresses. With RedSeal data, we translated the internal IP addresses to external IP addresses, combined that information with device data, and sent everything to Splunk for the organization’s security operations center so the IT team could quickly begin remediation,” Lloyd said. “It’s a team effort.”
CISA Executive Director of Cybersecurity Eric Goldstein said the Log4j vulnerability demonstrated the need for greater agency visibility into their own environments.
“I think this vulnerability reflects the work that we have yet to do, and I think that work will focus on ensuring that organizations have visibility into libraries and components in their environment,” he said.
The good news, Easterly noted during the initial Log4j response, “is that we’re really tackling the challenge with an unprecedented level of operational collaboration with our industry, the research community, and international partners. As I’ve often said, cyber has to be a team sport.”