I think I've seen this before - The Ivanti Déjà vu

If you google "it's always you three ivanti meme" these days, odds are that you will find something that indicates the real problem here is not just Ivanti. But adding a few other vendors still falls short of actually identifying what we really deal with here. In fact, what we can learn here is that there is a much more fundamental problem to be discussed that goes beyond addressing a few zero-days of a few particular vendors. And it's been there for years, and it's been known and flagged up for just as long.

So let's take a look at what the Ivanti case tells us on a wider scope. Let's see where we have seen this all before, and why we unfortunately will likely see this again in the future, like the needless sequel of a movie that already was bad to start with.

It's a given that software at this stage simply cannot be developed without flaws. That includes bugs in commonly used libraries, simple programming errors, and unfortunately security vulnerabilities. Dozens of these vulnerabilities are discovered and published every single day and of course researchers focus primarily on most commonly used technologies.

For instance, let's take a look at Microsoft. Our CERT issues several hundred advisories (called "Signals") every year, including warnings on vulnerabilities and threats. MS vulnerabilities featured the most by far in comparison to any other vendor (30 mentions, compared to 6 for the next one) and we have seen this consistently for the last few years. In our vulnerability scanning operations 52.1% of the "critical" and 62.3% of the "high" findings are related to Windows 10. It's important to mention that this does not mean Microsoft is an insecure vendor or that Windows 10 is an insecure system. It means primarily that they are commonly used and thus in the focus of a lot of research [source: Security Navigator 2024]. Of course that applies to security tools like Ivanti just as much as any other piece of software.

Big brands like Microsoft will always feature highly, but in 2020 we observed with curiosity the sudden prevalence of several leading security product vendors in the very short list of technology vendors who featured multiple times in our Signals that year.

We noticed a distinctive ‘bump’ that occurred in May that year, where an unusually high number of vulnerabilities was reported in these security technologies. Indeed, there was a four-fold increase in vulnerabilities reported in selected security technologies between March and May 2020.

In the below chart we have extracted what could be described as a "research cascade", showing how related CVEs have been researched which led to more research and subsequently to the discovery of more CVEs in similar product families [source: Security Navigator 2021].

Some of the vendors mentioned might appear familiar. We believe this extraordinary surge in security product vulnerabilities was the function of three factors:

• The notable 'success' of Pulse Vulnerability, CVE-2019-11510, from May 2019, which had been exploited in several high-profile attacks.
• The rapid and sometimes reckless adoption or expansion of secure remote access capabilities to accommodate remote workers during the COVID lockdowns, which made these technologies a very attractive target.
• A cascade effect in which the discovery of one vulnerability created knowledge, experience and ideas, and thus led to the discovery of different vulnerabilities in the same product, or similar vulnerabilities in different products.

It is important to note that, when properly dealt with, responsibly disclosed vulnerabilities are beneficial to a system’s security. They help vendors patch and defenders to avoid gaps with early countermeasures. What this is perfectly demonstrating is that the discovery of a vulnerability triggers more research which commonly leads to the discovery of yet more vulnerabilities. No surprise that we have seen something very similar in the past few weeks.

In July 2021 the US Cyber security and Infrastructure Security Agency (CISA) co-authored an advisory providing details on the top 30 vulnerabilities routinely exploited by malicious cyber actors in 2020 and 2021 [source]. CISA considers the vulnerabilities listed to be the topmost regularly exploited CVEs by cyber actors since 2020. Of the nine software companies appearing on this list, five would be categorized as security or ‘secure remote access’ vendors. That’s 55%. 

Table: topmost regularly exploited CVEs by cyber actors during 2020 according to CISA, ACSC, NCSC and FBI [source]

This dramatic datapoint correlates with our impressions, data and reporting on this issue over the last few years. Again, we emphasize that this is not a suggestion that these vendors build less secure products.

Rather this heightened level of activity involving these products is the function of three factors:

1. These technologies are located on the perimeter of the enterprise network – connected to the inside of the network while also presenting an Internet-facing attack surface – and are thus a natural target for attackers.
2. The importance of these technologies increased dramatically due to the increased levels of remote working. This attracted the attention of researchers, whose findings in turn, led to further research and weaponization.
3. The critical role of these technologies in the new ‘remote work’ reality, combined with additional challenges that emerge from the complex relationships between businesses, vendors, and service providers have ironically meant that these technologies are not being regularly and efficiently patched. 

Going back to our 2022 Security Navigator we took a closer look at the problem of managing vulnerabilities in security products. As the chart below illustrates, the overall volume of security product vulnerabilities had even been decreasing for a period of time. One might think that this had led to an opportunity to take a breath and relax, but one would actually be wrong.

With over 50 advisories across 9 vendors in August that year, the effort required to maintain appropriate patch levels or mitigations for these technologies was significant. [source: Security Navigator 2022]

At Orange Cyberdefense we believe this situation needs to be improved, and we proposed back then (and still propose) a conversation should urgently be held with various security product vendors about the challenge of managing vulnerabilities in products like firewalls and VPNs.

These are the conclusions we draw:

1. Attackers are targeting security products: Several datapoints and anecdotes suggest that security technologies are very much in the crosshairs of criminal and state-backed hacker groups. Research into vulnerabilities in security technologies is accelerating, and such vulnerabilities are being used to affect serious compromises at an alarming rate.
2. Effective vulnerability and patch management are critical: Given this new reality, it’s more important than ever that organizations can learn about new vulnerabilities, patches and workarounds quickly, easily identify affected equipment, and apply mitigations and confirm their effectiveness with minimum friction.
3. Many customers manage diverse estates, MSSPs almost always do: The challenge of vulnerability and patch management is exacerbated by having to manage different vendors and tools. Diverse sets of firewalls, for example, are common. This is, even more, the case for Managed Service Providers like us, who have to maintain different technologies, of different versions and configurations, across their customer estates frequently and fast. Failure to do so, especially on internet-facing and perimeter technologies, can have serious consequences.
4. The current processes are chaotic and ineffective: Direct feedback from our Security Operations Centers, supported by data we’ve collected on the issue, suggest that there is much that could be done to improve the state of vulnerability management in security technology.
   Challenges identified by our SOCs include:
   • Each vendor has their own format and distribution process – RSS, email, web page or authenticated web portal. Thus, automation is next to impossible.
   • Vulnerability classifications, rating and prioritisation vary across vendors.
   • Vulnerability and disclosure timing philosophies vary across vendors, leaving SOCs with no opportunity to plan or structure their efforts.
   • It's a challenge to map vulnerabilities and patches to inventory under management, to provide assurance that all potentially impacted systems have been appropriately protected.
   • Licencing and service fees are an issue. Patches and security upgrades frequently need to be paid for, creating conflicts of interest and latency.
   • The threat and potential impact associated with a mitigation are frequently difficult to articulate, leading to customers deferring necessary actions or inappropriately accepting avoidable risks.
5. We should be solving these problems, not creating them: As a major product and services provider, we believe that we have an obligation to work with our vendor partners to improve this situation for ourselves and our customers. It is a moral and commercial imperative to us as an industry to show leadership and fundamentally contribute to a safer digital society.

Plain and simple: this is not about Ivanti. Moving to another tool head over heels now is merely replacing one single point of failure with another single point of failure. The real issue at hand is to avoid having just one single point of failure in the first place and instead setting up cyber security in multiple layers.

The second important point to note is that we need to improve the way we handle managing vulnerabilities in security tools. Given the arguments raised above, we believe an industry-wide discussion needs to be had to determine whether the problem is as real as we perceive it is, identify existing efforts that may already be underway to address the issue, or create some form of partnership to work toward a better situation for ourselves and our customers.

Specifically: could we as an industry agree on standards and norms for vulnerability advisories?

Can we improve our ability to technically interrogate a product so that it can be matched with an advisory?