Case study: Technical benchmarking of EDR solutions in an OT environment

One of our customers asked us to compare EDR in OT systems. Here are the results.

What were the study’s main goals?

One of our customers decided to compare EDR (endpoint detection and response) solutions for their OT (operational technology) information system. The company called upon our services to analyze different EDR solutions, first from a theoretical aspect and then from a technical one. The theoretical analysis allowed our customers to shortlist 3 out of 9 EDR which comply with their needs. Then during the second phase, the objective was to study which solutions meet their technical requirements.

What methodology do we use for this type of technical analysis?

To compare EDR solutions from a technical point of view, we build attacks based on references. Then, we launch the attack scenarios in our lab environment. Each phase will be described below.

1. Public references

To simulate attacks that look like real ones and similar to those on the field, we use the matrix MITRE ATT&CK for ICS. We often map attacks that are carried out during breach tests on this matrix. The framework ATT&CK is built according to attacks (malware or manual attacks) that have occurred on the ICS environment in the past years. This public knowledge database is meant to describe actions an opponent can perform while operating within an ICS network. The matrix maps an attack scenario that starts after the first access on an OT network has been established. Others MITRE matrices (ATT&CK enterprise or PRE-ATT&CK) describe how to gain this access (entering the information system or if needed the compromising the IT network).

The matrix is divided into 11 categories called “tactics”. For each tactic, several types of attacks, called “techniques” are referenced. For each technique, you will find a description, the attack group that used it, several types of possible mitigations. This framework maps different attack entry vectors on the ICS environment, but it also maps the impacts that an intrusion could have.

For instance, an attack scenario that deletes alarms on a SCADA software, and therefore prevents operators from being notified of critical conditions can have an impact on the “visual impairment” or even, depending on the conditions, a “loss of safety”.

Designing attack scenarios

After choosing the MITRE ATT&CK for ICS as a reference, we study techniques that can be detected by an EDR. As mentioned before, the framework provides different types of mitigation for each technique, to select the technique, we look for the mitigation that matches EDR functionalities. The tactics “execution” and “evasion” are the ones we mainly target for this assessment. The following techniques (in blue) were included:

Techniques covered by the study – Source: Orange Cyberdefense

After mapping the techniques that apply to this benchmark, we define checks each with its complexities. Indeed, different levels of tests (from well-known attack payload to completely customize one) are created for each highlighted technique above.

For the payloads, we rely on public ones, some of them being used during a breach test exercise. We also develop new payloads in C#/C++ in order not to make their signatures public.

Then with the defined unitary tests, we correlate several of them to build an entire attack scenario with OT consequences.

For example, we designed, developed, and tested the following use case map according to the ICS Kill chain phases:

The matching coverage on the ICS ATT&CK matrix is the following:

2. Lab construction

After developing various attack scenarios, we build the test environment.

Windows computers with widely used versions (W7,10,2012,2016,2008R2) were set up as well as some older ones like Windows XP. Indeed, in OT environments, the machine lifecycle is longer than in an IT environment. Sometimes, migration is not an option to ensure compatibility with older software that is critical for the production line. One of the customer’s goals is also to see the results of an attack on this kind of system.

Moreover, to simulate a more realistic OT network, several SCADA programs were installed on the systems. For example, PLCs are monitored by Topkapi supervision from the windows 2008R2 server.

To be more efficient, all EDR solutions need to be linked to the constructor’s cloud. It’s necessary to have access to all functionalities such as threat intelligence. However, on an ICS perimeter, assets should not have a direct Internet connection. Therefore, the tested EDRs were installed in hybrid mode. This mode allows the agent to connect to a local console server that should be installed in an industrial DMZ (Demilitarized Zone) and only the console server connects to the Internet. A trusted connection needs to be established between the agent and the console and between the console and the cloud. The interconnections can be represented as follows on the Purdue model:

3. Attack scenario execution on the Lab

After building the use cases and setting the lab, we ran the tests on the different operating systems. For this phase, each EDR publisher helped us interpret the results and explained how the detection can be improved by configuring the solution differently. Each result was described and synthesized in a report for our customers.

What is the feedback from the benchmarks?

The EDR configuration is the most important point, it should be defined carefully in an OT environment to prevent any impact on availability. For instance, we use for all EDR a detection policy configured to maximize the detection rate and see the solution’s limits. With this policy, one of the SCADA software was detected and blocked. That is, of course, not a policy that should be chosen in a production environment. With this type of policy, several false positives were also raised that’s why the configuration part is crucial. In an OT environment, the reaction after a threat detection (such as killing the affected process) should be set with more caution for the same reason as before, the impact on availability.

Moreover, another important point on an OT network is the bandwidth. Indeed, on some OT network bandwidth, it can be very limited in the case of a remote working site. During the benchmark, we compared EDR agents’ bandwidths on our lab to the one estimated by the publisher. For some EDR, there were major differences between them, which can be explained by the policy they have chosen that maximized the detection but induce false positives and bandwidth consumption.

Regarding the results, we obtained, we noticed that from our 8 scenarios that correlate different attacks, at least one step was detected by the solutions. In a real attack, that could be sufficient to raise an alert to the teams and initiate an action according to the incident response plan. For instance, if we analyzed the killdisk component that is a part of the black energy family. This component impacts the target’s availability. It performs, among others the deletion of system logs or stopping a process. That type of action, tested in the use case explained in this article, can be detected by a well-configured EDR solution.

However, if we take another example of malware like PLC-blaster. This malware scans the network to find S7 PLC devices and infects them. These types of actions are hard to detect with an endpoint protection solution and need other defense solutions such as network probes. These solutions need to be correlated with an appropriate incident response plan and can be integrated into a detection service.

That’s why at Orange Cyberdefense, we’ve built a use case factory dedicated to OT environments. Not only for endpoint protection, but for all kinds of defense solutions. It can be used to analyze and improve attack detection for the different attack scenarios that can happen in an OT environment.