Ivanti 0-days alert: Threat level 5/5
Updated - Watchtower provides more details on Ivanti 0-days, increased exploitation to be expected
Update 11, 29/02/2024 - Ivanti released a new version of its ICT tool, Mandiant details new malware used
Ivanti released on February 27 a new version of its external Integrity Check Tool (ICT). When run on a device, it now includes more information on files present on the system, and no longer require Ivanti's assistance to decrypt the snapshots. This means that customers will now have directly access to information on the status of their system. Nevertheless, running this new version results in some false positives (We for example found it flags 2 "newly detected files" tied to the output prepared by ICT). No false positives related to "Mis-matched" files have been found. This new ICT presumably now checks some internal directories previously whitelisted and abused by at least one threat actor (see below).
Ivanti servers managed by OCD will all be checked using this updated external ICT.
In addition, specific recommendations have been provided by Ivanti for customers using virtual appliances. Customers that did not complete a factory reset, or deploy a new build, and patch their appliance, Ivanti is now recommending that a new build of Ivanti Connect Secure be deployed, versus executing a factory reset.
Ivanti also emphasized that they did not confirm malicious actors successfully maintaining persistence if the security patches and factory resets were implemented. This precision was made to clarify and deny findings that CISA published in their February 29 advisory. In this warning, the US agency explained they were able to gain and retain persistence on a patched instance in a lab environment. Ivanti reminded instances were at risk if mitigations were applied after a compromise.
Mandiant published a 3rd report focusing on the new tactics, techniques and procedures (TTPs) employed by one probable Chinese threat actor they numbered UNC5325 (compared to UNC5221 for the first wave of attacks). We believe these actors may actually be one and the same, but are tracked separately for now by Mandiant. UNC5325 used the CVE-2024-21893 zero-day vulnerability as early as January 19, more than ten days before it being released by Ivanti on January 31. The attackers relied on LotL ("Living-off-the-Land") techniques to evade detection while deploying new malware strains named LITTLELAMB.WOOLTEA, PITSTOP, PITDOG, PITJET, and PITHOOK, to persist even if system upgrades, patches and factory resets were applied. It purposely had hidden the BUSHWALK webshell in a directory whitelisted by the former external ICT. Fortunately, persistence attempts have not been successful according to Mandiant due to flaws in the malware's code.
This second campaign has been attributed to a threat actor loosely associated to another Chinese one numbered UNC3886, based on similarities between the codes and techniques. UNC3886 primarily targeted the defense industrial base, as well as technology and telecommunications organizations in the US and Asia-Pacific. UNC3886 was for example behind the use of zero-days against Fortinet or VMware solutions.
These new elements confirm the advanced capabilities that China-backed espionage actors have against edge infrastructure. UNC5325 (along with its "older brother" UNC5221) had a deep knowledge of the Ivanti Connect Secure product.
To help detect the new malware strains identified, Mandiant has shared additional Yara detection rules, and a few host-based IOCs, that were included in Datalake (our CTI repository used by our Managed Detection and Managed Threat Intelligence services). We still recorded a few days ago opportunistic attempts to deploy for financial gain cryptominers on affected instances, using Chaos RAT.
The risk associated with this advisory remains at 4 out of 5 on our side, as no new vulnerabilities are revealed, just new malware strains used in limited, targeted attacks.
Updated on : 2024-02-21 10:33:34
Update 10, 21/02/2024 - Risk related to Ivanti vulnerabilities reduced as patches are applied and attacks decrease
Ivanti released patches for all versions of their products on February 14, allowing to fully fix vulnerabilities yet disclosed on all branches.
Eclypsium published a week ago an analysis of an Ivanti PSA3000 device. They figured out that the solution does embed many outdated and even EoL (End-of-Life) components, including libraries and the CentOS core used:
- Linux kernel 2.6.32 (end of life in February 2016)
- OpenSSL 1.0.2n (December 2017)
- Python 2.6.6 (August 2010)
- Perl v5.6.1 built for i386-linux (not x64, April 2001)
- Bash 4.1.2 which, surprisingly, has been patched for Shellshock
It includes presumably a number of outdated libraries with known CVEs, including 111 with known exploits. They also believe the Perl GUI and the internal firmware have multiple vulnerabilities, so expect more flaws will be revealed in the coming months.
Eclypsium figured out also that the (internal) ICT does bypass multiples folders (possibly to avoid False Positives), thus leaving a large post-exploitation persistence surface for attackers that may locate their backdoors specifically in these directories (/data, /etc, /tmp, /var, etc.).
The threat actor behind the 0-days managed in the early cases to deploy a memory-only resident webshell to avoid being detected by ICT:
- they made a copy of the original cav.egg file,
- replaced it with one containing the webshell,
- restarted the cav uwsgi worker which loaded the infected egg file,
- and finally restored the original, clean, file on disk.
A reboot would nevertheless remove such a webshell. Some artefacts left in /home/venv3/lib/python3.6/site-packages/pycache/ enabled us to date precisely when some implants were dropped.
Furthermore, Ivanti removed fortunately some internal tools that could be of interest to hackers having breached a device, but on the opposite did add others such as nmap or psexec, that could be used directly in post-exploitation activity without the attacker needing to retrieve them (and risk being discovered doing so).
Less than 200 (as of last week) hacked instances still publicly expose online the index2.txt malicious file tied to the backdoor we detailed in the last update. However, an unknown amount of devices remain compromised according to us, some even from the first mass exploitation phase in mid-January. Since mitigation and/or patches were applied to most appliances, we don't have visibility anymore on these hacked appliances such as the ones exposing publicly a tampered-with logo/login.gif file.
We still see some opportunistic attacks, mostly attempting to deploy cryptominers onto vulnerable instances. Yet, considering that most instances are now protected, we have reduced the risk level associated to this threat to 4 out of 5. Our recommandations made in the previous updates remain applicable.
OCD proposes you a webinar on this topic on February 29.
Updated on : 2024-02-09 10:46:06
Update 9, 09/02/2024: Ivanti patches yet another critical bug (CVE-2024-22024), CERT OCD share new technical details about a backdoor
Vulnerabilities:
Ivanti released details and patches fixing a new, high severity vulnerability they found in Connect Secure (and Policy Secure), tracked as CVE-2024-22024 (link for our clients). A remote attacker could exploit it by sending a specially crafted XML filein order to access specific resources without SAML authentication. Watchtowr earlier in the day teased publicly the CVE number on X (Twitter), as they independently found and reported it first to Ivanti. This also explain why Watchtowr criticized in a blogpost Ivanti for self-attributing the findings and released the actual timeline of the disclosure made to the vendor.
The vulnerability researchers took care to release explanation of the vulnerable endpoints, (in particular: /dana-na/auth/saml-sso.cgi), without directly providing a fully working PoC for low-level hackers. Advanced attackers could nevertheless exploit this information to successfully hack unpatched appliances. Furthermore, Watchtowr explained the issue comes from a regression introduced by Ivanti on January 31, when patching the 4 previous vulnerabilities we already discussed. This vulnerability is thus not believed to be exploited in the wild at this time, but can also be chained with one of the remaining unpatched command injection bugs existing in the product. A patch is therefore available for all supported versions, and include (if need be) the fixes for the 4 vulnerabilities we discussed earlier.
A Metasploit module that chains the previous SAML authentication bypass 0-day (CVE-2024-21893) with the command injection (CVE-2024-21887) is by the way now publicly available, making it even easier for low-skilled hackers to launched automated attacks against unprotected (not mitigated or patched) appliances.
Stats and DSLog backdoor:
The number of vulnerable instances is getting lower, as shown by ShadowServer. Additionally, the amount of instances compromised and embedding a backdoor within DSlog.pm keeps decreasing according to our daily scans, with less than 500 such appliances still remotely accessible today (and a very few more). We've proactively warned our clients identified in this list.
We also provided technical details on this backdoor in a public report available in PDF here. As already said, we detected these compromises indirectly, i.e. passively, with simple unauthenticated GET requests to the illegitimate files created by the backdoor and located at: /root/home/webserver/htdocs/dana-na/imgs/index2[.]txt.
Remediation:
We recommend you to patch the CVE-2024-22024 bug using the latest versions released by Ivanti.
Factory reset is not needed to fix CVE-2024-22024 according to Fortinet, if it was already done previously. Furthermore, 2 consecutive factory reset operations as we mentioned remain unnecessary. It is on the opposite strongly recommended to apply January 31's upgrades twice to make sure a rollback version does not stay in a compromised state.
Updated on : 2024-02-05 16:31:34
Update 8, 05/02/2024 - Rapid7/AssetNote disclose PoC for the critical SAML vulnerability affecting Ivanti CS, opportunistic exploitation ongoing
Vulnerabilities:
What we thought was the start of mass exploitation on Thursday evening was actually false positives generated by the internal ICT checks (running by default every two hours). These checks were actually finding new files from the package loaded onto the appliance when upgrading it. Nevertheless, in the wild exploitation by opportunistic attackers actually did start Friday night, right after Rapid7 then AssetNote again somehow irresponsibly disclosed publicly how to exploit the SAML 0-day, and a working PoC that anyone could trivially use. We unfortunately identified attacks just a few hours after these releases. For example, we noticed Mirai-based variants being dropped on compromised instances by opportunistic threat actors.
As we assumed, the flaw abuses a third-party open source library, the "xmltooling" dependency, maintained by Shibboleth. This component was indeed vulnerable to a SSRF flaw fixed last June and tracked as CVE-2023-36661 (link for our Vulnerability Intelligence Watch clients).
The vulnerable endpoints, which pass to "saml-server" a XML that can be specially crafted, may include:
- /dana-ws/saml.ws,
- /dana-ws/saml20.ws,
- /dana-ws/samlecp.ws,
- /dana-na/auth/saml-sso.cgi
- /dana-na/auth/saml-logout.cgi
- /dana-na/auth/saml-consumer.cgi
- ...
When processed by xmltooling using the function "XMLToolingFIPS.XMLObject.Signature", the request will give access to the machine to a remote attacker, even if not authenticated. It can be chained with the first command injection 0-day (CVE-2024-21887) directly from within the device (i.e. http://127.0.0.1:8090/api/v1/license/keys-status) in order to bypass the initial XML mitigation.
Compromised assets:
Since the beginning of this crisis, CERT Orange Cyberdefense has been trying to identify vulnerable or compromised instances, using various discrepancies in responses when requesting some files or endpoints (i.e. the 403 "empty" response, the GIFTDEVISITOR webshell, the logo/login.gif fake file, etc.). After analyzing this weekend a compromised vs. a clean instance, we found the legitimate module processing logs (DSlog.pm) had been backdoored and non-sensitive data about the appliance sent to a newly created .txt file. Our team checked remotely for assets displaying this behavior yesterday, and found 670 instances possibly infected using this backdoor.
As a reminder, the original "403 - Forbidden: empty response" scan technique described in the link above, which was used to assess whether the initial XML mitigation is applied or not, should not be used anymore since the patches are available. Instances already upgraded indeed don't have the XML mitigation anymore but are not vulnerable.
A few days ago on Twitter, a researcher explained another technique to remotely check if one instance was backdoored using the SAML 0-day, with the response from another query. We could nonetheless not validate this technique ourselves.
Recommendations:
Our SOC teams followed a risk-based approach to protect our managed clients, and quickly upgraded (or applied the new XML workaround) last week as soon as we learned about the new vulnerabilities. Appliances that showed signs or suspicions of compromise were "snapshoted", disconnected and forensically analyzed, with factory reset of the appliance done before the upgrade.
Last week, CISA urged Federal agencies to disconnect (and factory reset) all their appliances. Ivanti and Mandiant now even recommends 2 consecutive factory resets, to prevent a possible future rollback that would put the device back in its previous compromised state. This makes sure instances are indeed clean, but you should prioritize patching (or at least mitigating) devices sooner vs. factory resetting them later (except in case of compromise or if a target of interest for the Chinese threat actor behind these 0-days).
ICT scans will not detect the whole extent of possible compromises, but still remain the most practical source of detectionin many cases. If:
- your appliance was mitigated early on (around January 11th onward)
- no historical ICT nor external ICT scans showed signs of compromise,
- and no other suspicious behavior (i.e. in IOCs, logs or alerts from security solutions) was found in the rest of the infrastructure,
Then, your device is probably not compromised at this point. We nevertheless recommend you to keep hunting for such signs, including by scanning regularly the appliance with the internal / external ICT, and using the latest available IOCs.
A system snapshot (and extract of relevant memory and log data) is recommended before running the ICT external scan, as it reboots the appliance. If performance issues associated to the logs for unauthenticated requests can be handled, we advise you activate this debug-type level of visibility (which is not on per default), in order to more easily detect exploitation attempts.
If confirmed compromised, you need to roll out your incident response plan, that may include:
- isolating appliance from other systems
- auditing recently created or modified privileged accounts
- revoking credentials including certificates or passwords possibly exposed
- monitoring suspicious authentication attempts
- hunt for anomalies in services or devices connected previously to the appliance
- ...
Update 7, February 1 - Mandiant shares investigation feedback, CISA requests appliances be disconnected by Friday
CISA:
Speed is particularly important in such situation, as more massive exploitation using the latest 0-day (SAML) has possibly again started according to our first analysis. This could be why the US government cybersecurity agency CISA requests all Federal agencies to disconnect appliances by tomorrow Friday (February, 2), and have them back online only after the patch is applied.
Ivanti:
The new XML mitigation provided by Ivanti can be applied to all product versions, and the patches just fix the security issues (i.e. do not include any new features). Another patch for a new branch (numbered 22.5R2.2) was released today. The latest updated external ICT doesn't include IOCs or new detection techniques, but enables to check an instance even running on latest (i.e. patched) versions.
Statistics:
Our latest scan, run on January 31, shows out of 24,986 exposed and responding devices:
- mitigated assets (XML mitigation): 20,674
- vulnerable assets (no XML mitigation): 3,254
- compromised assets (NB: GIFTEDVISITOR webshell only): 684
- 451 of those have the XML mitigation applied, thus possibly after a compromise
- assets leaking internal sensitive data (logo/login.gif): 680
Disclaimer: this doesn't identify the latest webshells discussed below by Mandiant.
Mandiant:
The Google's subsidiary released a technical follow-up blogpost about the malware strains used by the main, original threat actor. The incident provider confirmed they saw highly targeted post-exploitation activity. But the provider did not named any of the victims, nor their countries or sectors. The threat actor managed in some cases to compromise appliances then revert them to a clean state, to evade detection including from the external ICT scan. Attacker tampered the internal ICT files in order to prevent it from running. Some Event IDs are nevertheless provided to detect such modifications in logs, as are other IDs linked to the clearing of system logs done by this APT group.
New malware strains attributed to the Chinese threat actor they refer to as UNC5221, are also detailed:
- a new Perl webshell called BUSHWALK, embedded into a legitimate file (querymanifest.cgi) that enables read and write actions
- a LIGHTWIRE variant found in another legitimate file: "compcheckresult.cgi". Mandiant recommends hunting for this GET request '"/dana-na/auth/url_default/compcheckresult.cgi?comp=comp&compid=%3Cobfuscated%C2%A0command%3E") within web logs, unallocated space, and memory images
- another variant of WIREFIRE, dubbed CHAINLINE, found in a newly created health.py file, is requested through the endpoint: "/api/v1/cav/client/health"
- FRAMESTING webshell (in file category.py) is accessed by the attacker through requests to "/api/v1/cav/client/categories"
- multiple variants of the WARPWIRE credential stealer were also found by the incident responders.
The code of ZIPLINE's strain was further detailed by Mandiant, which listed also the attacker's open-source toolset: Impacket, Iodine, CrackMapExec or Enum4Linux. As they already mentioned 2 weeks ago, a .tar archive masquerading as a randomly generated 10-character CSS file within the directory "/home/webserver/htdocs/dana-na/css/" was used to store stolen information (dump of configuration and cache). As we already knew, stolen data was in some cases appended to legitimate "logo.gif" or "login.gif" file, for the same objective.
Recommendations:
We highly recommend you follow Ivanti's process and apply either the new XML mitigation or ideally directly the patches released yesterday, available for most of the actual currently deployed versions.
We again advise you to run the updated external ICT, but also check for historical logs containing the logs of previously ran internal ICT scans, to identify possible prior compromise. Additional hunting can be done using the network/host-based IOCs (available to our MTD clients through Datalake), detection rules, or analysis of suspicious behaviors directly in logs or on the device. Remember appliances could be configured in active-passive and/or involve cluster of devices. In case of doubt, we can help you assess if you instance was indeed compromised (if so, please contact our Incident Response hotline).
1. Analysis
Update 6, 31/01/2024 - Ivanti first patches out but additional 0-days released, external ICT checks subverted
Vulnerabilities:
Two new 0-days have been announced by Ivanti in connection to the attack investigation here, forCVE-2024-21888 and CVE-2024-21893. The first one enables an attacker to conduct internal (local) privilege escalation, and doesn't change the threat level.
The other one, which is way more critical is a server-side request forgery in the SAML component to bypass authentication. The flaw is actively exploited by the Chinese threat actor in targeted attacks against a limited number of clients. Technical details are embargoed so far, because it may involve a third party library in another software, which could facilitate mass exploitation of exposed assets again once details are revealed by an incident response provider later today or tomorrow.
Attacks:
Exploitation of the vulnerabilities are still ongoing, even though the number of exposed vulnerable appliances greatly decreased over time, with only some hundreds to thousands remaining online. However, we confirmed on January 25 that at least 500 instances were exposing online sensitive data through the "fake" logo.gif file created by the threat actor.
Volexity first talked on January 18 about a Rust-based malware strain for Linux that was leveraged by the threat actor, without providing more details at the time. One researcher published a few days later a few IOCs about it here, then further information about this new strain he calls "KrustyLoader" (because coded in Rust). He even provides a decryption script for retrieving the URLs used by the loader to drop a Sliver backdoor, a well-known advanced post-exploitation tool that our "Managed Threat Intell-detect" service proactively tracks since a long time.
Forensics:
During our multiple CSIRT engagements, we identified that devices compromised with the Metasploit framework often involve the presence of specific artefacts in the "/imgs/Ivanti.zip/" folder. Furthermore, we believe checks with the (even external) ICT solution may in some cases not identify a compromised appliance. Ivanti already mentioned it previously and the US government agency CISA stated on January 30 that attackers managed to subvert the external ICT results to evade detection.
Detection:
The initial vulnerable endpoints are well-known since more than 10 days now, with many security vendors adding signatures for requests made to those endpoints (i.e. Cloudflare, Snort/Suricata, etc.). Picus listed many of them at the bottom of their article. SIEM providers such as Splunk also provided examples of queries to hunt for suspicious requests. However some endpoints tied to the vulnerable path " https:///api/v1/totp/user-backup-code/.." are not well documented by the vendor, and our experts found out that the product does not correctly log some of those requests. Exploitation attempts by attackers before XML mitigation was applied may have left little evidence in logs, and more advanced investigation may be needed.
Remediation:
On January 26, Ivanti announced that unfortunately the first patch will not be rolled according to the planned schedule and actually delayed them for a "few days". Worse, the timetable mentioned per product line was removed from the advisory. But today, January 31, Ivanti started releasing patches fixing the 4 vulnerabilities, in these first versions:
- 9.1R14.4,
- 9.1R17.2,
- 9.1R18.3,
- 22.4R2.2,
- 22.5R1.1.
Unfortunately, the vendor recommends doing a factory reset of devices before patching, with few hours needed for each operation. We nevertheless encourage you to patch appliances as soon as security fixes for your versions get released, even if more complexes than applying the XML mitigation.
If unable to patch, Ivanti also announced a new mitigation XML file(mitigation.release.20240126.5.xml) to protect against the two additional 0-days released today. When available (NDLR: the download link does not work at 2pm Paris time), we highly encourage users to deploy at least this new protection until the devices can be fully patched. This updated workaround blocks in particular SAML authentication, leveraged in CVE-2024-21893:
"The new mitigation will block all SAML communication and authentication. This will have limited functionality impact on customers who use LDAP for authentication" said Ivanti.
On top of the new XML, Ivanti released a new version of their external ICT:
- ICT-V22622 Ivanti Connect Secure Generic Integrity Checker Software 22.xR2
- ICT-V22611 Ivanti Connect Secure Generic Integrity Checker Software 22.xR1
- ICT-V91183 Ivanti Connect Secure Generic Integrity Checker Software 9.x
Any asset exposed on Internet that did not apply one of the mitigation yet should be considered probably hacked. This echoes the latest CISA blog that recommends defenders to keep hunting for sign of compromise in any case. The threat level attributed to this advisory remains at maximum, i.e. 5 out of 5. We will continue to monitor the latest evolutions and provide yet another update tomorrow if necessary.
Update 5, 24/01/2024 - Ivanti mass exploitation spree still ongoing, sensitive data of compromised assets exposed
AssetNote explained again how to leverage the Ivanti vulnerabilities in a blog post here. The researchers confirmed that some old versions such as 9.1R11.4 may not be directly compromised using the publicly available PoC, but presented how to still do it using other vulnerable endpoints.
They also detailed how we (and other security teams) have been checking remotely since more than a week now whether a device was vulnerable, using the different responses from specific endpoints, with blank content replied back for vulnerable instances vs. more verbose for mitigated assets.
Disk decryption
In order to forensically investigate possible compromises, it is possible to retrieve artifacts on the disks of machines running the Ivanti product. Unfortunately, some LILO-based disks are encrypted by the loop-AES encryption software, used to encrypt partitions or files on a Linux system. It works as a kernel module that can be dynamically loaded to create encrypted loop devices. Incident responders at Northwave provide a Python script available on GitHub and the detailed process to decrypt the disks, using QEMU to run an imaged disk. The Python script isn't very optimized, however, a faster Go-based decryptor can be made available if required. Some of the encryption keys already found are currently being shared privately within the CERT community to speed up those forensic investigations.
Sensitive data exposure
In our incident response engagements, we could confirm Volexity's claims that UTA0178 stole data and embedded them in a remotely accessible GIF file named "logo.gif". Moreover, this tactic poses additional security risks for these compromised devices, as massive amount of critical information (passwords, configurations, secret keys, certificates, etc.) are thus currently publicly exposed, with anyone knowing the specific pattern to look for being able to retrieve the information freely. Even worse, some previously compromised devices now appear mitigated (i.e. running the XML workaround) but do still expose these files.
Compromised instances
Censys identified 412 assets still compromised by the credential harvester module (i.e. lastauthserverused.js), but this only shows part of the hacked devices, as hundreds more still also are tampered with the GIFTDVISITOR webshell. HarfangLabexplained how the number of webshells has dropped (because device owners cleaned or took them offline), but that the JavaScript credential harvester count keeps growing, and now surpassed the webshells' one.
Additionaly, Quointell identified yesterday a variant of the webshell leveraged by UTA1078 (also named UNC5221), in order to evade some detections currently using Mandiant's public Yara rule. A new one, less restrictive, is thus provided in Appendix by this vendor. The new webshell was dropped at a new path: ../api/resources/category.py (instead of /visits.py).
Furthermore, automated exploitation will increase even more as a Metasploit module was released on January 20.
Remediation warning
As we already mentioned, Ivanti updated their KB on January 20 to recommend clients not to apply backup configurations to new instances that already have the XML mitigation, as it can deactivate it:
"Important: customers should stop pushing configurations to appliances with the XML in place, and not resume pushing configurations until the appliance is patched. When the configuration is pushed to the appliance, it stops some key web services from functioning, and stops the mitigation from functioning. This only applies to customers who push configurations to appliances, including configuration pushes through Pulse One or nSA. This can occur regardless of a full or partial configuration push."
All IOCs related to the remote-attackers' C2 infrastructure have been added to our Datalake repository. We have started multiple engagements for clients in multiple sectors, and can help you remove doubt if your device is compromised or not. We also continue to try to identify on a regular basis vulnerable, mitigated or compromised devices exposed online. The risk level remains at 5 out of 5 for now.
Updated on : 2024-01-19 16:27:19
Update 4, 19/01/2024 - New details on the UTA0178 attacks exploiting Ivanti 0-days
As GreyNoise publicly mentions here, more threat actors have started using the PoC for CVE-2023-46805 and CVE-2024-21887 (link for our Vulnerability Intelligence Watch clients) publicly released by Rapid7. Most of these campaigns aim at deploying cryptomining or other unsophisticated payloads on vulnerable devices. Volexity researchers also observed such attacks involving XMRIG and several additional Rust-based payloads. Unfortunately, analysis of these malware is still ongoing.
Since Wednesday, we have identified more than 4,000 servers on which the XML mitigation file has still not yet been applied, as well as 1,300 IP addresses that presumably have the GIFTEDVISITOR implant installed (using a remote check). Volexity revealed UAT0178 continued its activity, with now over 2,100 Ivanti Connect Secure VPN devices affected using this particular payload.
Some national CERTs have confirmed us that they have received lists of compromised assets directly from Volexity, and that they have begun notifying compromised assets owners starting Monday. ShadowServer, which uses a different checking methodology, found only 600 compromised instances so far, but also began notifying automatically recipients of their daily reports since Tuesday night.
Furthermore, UTA0178 is making modifications to the built-in Integrity Checking Tool in order to evade detection by organizations seeking to identify if compromised or not. Furthermore, according to them, issues have been identified when bringing new Ivanti Connect Secure VPN appliances back online, leaving them vulnerable again. Indeed, the mitigation may not be present anymore if this one is applied before restoring the device using a backup configuration file. Also, our CSIRT team recommends if ICT was already run multiple times that historic logs for these checks are verified for any signs of mismatch/new files, and not just the latest run.
We do have initiated multiple engagements with customers, but no OCD-managed equipment were compromised, as our SOC teams quickly reacted after receiving the vendor's advisory last week.
The threat level for this advisory nevertheless remains at the maximum level for now, as thousands of servers remain vulnerable and/or compromised, and new threat actors start targeting them. If you need help checking whether your instance is vulnerable or if you suspect it has been compromised, immediately contact our CSIRT through your sales representative (or here).
Updated on : 2024-01-16 17:38:44
Update 3, 16/01/2024 - Exploitation of Ivanti 0-days explodes worldwide and made easier
As we predicted, the exploitation of the Ivanti 0-days tracked as CVE-2023-46805 and CVE-2024-21887 has widely expanded. According to the latest information provided by Volexity, more than 1,700 ICS VPN appliances are now believed to be compromised since January 11. These instances are located all around the world and have been targeted indiscriminately. Indeed, these victims greatly vary in size, ranging from SMBs to Fortune500 companies, across several verticals.
Volexity was able to identify the scope of this mass exploitation by discovering a webshell variant on one targeted system. As a reminder, this strain was first detected on a visits.py legitimate file, in the incident Volexity investigated and detailed initially on January 10. The company added that this variant they call GIFTEDVISITOR was found on 1,700 instances out of roughly 30,000 IP addresses running ICS. Volexity assesses with medium confidence that this mass campaign was launched by UTA0178 (threat actor tracked as UNC5221 by Mandiant). They shared the specific compromised IP addresses with major national CERTs (such as ANSSI in France) for further victim notification.
However, we believe the total number of compromised instances may in fact be even higher as more threat actors have started to try to exploit these 0-days. Volexity for instance identified another threat actor it tracks as UTA0188 that was seen through logs trying to hack an instance that has already deployed the XML mitigation.
Among the close to 15,600 IPs found by Shodan running ICS, we checked and confirmed (when tested today around 2PM):
- 10,600 devices had already been mitigated,
- 4,000 have not the mitigation applied,
- and close to 1,000 have an undetermined status (disconnected instance, connection error, etc.) .
We will try to identify which ones are actually compromised with the implants, but have also started to notify our clients if a still vulnerable asset is discovered.
Furthermore, we believe the mass exploitation campaign will even increase over the next few hours to days, as Rapid7 shared publicly details about the command injection vulnerability here. This may enable medium or even low-level attackers to get easily a "backconnect" from one vulnerable instance. However, exploiting more massively vulnerable instances can be achieved using webshells. On top of it, as we already mentioned yesterday, numerous vulnerability researchers have by now managed to reproduce the flaws.
This researcher also provides some hints on how to decrypt malicious files contained in the Integrity Checker Tool output that are encrypted by Ivanti. (Volexity also mentioned us having this capability). Intelligence on post-exploitation actions were discovered by Ivanti by analyzing ICT outputs from compromised instances. A legitimate file named "cav-0.1-py3.6.egg" is for example known to have been tampered with by the threat actor. But no hash value of the malicious version, nor details on the webshell implanted in it, was provided, but you could check if needed the file last modification timestamp. The system cache and running configuration the threat actor tries to steal were masqueraded in a logo.gif or {random character}.css compressed file.
Finally, Ivanti shared again today the needed recovery steps for impacted servers here.
The threat level thus remains at the maximum level for now. If you need help checking whether your instance remains vulnerable or if you suspect it has been compromised, immediately contact our CSIRT through your sales representative (or here).
Updated on : 2024-01-15 11:40:05
Update 2, 15/01/2024 - Watchtower provides even details on Ivanti 0-days, increased exploitation to be expected
Ivanti updated its security advisory several times since last Friday, introducing the following changes:
- Since January 12th, Ivanti recommends for impacted instances to do a factory reset, on top of resetting passwords (even if MFA was used), investigating service accounts possibly created, and conduct advanced forensic analysis.
- On January 13th, they released a new Integrity Checker Tool version for Azure, fixing a previous version that did not run correctly,
- Ivanti also now states that less than 20 clients have been impacted (instead of 10 previously). This means the exploitation may have been a little less targeted than first hoped, with victims possibly unrelated to each other. But no details on the victimology is available at this point. This number is also most probably not final, as some compromised customers may have not yet detected and/or reported it to Ivanti, in particular since attackers may have hidden their tracks.
Furthermore, more threat actors will soon try to exploit ICS instances, as some details have been provided publicly (and even more will be shared by a company called Watchtower, after Ivanti releases the patches).
Our World Watch expert team provides ongoing analyses and updates on security incidents. Sign up for our newsletter to receive World Watch analysis here: World WatchOn January 13th, Watchtower Labs indeed published a blogpost announcing they managed to reproduce the flaws. They explained they did it by finding out the endpoints blocked by Ivanti in its workaround (i.e. the encrypted XML mitigation file), as differences are visible in responses from vulnerable vs. mitigated instances. This behavior enables anyone with access to an instance (even remotely and without authentication) to confirm whether the XML mitigation was applied or not.
To do so, you may for instance request one of the endpoint located here:
- /api/v1/configuration/users/user-roles/user-role/rest-userrole-1/web/web-bookmarks/bookmark
- /api/v1/configuration/users/user-roles/user-role/rest-userrole1/web/web-bookmarks/bookmark
(this REST API has been introduced in the 8.3R3 Release of Pulse Secure, and exists in the documentation for all the 9.XRX releases).
A nmap script may ease this check for you. The response will include in both cases a 403 Forbidden, but the response content will contain:
- an empty page for a vulnerable instance
- a full HTML content embedding "Access to the Web site is blocked by your administrator. [...]" in the case the XML fix was applied.
Multiple reconnaissance scans have been conducted these last days in order to list currently exposed servers still presumably vulnerable. At this time, using the above methodology, Onyphe believes out of 27,000 online instances they identified:
- 9,000 instances are still vulnerable,
- 15,000 have installed the workaround,
- and the actual status of 3,000 ones remain unknown.
Some associated logs that may be investigated would potentially be:
msg="WEB31809: Access for /api/v1/configuration/users/user-roles/user-role/rest-userrole1/web/web-bookmarks/bookmark is blocked due to patch(Patch 2)."
Finally, the details provided by Watchtower furthermore prove that reproducing the vulnerability is quite trivial, as it took them less than 48 hours to do so. That means other vulnerability researchers have currently but privately also found how to exploit this VPN SSL product. People with the skills to do so may range from vulnerability experts within military or intelligence services, to experienced red teamers or even sophisticated APT groups. Remember also that Chinese threat actors do regularly share 0-day intelligence between them. This is why we expect an increased in exploitation attempts and actual compromises to occur in the next weeks.
Fortunately, Watchtower did not publicly provide yet the full details for opportunistic attackers to exploit the 0-days, preventing mass compromise of vulnerable instances at this point.
Nevertheless, we consider the threat level associated to this advisory to be very high, and increased it to level 5 out of 5. We again advise customers to run the ICT tool, and hunt for signs of compromise using IOCs and other suspicious behaviors on top of quickly applying the vendor's mitigation.
All our Managed Service clients are protected as our dedicated teams have applied the vendor's recommendations starting last Thursday afternoon. No compromise was identified among the hundreds of instances directly managed by Orange Cyberdefense. And our Managed Threat Defense teams did conduct proactive hunting for clients with this service.
Updated on : 2024-01-12 16:34:07
Update 1, 12/01/2024 - Mandiant details 5 malware used by the threat actor behind Ivanti 0-days
After Volexity, Mandiant also shared technical details of the malware associated with the recent attacks attributed to the Chinese cyberespionage cluster UTA0178 (tracked as UNC5221 by Mandiant). Post-exploitation activity involves the use of several custom malware dubbed ZIPLINE, THINSPOOL, LIGHTWIRE, WARPWIRE and WIREFIRE.
- Mandiant detailed the GLASSTOKEN webshell mentioned in the Volexity article. Tracked under the name LIGHTWIRE by Mandiant, this malware is embedded in a legitimate Secure Connect file (a CGI script), and allows the threat actor to intercept specific requests and interpret them as Perl code. It is important to note that this LIGHTWIRE only intercepts requests to "compcheckresult.cgi" that contain the 'comp=comp' and 'compid' parameters.
- THINSPOOL acts as the Perl script to drop the LIGHTWIRE webshells to a legitimate Ivanti Connect Secure file. This allows the threat actor to persist on compromised devices and manipulate Ivanti Connect Secure files or processes in such a way as to avoid detection. However, according to Mandiant's observations, this attempt failed, as it was detected by Ivanti's Integrity Checker Tool.
- WIREFIRE is the Python webshell that functions as trojan horse for a specific component of the Connect Secure appliance (the visits.py file mentioned in our initial advisory). This webshell also has the ability to download files and execute arbitrary commands on the compromised device. In addition, it decodes, decrypts, and decompresses all raw data existing after a GIF (Graphics Interchange Format) header to run as a sub-process. WIREFIRE undoubtedly uses this header to mask or transport malicious data.
- WARPWIRE is the credential collection tool written in Javascript and embedded in a legitimate Connect Secure file, also briefly mentioned by Volexity. Its main purpose is to collect credentials, including usernames and passwords in plain text. Once collected, this information is Base64 encoded and submitted to a command-and-control server using HTTP GET requests.
- ZIPLINE is the only newly detailed malware, and more precisely an ELF passive backdoor that hijacks the accept() function in libsecure.so in order to intercept network traffic and take control of the incoming connection discreetly, without awakening suspicion. Once this has been achieved, the malware carries out various malicious actions, such as gathering information, downloading files, setting up proxy servers and tunneling servers. But no hash of a sample for this strain was provided by Mandiant.
To detect these malware, Mandiant has shared Yara detection rules. In addition, the Google Cloud subsidiary has announced that the threat actors are mainly using compromised Cyberoam VPN appliances for command and control (C2) communications. It is therefore important to integrate these rules into your detection measures, and to check your logs for unusual connection attempts from Cyberoam VPN appliances.
Finally, it is important to remember that Ivanti provides an Integrity Check Tool (ICT) for their customers to check for potential compromise. But such advanced attackers may have tampered with the systems in such a way that this tool is not enough for making sure a compromise happened or not.
ShadowServer do share the exposed Ivanti instances they found in their daily scan reports, and currently identified around 18,000 instances online. The risk associated with this advisory remains at 4 out of 5 on our side.
Initial alert on : 2024-01-11 09:45:07
Two new 0-day vulnerabilities actively exploited against Ivanti Connect Secure VPN
1.1 Executive summary
Ivanti just alerted its customers that two new 0-days in the Ivanti Connect Secure and Ivanti Policy Secure products are being currently exploited in the wild. These vulnerabilities impacts all versions of Ivanti Connect Secure (a VPN solution formerly known as Pulse Connect), Policy Secure and in limited ways Neurons for ZTA gateway. However, at the time of writing, only Connect Secure (i.e. ICS) has been confirmed as exploited by the threat actors. The two vulnerabilities tracked as CVE-2023-46805 and CVE-2024-21887 (link for our Vulnerability Intelligence Watch clients) were chained together in order to provide unauthenticated Remote Code Execution.
Volexity first discovered the 0-days, by identifying early December webshells they call GLASSTOKEN being implanted on a client server through these flaws. The US incident response company dubbed the advanced Chinese nation-state threat actors behind this targeted espionage attack UTA0178.
Finally, according to Censys, at least 30,000 machines vulnerable to these security flaws are exposed on the Internet. But less than 10 customers are known to be affected on Ivanti side so far.
Users of this solution are strongly advised to apply the mitigation measures provided by Ivanti as soon as possible, and patch whenever the vendor will supply fixes (starting end of January). But hunting for any sign of compromise should be done prior to deploying the mitigation, as the provided workaround will not remove the possible unauthorized foothold gained by attackers.
1.2 What you will hear
Ivanti Connect Secure VPN yet again leveraged by Chinese APT for espionage purposes, chaining 2 critical 0-day vulnerabilities.
1.3 What it means
CVE-2023-46805 is an authentication bypass vulnerability, while CVE-2024-21887 is a command-injection security flaw. Chained together, they provide threat actors unauthenticated access to the Ivanti ICS server, enabling them to further gain ground on the network and move laterally, in this case using RDP, SMB and SSH.
These Chinese nation-state hackers are not yet associated to an already known group, thus were numbered “UTA0178″ by the US company Volexity that first warned the vendor in mid-December, after a client got compromised.
The APT deployed in this case custom webshells Volexity named GLASSTOKEN. The level of sophistication and stealth of the threat actors is high, suggesting targeted espionage as the main objective. One recovered artefact was for instance a modified Perl module, executing a Perl script that launched a bash one, aiming at implanting code within an existing CGI script from the ICS VPN, to allow RCE over the Internet through specific parameters supplied by the attacker. The attackers also modified other legitimate components such as a JavaScript from the web app or a Python script called visits.py.
The threat actors were careful to leave as little evidence as possible, including by:
- using Living off the Land techniques,
- erasing logs and the malicious files dropped on the host (such as a SOCKS5 proxy in Python),
- grabbing the Active Directory NTDS file from a backup instead of through the running one,
- collecting credentials from a live Veeam backup instance,
- implanting GLASSTOKEN webshells on multiple Internet-accessible webservers,
- …
We classify this advisory’s threat level as 4 out of 5.
1.4 What you should do
Multiple IP addresses associated to the attackers were recorded by Volexity (and can be found in our Datalake portal, see below), with stolen credentials sent to an attacker-controlled domain infringing Symantec (symantke[.]com). These IOCs should be hunt for, along with any sign of compromise. Various detection strategies can be used such as:
- reviewing suspicious behaviors and logs (requests to remote addresses, connections bypassing usual SSO/MFA providers, etc.)
- checking the known malicious files locations (to do so, the Integrity Checker Tool should be run, but as the attackers may have tampered with the internally supplied one, thus Ivanti recommends re-downloading it from here).
As patches won’t be ready before end of January, Ivanti provided a XML package to mitigate the threat. But this workaround may impact some legitimate features (see the vendor’s advisory or our MVI-Watch dedicated “Solutions” page for the full list).
Orange Cyberdefense’s Datalake platform provides access to Indicators of Compromise (IoCs) related to this threat, which are automatically fed into our Managed Threat Detection services. This enables proactive hunting for IoCs if you subscribe to our Managed Threat Detection service that includes Threat Hunting. If you would like us to prioritize addressing these IoCs in your next hunt, please make a request through your MTD customer portal or contact your representative.
Orange Cyberdefense’s BlackLake service offers the ability to automatically feed network-related IoCs into your security solutions. To learn more about this service and to find out which firewall, proxy, and other vendor solutions are supported, please get in touch with your Orange Cyberdefense Trusted Solutions representative.
2. Appendices :
Updated on : 2024-01-15 11:38:41
Update 2, 15/01/2024 - Watchtower provides more details on Ivanti 0-days, increased exploitation to be expected
2.1 External links
2.2 OCD links
n/a
2.3. IOCs
https://datalake.cert.orangecyberdefense.com/gui/search?query_hash=6001622854ef07759c5843514e52ff7e
2.4. Other
n/a
2.5 mainCategory
mainCategory=Vulnerability
Updated on : 2024-01-12 16:34:04
Update 1, 12/01/2024 - Mandiant details 5 malware used in the UTA0178 operation abusing Ivanti 0-days
2.1 External links
Mandiant: https://www.mandiant.com/resources/blog/suspected-apt-targets-ivanti-zero-day
2.2 OCD links
n/a
2.3. IOCs
Our Managed Threat Intelligence [detect] clients can directly consult and consume the IOCs from this address on our Datalake platform:
https://datalake.cert.orangecyberdefense.com/gui/search?query_hash=6001622854ef07759c5843514e52ff7e
If you’re interested to know more about this OCD managed service, please reach us at team[AT]cert.orangecyberdefense.com, indicating you’re a World Watch beneficiary.
2.4. Other
Yara rules:
rule M_Hunting_Backdoor_ZIPLINE_1 {
meta:
author = "Mandiant"
description = "This rule detects unique strings in ZIPLINE, a passive ELF backdoor that waits for incoming TCP connections to receive commands from the threat actor."
strings:
$s1 = "SSH-2.0-OpenSSH_0.3xx" ascii
$s2 = "$(exec $installer $@)" ascii
$t1 = "./installer/do-install" ascii
$t2 = "./installer/bom_files/" ascii
$t3 = "/tmp/data/root/etc/ld.so.preload" ascii
$t4 = "/tmp/data/root/home/etc/manifest/exclusion_list" ascii
condition:
uint32(0) == 0x464c457f and
filesize < 5MB and
((1 of ($s*)) or
(3 of ($t*)))
}
rule M_Hunting_Dropper_WIREFIRE_1 {
meta:
author = "Mandiant"
description = "This rule detects WIREFIRE, a web shell written in Python that exists as trojanized logic to a component of the pulse secure appliance."
md5 = "6de651357a15efd01db4e658249d4981"
strings:
$s1 = "zlib.decompress(aes.decrypt(base64.b64decode(" ascii
$s2 = "aes.encrypt(t+('\\x00'*(16-len(t)%16))" ascii
$s3 = "Handles DELETE request to delete an existing visits data." ascii
$s4 = "request.data.decode().startswith('GIF'):" ascii
$s5 = "Utils.api_log_admin" ascii
condition:
filesize < 10KB
and all of them
}
rule M_Hunting_Webshell_LIGHTWIRE_2 {
meta:
author = "Mandiant"
description = "Detects LIGHTWIRE based on the RC4 decoding and execution 1-liner."
md5 = "3d97f55a03ceb4f71671aa2ecf5b24e9"
strings:
$re1 = /eval\{my.{1,20}Crypt::RC4->new\(\".{1,50}->RC4\(decode_base64\(CGI::param\(\'.{1,30};eval\s\$.{1,30}\"Compatibility\scheck:\s\$@\";\}/
condition:
filesize < 10KB
and all of them
}
rule M_Hunting_Dropper_THINSPOOL_1 {
meta:
author = "Mandiant"
description = "This rule detects THINSPOOL, a dropper that installs the LIGHTWIRE web shell onto a Pulse Secure system."
md5 = "677c1aa6e2503b56fe13e1568a814754"
strings:
$s1 = "/tmp/qactg/" ascii
$s2 = "echo '/home/config/dscommands'" ascii
$s3 = "echo '/home/perl/DSLogConfig.pm'" ascii
$s4 = "ADM20447" ascii
condition:
filesize < 10KB
and all of them
}
rule M_Hunting_CredTheft_WARPWIRE_1
{
meta:
author = "Mandiant"
description = "This rule detects WARPWIRE, a credential stealer written in Javascript that is embedded into a legitimate Pulse Secure file."
md5 = "d0c7a334a4d9dcd3c6335ae13bee59ea"
strings:
$s1 = {76 61 72 20 77 64 61 74 61 20 3d 20 64 6f 63 75 6d 65 6e 74 2e 66 72 6d 4c 6f 67 69 6e 2e 75 73 65 72 6e 61 6d 65 2e 76 61 6c 75 65 3b}
$s2 = {76 61 72 20 73 64 61 74 61 20 3d 20 64 6f 63 75 6d 65 6e 74 2e 66 72 6d 4c 6f 67 69 6e 2e 70 61 73 73 77 6f 72 64 2e 76 61 6c 75 65 3b}
$s3 = {2b 77 64 61 74 61 2b 27 26 27 2b 73 64 61 74 61 3b}
$s4 = {76 61 72 20 78 68 72 20 3d 20 6e 65 77 20 58 4d 4c 48 74 74 70 52 65 71 75 65 73 74}
$s5 = "Remember the last selected auth realm for 30 days" ascii
condition:
filesize < 8KB and
all of them
}
2.5 mainCategory
mainCategory=Vulnerability
Initial alert on : 2024-01-11 09:45:07
Two zero-day vulnerabilities actively exploited in the Ivanti Connect Secure VPN
2.1 External links
2.2 OCD links
Vulnerability Intelligence Watch: https://portal.cert.orangecyberdefense.com/vulns/59848/solutions
2.3. IOCs
Our Managed Threat Intelligence [detect] clients can directly consult and consume the IOCs from this address on our Datalake platform:
https://datalake.cert.orangecyberdefense.com/gui/search?query_hash=6001622854ef07759c5843514e52ff7e
If you’re interested to know more about this OCD managed service, please reach us at team[AT]cert.orangecyberdefense.com, indicating you’re a World Watch beneficiary.
2.4. Other
Yara rule for UTA0178 webshells from Volexity:
rule apt_webshell_pl_complyshell: UTA0178
{
meta:
author = “threatintel@volexity.com”
date = “2023-12-13”
description = “Detection for the COMPLYSHELL webshell.”
hash1 = “8bc8f4da98ee05c9d403d2cb76097818de0b524d90bea8ed846615e42cb031d2”
os = “linux”
os_arch = “all”
report = “TIB-20231215”
scan_context = “file,memory”
last_modified = “2024-01-09T10:05Z”
license = “See license at https://github.com/volexity/threat-intel/blob/main/LICENSE.txt”
rule_id = 9995
version = 4
strings:
$s = “eval{my $c=Crypt::RC4->new(“
condition:
$s
}
rule apt_webshell_aspx_glasstoken: UTA0178
{
meta:
author = “threatintel@volexity.com”
date = “2023-12-12”
description = “Detection for a custom webshell seen on external facing server. The webshell contains two functions, the first is to act as a Tunnel, using code borrowed from reGeorg, the second is custom code to execute arbitrary .NET code.”
hash1 = “26cbb54b1feb75fe008e36285334d747428f80aacdb57badf294e597f3e9430d”
os = “win”
os_arch = “all”
report = “TIB-20231215”
scan_context = “file,memory”
last_modified = “2024-01-09T10:08Z”
license = “See license at https://github.com/volexity/threat-intel/blob/main/LICENSE.txt”
rule_id = 9994
version = 5
strings:
$s1 = “=Convert.FromBase64String(System.Text.Encoding.Default.GetString(” ascii
$re = /Assembly\.Load\(errors\)\.CreateInstance\(“[a-z0-9A-Z]{4,12}”\).GetHashCode\(\);/
condition:
for any i in (0..#s1):
(
$re in (@s1[i]..@s1[i]+512)
)
}
More Yara rules for ReGeorg or pysoxy here.
2.5 mainCategory
mainCategory=Vulnerability
You may access to this World Watch report on the Threat Defense Center Extranet portal by following the below link :
https://portal.cert.orangecyberdefense.com/worldwatch/839001.