Detect VMware Aria Operations for Logs exploitation with Dynatrace and DQL

Earlier this week, virtualization services provider VMware alerted customers to the existence of a proof-of-concept (PoC) exploit for a recently patched security flaw in VMware Aria Operations for Logs.

VMware Aria Operations for Logs (formerly known as vRealize Log Insight) is used across enterprises to collect logs and provide analytics. The company recently announced a high-severity vulnerability in an earlier version of the tool. Tracked as CVE-2023-34051 (CVSS score: 8.1), the Aria Operations for Logs vulnerability relates to a case of authentication bypass that could lead to remote code execution. Basically, this means that an unauthenticated malicious actor could inject files into the operating system of an impacted appliance, which can result in remote code execution.

James Horseman from http://Horizon3.ai and the Randori Attack Team have been credited for the discovery and reporting. They have made a PoC for the vulnerability available and published the relevant indicators of compromise (IoC).

This vulnerability is a so-called “patch bypass” for a set of critical flaws that were addressed by VMware earlier this year, also discovered by Horizon3.ai. In their report, they presented how an attacker could use three different CVEs to achieve remote code execution. Since the patch only blocks access to Thrift services by IP and does not fix the other CVEs in VMSA-2023-0001, all an attacker needs to do is spoof an IP address and use the published attack again.

In this blog post, we show how to discover the original attacks toward the Aria Operations for Logs vulnerability using Dynatrace and DQL by finding the IoC-s from the log records.

How to detect VMware Aria Operations Logs exploitation with Dynatrace

Since we’re talking about a patch bypass, we’re looking at the original attack vector described in this blog post. To exploit this vulnerability to gain remote code execution (RCE), the following steps have to be taken by the malicious actor:

1. Create a Thrift client to gain unauthenticated access to the Log Insight Thrift server.
2. Spoof the IP address of the known worker.
3. Create a malicious TAR file containing a directory traversal using a valid PAK file.
4. Using remotePakDownloadCommand, upload the malicious PAK file to /tmp/<filename>.pak.
5. Extract the file using pakUpgradeCommand. This writes the file to where it’s needed on the filesystem.

A technical deep dive about the attack can be found in this article, VMware vRealize Log Insight VMSA-2023-0001 Technical Deep Dive.

Discovering the attack with Dynatrace

To discover this attack, you have to go through the log files that Aria Operations for Logs creates. The Log Insight server stores relevant logs in the file /var/log/loginsight/runtime.log. This log file is used to track all runtime information related to Log Insight. To analyze these logs with Dynatrace and DQL, the logs need to be ingested into Dynatrace beforehand.

Forwarding logs away from the main system is beneficial in case of any kind of successful attack by a malicious actor: if an attacker decides to remove all evidence from the system (including log files), then a security investigator wouldn’t have traces left to analyze. Sending logs to Dynatrace at runtime safeguards the log files for later analysis in a remote and secure location.

Security-related log records of Aria Operations for Logs are structured in the following format:

[2023-10-25 11:28:29.709+0000] ["https-jsse-nio-443-exec-9"/10.153.234.136 DEBUG] [com.vmware.loginsight.web.actions.misc.LoginActionBean] [User login success: vIDM: SAM=myusername, Domain=vmware.com, UPN=myusername@vmware.com]

When a new remote PAK file is downloaded as a tarball, the following log lines are created:

[com.vmware.loginsight.daemon.commands.SystemCommands] [PAK download initiated by node f2449ed5-11ee-45fd-a0a0-6225a33a8ac6] 
[com.vmware.loginsight.daemon.commands.SystemCommands] [Deleting existing PAK file: /tmp/exploit.pak] 
[com.vmware.loginsight.daemon.commands.SystemCommands] [Downloading http://192.168.4.133:8080/exploit.tar to /tmp/exploit.pak] 
[com.vmware.loginstght.daemon.commands.SystemCommands] [Max allowed pack size is 1505916450 bytes] 
[com.vmware.loginsight.daemon.commands.SystemCommands] [Current downloaded size is 21910]
[com.vmware.loginsight.daemon.commands.SystemCommands] [Total downloaded size is 21910]

From these log lines, the best candidate to look for in the logs is the third populated log, Downloading http://192.168.4.133:8080/exploit.tar to /tmp/exploit.pak].

Now, let’s fire up the DPL Architect and create a suitable pattern for this line:

With the created DPL pattern, you can extract the timestamp field ts, the logger class name as class that populated the log lines and the action that was written to the log record. These extracted fields give you the option to filter out the relevant records:

However, there’s another goal to keep in mind: you need all the log records that contain the Downloading statement and URL from where the tarball was loaded and the destination path in /tmp/.

As the blog post states: “The URL and filename will likely be different. However, the filename will always have the format /tmp/<filename>.pak. Also, note that this log may be legitimate. To determine if an attack has occurred, you need to have your administrator evaluate the URL to determine if it is a legitimate download URL.”

Let’s continue with our DPL pattern and extract the full URL where the payload was downloaded and the relevant filenames. Again, we’re opening up the DPL Architect and creating the following pattern:

'Downloading ' (ld (<<('/') [!/.]*:download_file '.tar')):url ' to /tmp/' (ld:filename '.pak'):saved_file

With the created pattern, we’re extracting the url and the tarball’s name as download_file into separate fields using the look-around functionality of DPL. Additionally, we’re extracting the filename from the destination file path as well as the whole saved_file value for simpler analysis.

DPL pattern extracts the following result set from the log record:

Applying this pattern and filtering the records as described in the blog post gives you the following DQL query:

fetch logs
| filter log.source == "/var/log/loginsight/runtime.log"
| parse content, """'[' timestamp('yyyy-MM-dd HH:mm:ss.SZ'):ts '] [' ld '] [' ld:class '] [' ld:action ']'"""
| filter class == "com.vmware.loginsight.daemon.commands.SystemCommands"
| parse action, """'Downloading ' (ld (<<('/') [!/.]*:download_file '.tar')):url ' to /tmp/' (ld:filename '.pak'):saved_file"""
| filter download_file == filename
| fields ts, url, saved_file

The query produces the following result, which you can now pass on to your administrator for manual evaluation.

A longer description of how the VMware Aria Operations for Logs exploitation works can be found in the Horizon3 blog post.

Further reading about how to use DPL Architect for security use-cases can be found at Speed up your security investigations with DPL Architect.

How to mitigate the Aria Operations for Logs vulnerability exploit

To mitigate this exploit, update to the latest version of VMware Aria Operations for Logs by applying the latest patches. Or follow the instructions within the VMSA-2023-0021.

Sources

1. VMware vRealize Log Insight VMSA-2023-0001 IOCs
2. GitHub – horizon3ai/CVE-2023-34051: VMware Aria Operations for Logs CVE-2023-34051
3. VMware Aria Operations for Logs CVE-2023-34051 Technical Deep Dive and IOCs
4. VMSA-2023-0021
5. Alert: PoC Exploits Released for Citrix and VMware Vulnerabilities