ARCHIVED STORY

Everybody Wins in MITRE ATT&CK Evaluations

By Fred House, Ryan Delany · April 26, 2021

MITRE Evaluation

By now you've probably read numerous vendor press releases and blog posts claiming they “won” this year's MITRE evaluation. These claims ultimately boil down to the interpretation that the best product is the one that produces the most alerts or captures the most data (telemetry). While there is no doubt that the MITRE evaluation provides a huge benefit to the security community, customers, and the vendors that participate, the resulting metrics are difficult for customers to interpret into practical security value (to the point that analyst blogs are written to interpret vendor blogs). In the case of FireEye Endpoint Security, while we too have numbers to market (e.g., visibility into 100% of the Linux activity—sorry, we couldn’t resist) we feel a more valuable approach is to explain how our product’s capabilities—accelerated by its modular architecture—have enabled us to leverage the last three years of MITRE evaluations to rapidly expand our visibility, breadth, and depth of coverage. Case in point: in less than one year the number of FireEye Endpoint detection engines and modules that contributed to our detection has increased from four to 10. From in-memory detection of malicious PowerShell scripts to detection of malicious lateral movement to dynamic malware analysis in the cloud, Endpoint Security has evolved rapidly using the MITRE evaluation to inform our efforts.

Evaluation Overview

MITRE’s evaluation is a detailed capability assessment of each solution’s ability to detect and respond to techniques used by FIN7 and CARBANAK. It is important to note that the MITRE assessment does not provide any quantitative scoring of the solutions that were evaluated and does not rank vendors.

In this year’s test, MITRE Engenuity used the MITRE ATT&CK® knowledge base to emulate the tactics and techniques of CARBANAK and FIN7, both of which have compromised financial services and hospitality organizations through the use of sophisticated malware and techniques, resulting in the theft of more than $1 billion across hundreds of businesses over the past five years. FireEye has covered FIN7 and CARBANAK extensively in the past, and in fact FIN7 is a FireEye designation dating back to 2015.

For this evaluation, the following FireEye products were used:

• FireEye Endpoint Security 5.0.1 with agent 32.30.12
• Helix Security Platform
• Detection On Demand

Innovation Architecture (IA) Modules

This year’s evaluation gave us the opportunity to put everything we learned from the previous two evaluations (APT3 and APT29) to the test. For FireEye, the real value in these evaluations is taking what we learn and enhancing our product with that knowledge. So how did we do? Consider this: in the 2018 APT3 evaluation 100% of our analytic detection originated from the core engines that ship with FireEye Endpoint Security. In this year’s evaluation, FireEye Endpoint Security modules contributed to 35% of the overall analytic detections, highlighting the practical application of our Innovation Cycle. Table 1 shows a summary of the modules that contributed to our detection in this year’s evaluation, including two of our newest modules, AMSI and Event Streamer.

Module	Version	What it does	Evaluation Contribution	Resources
AMSI	1.0.0	Integrates with the Microsoft Antimalware Scan Interface (AMSI) to produce alerts for a wide variety of scenarios including detection of malicious PowerShell scripts (on-disk and in-memory), suspicious WMI queries, and even in-memory .NET malware	Provided detection of attacks pertaining to suspicious API usage, system discovery, password spraying, file deletion, sandbox evasion, screen captures, obfuscated file usage, UAC bypasses, and more!
Logon Tracker	0.4.4	Enables the investigation of lateral movement within Windows enterprise environments	Detected the use of local and domain accounts, RDP activity, and even the mapping of SMB shares.	Download
Enricher	1.3.5	Adds Mandiant Threat Intelligence information to files by submitting files to FireEye’s cloud sandbox, Detection on Demand, to help determine when a file is malicious and aid in incident response investigations	Enriched detections of tools such as Meterpreter and Mimikatz with FireEye’s intelligence context and submitted them to Detection on Demand to provide additional information.	Download
UAC Protect	1.0.9	Enables alerting and blocking of User Access Control bypass attacks in Windows enterprise environments.	Detected all UAC bypass attacks.	Download Video
Process Tracker	1.2.4	Collects metadata associated with new processes identified on on Windows, Mac and Linux endpoints and streams the data to the Endpoint Security console	Identified the execution of tools such as Meterpreter and Mimikatz and submitted them for further analysis via Enricher.	Download
Process Guard	1.4.1	Prevents credential theft by preventing access to credential data or key material stored within the Windows OS	Detected all attempts to dump OS credentials from LSASS process memory.	Video
Event Streamer	1.1.7	Forwards Windows Event Log data to FireEye Helix Server AND/OR third-party servers using the Syslog protocol.	Forwarded Windows Event log data to Helix to provide visibility into registry modification, service execution, API usage, data archival, and application shimming.	Download

Table 1: FireEye Endpoint Security modules

Detection Highlights

In this section we wanted to share some of the key detection highlights from this year’s evaluation, including the contribution of our newest modules and module features, our coverage of the newly added Linux scope, and our inclusion of cloud-based detection solutions such as Helix and Detection On Demand.

Module Highlights

AMSI Module

The FireEye Endpoint AMSI module integrates with the Microsoft Antimalware Scan Interface (AMSI) to provide visibility into a wide range of system activity, which we then leverage using custom detection logic. Development began on this module shortly after the 2019 MITRE APT29 evaluation to offer more robust detection of obfuscated script-based attacks (both in-memory and on-disk). Table 2 demonstrates just how well this investment has paid off, particularly when you look at the range of ATT&CK techniques detected by the module throughout the evaluation.

ATT&CK Tactic and Technique	Analytic Detections	Telemetry Detections
Execution
Command and Scripting Interpreter	1	1
Native API	1
Discovery
Account Discovery	1
Query Registry	1
Remote System Discovery	2
System Information Discovery	3
System Network Configuration Discovery	2
System Owner/User Discovery	1
Defense Evasion
Deobfuscate/Decode Files or Information	2	1
Indicator Removal on Host	1
Virtualization/Sandbox Evasion	1
Collection
Data from Local System	1
Screen Capture	1
Privilege Escalation
Abuse Elevation Control Mechanism	2
Credential Access
Brute Force	1	1

Table 2: AMSI Module detections

Some of the more compelling detections we observed during the evaluation with our AMSI module included:

• A PowerShell script that decrypts, decompresses, and base64 decodes a registry value into plaintext shellcode (Step 3.B.5)
• A PowerShell script that calls functions related to process injection such as CreateThread and VirtualAlloc (Step 3.B.6)
• An executable that made in-memory WMI queries related to system reconnaissance (Step 12.A.5)
• A PowerShell script that leverages the well-known ComputerDefaults UAC bypass (Step 15.A.5)

Logon Tracker

The first release of the FireEye Endpoint Logon Tracker module, which is used on virtually every Mandiant Incident Response (IR) engagement and was a key component to investigating the lateral movement discussed in this SolarWinds blog post, was available just before the 2019 evaluation. However, at the time it was primarily a data collection and visualization capability thus relegating its MITRE contribution to that of a telemetry collector. Fast forward one year and the module now includes an important addition that contributed to our results in this year’s evaluation: the ability to alert based on custom rules. Table 3 shows the various ATT&CK techniques in which Logon Tracker provided visibility.

ATT&CK Tactic and Technique	Analytic Detections	Telemetry Detections
Command and Control
Proxy		1
Defense Evasion / Persistence
Valid Accounts: Domain Accounts	6
Lateral Movement
Remote Services: Remote Desktop Protocol	2
Remote Services: SMB/Windows Admin Shares	2
Use Alternate Authentication Material: Pass the Hash	1

Table 3: Logon Tracker module detections

As you can see, we had numerous detections for the successful use of domain accounts. To accomplish this, we simply configured Logon Tracker to alert on every successful authentication with a domain account. Wait, what?! No product would ever be configured to alert every time a domain account is used! But in a test lab with no knowledge of the expected behavior, how could you detect when a logon using a domain account is malicious? Practically speaking, you can’t. This highlights the difference between a real-world configuration and a product evaluation configuration. So, how might you really leverage Logon Tracker’s customizable alerting capability to provide real-world detection value? Here are a few real-world examples used by Mandiant Investigators and FireEye customers:

• Alert when a domain account with Domain Administrator privileges logs into a user workstation (note: if your IT Support organization is using Domain Admin accounts to perform remote support stop reading this post now and go remedy the situation)
• Alert when someone RDPs into your organization from a public IP range (your employees are required to connect to a VPN before they RDP into your network, right?)
• Alert when someone RDPs into a Domain Controller from a system not in a high-security VLAN (e.g., a jump box)

Lastly, one of the more interesting pieces of detection telemetry we captured occurred during execution of the Proxy (T1090) technique in step 19.A.3. In this step, the attacker relayed RDP traffic from attacker infrastructure (attack-evals) through the itadmin system to the accounting system. Figure 1 shows Logon Tracker displaying this activity. So, what are we going to do with this information? You guessed it: we have just completed development of an “RDP Tunneling” detection capability for Logon Tracker to be released soon, and perhaps even demonstrated during the 2021 evaluation. In our opinion, this cycle of innovation exemplifies the value and intent of the MITRE evaluation for vendors and customers.

Figure 1: Logon Tracker—detailed view

Linux Coverage

This was the first time that the MITRE ATT&CK evaluations went beyond Windows systems and addressed techniques aimed at the Linux devices that are often used on networks as file, DNS, and web servers. While this evaluation only focused on a CentOS file server, FireEye Endpoint Security is available on many Linux distributions including CentOS, RedHat, Ubuntu, Suse, Open Suse, Oracle, and Amazon Linux ensuring the broadest coverage for enterprise customers. 

Table 4 shows our detection across the 12 Linux tests included in the evaluation. Our IOC and Logon Tracker modules provided the analytic detection, with the product’s forensic capabilities (namely, the Triage Summary and Audit Viewer) providing the telemetry detection, for visibility into all the Linux activity in this year’s evaluation.

ATT&CK Tactic and Technique	Analytic Detections	Telemetry Detections
Collection
Data from Local System	1	2
Discovery
File and Directory Discovery	1	1
Process Discovery	1	2
Remote System Discovery	1	1
Lateral Movement
Lateral Tool Transfer	3	3
Remote Services: SSH	2	1
Persistence
Valid Accounts: Domain Accounts	1	2

Table 4: Linux detections

Detecting “Outside the Box”

Event Streamer and Helix

A highly demanded feature by FireEye customers is the ability to stream the rich telemetry collected by our endpoint agent to Helix or a SIEM of their choosing. The FireEye Endpoint Event Streamer module was developed in 2020 for this purpose with an initial focus on streaming Windows Event Log data. For this year’s MITRE evaluation, we configured Event Streamer to stream Windows Event Log data to Helix where we captured telemetry related to various evaluation steps including execution of malicious PowerShell scripts, registry access and modification, API usage, Windows service state changes, and more. Table 5 shows the wide range of techniques that we received Telemetry Detection credit for using Event Streamer with Helix.

ATT&CK Tactic and Technique	Telemetry Detections
Defense Evasion
Modify Registry	1
Obfuscated Files or Information	1
Discovery
Account Discovery: Domain Account	1
Query Registry	1
Execution
Native API	1
System Services: Service Execution	1
Exfiltration
Archive Collected Data	1
Persistence
Event Triggered Execution: Application Shimming	1

Table 5: Event Streamer module and Helix detections

Detection On Demand

The FireEye Endpoint Process Tracker and Enricher modules were not new in 2020, however, the ability to automatically submit files to our cloud-hosted threat detection service was new. The workflow in Endpoint Security goes like this:

• Process Tracker detects a file that it has not previously observed executing on the endpoint
• Enricher acquires the file and submits it to FireEye Detection On Demand (DOD) for comprehensive threat analysis
• DOD applies static, dynamic, and machine learning (ML) analysis using FireEye’s Multi-Vector Virtual Execution™ (MVX) detection engine
• Enricher dynamically leverages Mandiant Threat Intelligence to obtain additional context on the file
• If the file is deemed malicious by either Mandiant Threat Intelligence or DOD, an alert is generated in the Endpoint Security console

So how did this work during the evaluation, you ask? Well, no endpoint product evaluation would be complete without running Mimikatz, which is a tool widely used by threat actors and red teams alike to acquire administrative credentials. Step 15.A.6 tested OS Credential Dumping (T1003) by running a version of Mimikatz renamed as “samcat.exe”. Process Tracker detected the execution, Enricher acquired the file, and DOD provided static, dynamic, and ML detections. Additionally, Process Guard also detected “samcat.exe”! Figure 2 shows the detection and process graph displayed in DOD, which is accessible from the Endpoint Security console.

Figure 2: Detection On Demand—detailed results view

Summary

This year’s evaluation of FIN7 and CARBANAK provided FireEye the opportunity to validate the efficacy of our modular approach, the breadth of our Linux coverage, and the power of the FireEye platform, including FireEye Endpoint Security, Helix, and Detection On Demand. As was the case in the previous two evaluations, this year’s evaluation has both validated our prior investment and informs our innovation in the year to come.

FireEye fully supports MITRE’s collaborative, open product evaluation process, how it helps inform organizations about the actions adversaries take, and most importantly, how those behaviors affect security efficacy. FireEye’s strong showing demonstrates FireEye’s deep understanding of what it takes to protect our customers and our pedigree of having world-class expertise responding to breaches, the best threat intelligence, and best-in-class product capability.

In case you are interested in all the details, see the full results for FireEye on MITRE’s website.