Why Is Encrypted Traffic Analysis Key to NDR?

How encrypted traffic analysis works in NDR solutions

NDR solutions employing ETA do not "look inside" the encrypted packet payload. Instead, they focus on the observable characteristics and behavior patterns of the encrypted traffic itself to reveal attacker behaviors.

This is achieved by understanding normal network behavior and flagging deviations that indicate an issue security teams should investigate.

Key techniques include:

• Transport layer security (TLS) fingerprinting (e.g., JA3, JA3S, JARM hashes) from the TLS handshake to identify traffic types and detect malicious activities in encrypted channels
• Extracting and leveraging TLS SNI (server name indication) information to identify malicious or suspicious access
• Reading packet header information for clues about the encrypted traffic
• Identifying anomalies such as unusual session lengths or connection patterns
• Extracting metadata including server certificates, IP addresses, domain names, session duration, and byte counts from the packet header and TLS/SSL handshaking
• Detecting the use of invalid TLS certificates or those issued by free certification providers, which are often tactics used by attackers
• Utilizing unsupervised ML to identify anomalous TLS sessions that could indicate command and control (C2) communications, and supervised ML to pinpoint patterns related to attacker tactics, techniques, and procedures (TTPs), such as remote access tools, reverse shells, or domains used for data exfiltration
• Classifying encrypted sessions based on traffic nature (e.g., remote shell, web browsing, file transfers) using deep neural networks and decision trees

AI-driven threat detection can uncover threats that bypass traditional methods, including novel malware and unknown attack techniques.

Top benefits of using encrypted traffic analysis in NDR tools

ETA enhances the capabilities of NDR solutions in a number of ways:

• Eliminating Blind Spots While Preserving Confidentiality. ETA ensures that the pervasive use of encryption does not create blind spots in threat detection. By analyzing encrypted traffic without decryption, NDR tools can provide visibility into these communications while upholding privacy laws, compliance regulations, and the inherent risks associated with handling sensitive data in plaintext. This means critical data confidentiality and integrity remain intact.
• Operational Efficiency and Resource Optimization. Decrypting and re-encrypting all network traffic demands significant computational resources, which can lead to increased costs, potential performance bottlenecks like increased latency, and new points of failure.
  
  ETA avoids these operational challenges and risks, offering greater benefit at a lower cost and with less effort. It also simplifies deployment as new log servers aren't needed on-premises for data collection and analysis.
• Effective Threat Detection Through Behavioral and Metadata Analysis. ETA operates by examining the observable characteristics and behavior patterns of encrypted traffic, rather than its payload. NDR solutions leverage artificial intelligence (AI) and machine learning (ML) models to analyze these behaviors and uncover anomalies that indicate an attack.
• Reduced Risk of Data Exposure: Handling sensitive data in plaintext after decryption introduces new risks, such as the potential for data storage vulnerabilities, and increased exposure to insider threats or targeted attacks that impersonate insiders with access. ETA inherently prevents token or decrypted data leaks by not performing full decryption.
• Ability to Detect Undecryptable Traffic. Decryption technologies struggle with certain applications that use certificate pinning or hardcoded server certificates (e.g., WhatsApp, digitally signed emails). ETA can still gain insights from such traffic by analyzing the encryption process and protocol information itself, identifying threats that full decryption might miss.
• Broad Attack Visibility Without Logging Gaps. NDR solutions with ETA provide ground truth for threat detection and illuminate all network activity to and from any asset on monitored segments, effectively eliminating blind spots. The network is an immutable source of truth that cannot be tampered with or disabled by attackers, unlike logs and endpoint agents.

Limitations of Full Decryption

Full decryption is not always feasible or effective. There is no universal standard for encryption, and attackers may use custom or old ciphers. In addition, decryption technologies struggle with applications using certificate pinning or hardcoded server certificates.

Furthermore, ETA can detect threats that full decryption might miss, such as invalid TLS certificates, by analyzing the encryption process and protocol information itself.

Notably, ETA generally cannot detect attacks leveraging encrypted Microsoft protocols such as MSRPC, SMB, Kerberos, or WinRM, where full decryption is often needed for detailed analysis. However, for a significant portion of threats, ETA provides crucial visibility and detection capabilities without the overhead and risks of universal decryption.

Get advanced encrypted traffic analysis with Trellix NDR

Since 95% of network traffic is encrypted, traditional deep packet inspection struggles to see threats. Trellix Network Detection and Response uses advanced ETA to identify threats without decryption, maintaining data confidentiality.

Key features include:

• Certificate and SSL Analysis. Identifies malicious certificates (GTI Certificate Reputation); flags suspicious SSL behaviors like self-signed or expired certificates and weak ciphers (SSL Anomaly Detection); and offers deep visibility via JA3/JA3S/JARM fingerprinting
• Communication Pattern Analysis. Detects connections to newly registered domains; identifies covert command-and-control channels via DNS Tunneling Detection (ML-powered); and flags anonymized communications with ToR Activity Detection

These capabilities use behavioral analysis, not content inspection, ensuring privacy and comprehensive threat visibility.