3 ways to monitor encrypted network traffic for malicious activity

Security experts have been screaming at you for years to encrypt all network traffic. They have a point: Making a secure configuration the default configuration is an obviously good idea. Both the standards and products that implement encryption are very mature. There’s no reason not to!

Well, not entirely. Like everything else in engineering, encrypted traffic presents trade-offs. A big one is that it makes network traffic less transparent to your own security people and monitoring systems. How are you supposed to check network traffic for malicious programs and problematic content?

The short answer is that you can’t. “Deep packet inspection” is not an option. The longer answer is that you can inspect traffic at the endpoints where encryption and decryption are performed and that you can learn a lot just from network traffic metadata, the information in the headers that tell the network where the packet came from and is supposed to go to.

According to Cisco, encrypted traffic nearly doubled from 21 percent in 2015 to 40 percent in 2016. The percentage of encrypted internal enterprise traffic is surely growing rapidly, as enterprise products, such as Microsoft Exchange, are increasingly configured by default to encrypt all traffic.

Most of the network analysis to find malicious traffic in a sea of legitimate encrypted traffic is performed by any decent host- or network-based intrusion and detection systems (IDS/IPS). However, it’s good to be able to go beyond what your tools do and understand your own traffic. The following looks at ways you can do that for protocol-level encryption, not application-level encryption, like that supported in Microsoft Office for data files, nor obfuscation techniques like steganography, which a malicious actor might use to sneak data past your prying eyes.

1. Use network anomaly detection tools

If you can’t look at the actual packet content, you have to monitor traffic flow for network anomalies. What constitutes abnormal network activity? To know this, you need to know what constitutes normal activity. For example, connections between machines that do not normally connect, either internally or between an inside machine and an unknown outside system are inherently suspicious.

Unusual use of TCP/UDP ports is another behavior worth monitoring. Use Dave’s Port List to look up well-known and non-so-well-known ports. It’s not just the use of unusual ports, but the use of ports for unusual applications, such as plain text traffic on port 443, which is reserved for Transport Layer Security (TLS).

These tasks are too complicated and detailed for hands-on work, and a large battery of security products attempt to detect network behavior anomalies, including IBM’s QRadar, Juniper Sky Advanced Threat Protection, even the open source Snort IPS. The most sophisticated products, like Cisco’s Encrypted Traffic Analytics, integrate the monitoring with intelligence services tracking anomalous behavior in systems across the world.

When you need to dig into the traffic to see the gritty details, you’ll need a network analysis tool. Wireshark is essential, but for HTTP/HTTPS I have had a lot of luck with Fiddler. This article by Fiddler author Eric Lawrence explains how to look at metadata and other tricks for examining TLS traffic.

Another interesting tool set is JA3 and JA3S from researchers at Salesforce. It “fingerprints” TLS connections, exposing more details of the communications and the TLS implementations used by the parties to the connection.

2. Use SSL/TLS proxy servers

One possibility for making a lot, if not all, of your encrypted traffic inspectable is a Secure Sockets Layer (SSL)/TLS proxy server. Communications, including encrypted communications, go through the proxy server, which accepts the encrypted connection on one end, decrypts the traffic, performs some operation, then reencrypts and sends the traffic to the destination. The operations the proxy server performs can include security operations, such as malware scanning and blocking prohibited sites. Many third-party security products operate as SSL/TLS proxies.

Proxy servers operating in this fashion add complexity to an already complex network configuration. While they can actually accelerate traffic through caching, they also have the potential to slow it. An alert from the CERT Coordination center at the U.S. Department of Homeland Security in 2017 warned that many of the products in this space do not properly validate certificates or forward error conditions. Many security experts, like Will Dormann at Carnegie Mellon, argue that SSL inspection is not worth the risks that it introduces.

3. Be prepared for non-TLS encryption

The traffic legitimately encrypted (at the level of network packets) is typically done so with SSL/TLS. You might encounter other encrypted protocols. Some might be legitimate, but others have no place on your network.

At the top of the list of ambiguous protocols is Secure Shell (SSH). A lot of administration and development work is done over SSH. The problem is that SSH is also exactly what the bad guys would use. There’s no magic bullet in this case; you have to allow SSH, although perhaps only for certain users in certain network segments. You also need to scrutinize SSH traffic that doesn’t conform to legitimate profiles.

If you use Windows terminals, you will likely see a lot of Remote Desktop Protocol (RDP) for Windows Terminal Server and Independent Computing Architecture (ICA) for Citrix servers on your network. These are encrypted protocols, usually using TLS, but they will likely be on different TCP ports and exhibit other differences. Someone in the organization should have control of the endpoints and the ability to inspect their activity.

On the dodgier side is the User Datagram Protocol (UDP)-based Quick UDP Internet Connections (QUIC), designed as a lower-latency replacement for TLS. The HTTP/3 standard incorporates QUIC, but running on UDP limits its application. For instance, firewalls typically block UDP from any ingress. This is all still rather cutting edge; you might not see it at all.

Beyond dodgy is Tor, the protocol of the dark web. It uses multiple nested levels of encryption and you would be perfectly justified in blocking it unless you have some need for the additional privacy it provides.

The upsides of TLS encryption — authentication, encryption and message integrity — are too good to miss. All that encryption makes some legitimate security practice more difficult, but it’s probably not enough of a problem to leave your traffic unencrypted, the equivalent of leaving your network’s pants down. Whether through metadata or at the endpoint, you still have plenty of tools to protect your users and network.