How to Defeat Advanced Malware: New Tools for Protection and Forensics (2015)

Chapter 4. Protection’s Weak Link

Abstract

A selection of existing defensive security technologies are analyzed, including antivirus systems, host intrusion prevention systems, desktop firewalls, desktop virtualization systems, patch management solutions, and application whitelisting solutions.

Keywords

sandboxing

antivirus systems

host intrusion prevention systems

desktop firewalls

desktop virtualization systems

patch management solutions

application whitelisting solutions

hypervisor

virtualization

honeypots

In response, an array of defensive security technologies has been developed that aims to complement traditional detection-centric approaches. They include antivirus systems, host intrusion prevention systems, desktop firewalls, desktop virtualization systems, patch management solutions, and application whitelisting solutions. Before we continue, it is worth reviewing these solutions before introducing what we believe to be the cyber-security panacea.

Antivirus (AV) systems detect malware by using signatures that are developed from samples of attacks that have successfully compromised other users. The addition of heuristics and cloud-based lookups has decreased the time needed for AV systems to detect known attacks, but with over 3 billion unique pieces of malware discovered in 2011 alone, today’s attackers have little problem avoiding these systems.

Host intrusion prevention systems (IPS) attempt to detect and block malicious attacks by comparing the behavior of vulnerable applications with a pattern that could indicate “malicious behavior.” The shortcomings of this technology are that malicious and benign code can perform the same types of operations within an endpoint, and singling out the behavior of a single piece of software can be challenging. A host IPS system that is tuned to be effective against unknown malware will also block many unknown but benign software functions leading to user dissatisfaction and an avalanche of corporate help-desk calls. Host IPS is often disabled or tuned to the point that malware is no longer blocked in reaction to these problems.

Desktop firewalls protect the host system by blocking low-level network requests to specific processes within the endpoint. Desktop firewalls do not provide any protection for the most risky applications like the web browser or opening files and attachments, as these processes must be able to communicate with the outside world to function.

Application whitelisting solutions restrict end users from using “nonapproved” programs on their systems. This approach typically has a large impact on user productivity that often results in users finding “workarounds” such as performing critical tasks on mobile or home products. Application whitelists provide no protection from attacks targeted at the “approved” program level which remain vulnerable to zero-day or targeted attacks routinely delivered within the content the applications are tasked with processing.

Patch management solutions attempt to address the root cause of security exploits by providing fixes or “patches” to the underlying vulnerabilities in the programs that are at risk. Unfortunately the sheer scale and attack surface of today’s operating systems and application suites provides endless vulnerabilities. Organizations spend huge amounts of time and money testing and deploying patches in an endless attempt to keep their systems secure with little impact on the number or frequency of successful attacks.

Although adding layers of security to the endpoint is intuitively appealing, it has downsides: It negatively impacts user experience, and more importantly, the security chain is only as strong as its weakest link – the OS kernel. All threat detection/prevention tools depend on the continued integrity of the kernel and are easy to bypass if the kernel can be compromised via a novel exploit – for example, a zero day. Unfortunately critical vulnerabilities in OS kernels are being discovered at an alarming rate.

4.1. Desktop virtualization does not secure the endpoint

In recent years, the growth of desktop virtualization has led to new challenges in endpoint protection. Agents that are deployed on physical Windows desktops do not function well in virtual desktops hosted on a hypervisor. Endpoint Protection Platform (EPP) suites are disk I/O heavy, and on a server running scores of VMs, this leads to collapse of the storage infrastructure and low VM/server density. As a result, each of the major vendors has had to rearchitect its EPP suite for virtualized environments. More importantly, however, it has led to the realization that the virtual infrastructure vendor has a key role to play in endpoint protection, since only the hypervisor has absolute control over all system resources: CPU, memory, storage, and network I/O, for all guests on the system.

Since all products for virtualized environments are in their earliest stages of development, the security of mission critical workloads or virtual desktops on virtual infrastructure is weak, since every compromise that is possible on a physical desktop can be achieved on a virtual one. Of note is a recent NIST study¹ in the area of security for fully virtualized workloads, which notes: “Migrating computing resources to a virtualized environment has little or no effect on most of the resources’ vulnerabilities and threats.”

Virtualization technology, however, will be the key to the delivery of the next generation of security, since a hypervisor can provide a new (more secure) locus of execution for security software. The hypervisor has control over all system resources (CPU, memory, and all I/O) and is intimately involved in the execution of all guest VMs, giving it an unparalleled view of system state and a unique opportunity to provide powerful insights into the security of the system overall. Since the hypervisor relies on a much smaller code base than a full OS, it also has a much smaller attack surface. Finally, it has an opportunity to contain malware that does successfully penetrate a guest, within the VM container. Ultimately, the hypervisor provides a new, highly privileged runtime environment with an opportunity to provide greater control over endpoint security. Bromium is the only vendor to specifically exploit virtualization to both protect endpoints and detect new attacks.

4.2. Detection and isolation using VMs

Many security vendors have attempted to use virtual machines as sacrificial run-time instances or “honeypots.” Traffic entering the system (an endpoint or a network) is first directed to a sacrificial VM containing the operating system and its applications. Although it is attractive, this approach has drawbacks: It relies on an attack occurring, and being detected in the honeypot before the traffic is passed on to its intended recipient. But most malware can detect that it is running in a virtualized environment, and modify its behavior to avoid detection.

A degenerate form of this approach relies on isolating an entire application, such as a web browser, in a VM to contain attacks. However, application performance suffers, and the approach lacks granularity: A successful but undetected attack from a single site can compromise all subsequent browser tabs and sites visited, including those that access trusted intranet and SaaS applications.