Date of Completion

12-1-2015

Embargo Period

12-1-2015

Keywords

malware, virtualization, virtual machine introspection, security, analysis, hypervisor

Major Advisor

Laurent Michel

Co-Major Advisor

Aggelos Kiayias

Associate Advisor

Bing Wang

Associate Advisor

Alexander A. Shvartsman

Associate Advisor

Bryan D. Payne

Field of Study

Computer Science and Engineering

Degree

Doctor of Philosophy

Open Access

Open Access

Abstract

Malware is one of the biggest security threat today and deploying effective defensive solutions requires the collection and rapid analysis of a continuously increasing number of samples. The collection and analysis is greatly complicated by the proliferation of metamorphic malware as the efficacy of signature-based static analysis systems is greatly reduced. While honeypots and dynamic malware analysis has been effectively deployed to combat the problem, significant challenges remain. The rapidly increasing number of malware samples poses a particular challenge as it greatly inflates the cost of the hardware required to process the influx. As modern malware also deploys anti-debugging and obfuscation techniques, the time it takes to formulate effective solutions is further exacerbated. There is a clear need for effective scalability in automated malware collection and analysis. At the same time, modern malware can both detect the monitoring environment and hide in unmonitored corners of the system. It has also been observed that malware modifies its run-time behavior to lead the analysis system astray when it detects a monitoring environment. Consequently, it is critical to create a stealthy environment to hide the presence of the data collection from the external attacker. Such systems also need to isolate critical system components from the executing malware sample while keeping the concurrent collection and analysis sessions separate. Furthermore, the fidelity of the collected data is essential for effective dynamic analysis. As rootkits now employ a variety of techniques to hide their presence on a system, the broader the scope of data collection, the more likely the analysis will reveal useful features. Over the last decade hardware virtualization has been proposed to develop such tools with promising results. In this dissertation we present a systematic evaluation of hardware virtualization as an underlying technology to construct effective malware collection and analysis systems. The evaluation is realized via the combination of four distinct objectives such systems need to fulfill: scalability, stealth, fidelity and isolation.

Download

Share

COinS