Architectural support for autonomic protection against stealth by rootkit exploits

Operating system security has become a growing concern these days. As the complexity of software layers increases, the vulnerabilities that can be exploited by adversaries increases. Rootkits are gaining much attention these days in cyber-security. Rootkits are installed by an adversary after he/she gains elevated access to the computer system. Rootkits are used to maintain a consistent undetectable presence in the computer system and help as a toolkit to hide all the malware activities from the system administrator and anti-malware tools. Current defense mechanism used to prevent such activities is to strengthen the OS kernel and fix the known vulnerabilities. Software tools are developed at the OS or virtual machine monitor (VMM) levels to monitor the integrity of the kernel and try to catch any suspicious activity after infection. Recognizing the failure of software techniques and attempting to solve the endless war between the anti-rootkit and rootkit camps, in this thesis, we propose an autonomic architecture called SHARK, or Secure Hardware support Against RootKits. This new hardware architecture provides system-level security against the stealth activities of rootkits without trusting the entire software stack. It enhances the relationship of the OS and hardware and rules out the possibility of any hidden activity even when the OS is completely compromised. SHARK proposes a novel hardware manager that provides secure association with every software context making use of hardware resources. It helps system administrators to obtain feedback directly from the hardware to reveal all running processes. This direct feedback makes it impossible for rootkits to conceal running software contexts from the system administrator. We emulated the proposed architecture SHARK by using Bochs hardware simulator and a modified Linux kernel version 2.6.16.33 for the proposed architectural extension. In our emulated environment, we installed several real rootkits to compromise the kernel and concealed malware processes. SHARK is shown to be very effective in defending against a variety of rootkits employing different software schemes. Also, we performed performance analysis using SIMICS simulations and the results show a negligible overhead, making the proposed solution very practical.M.S.Committee Chair: Lee, Hsien-Hsin; Committee Member: Blough, Douglas; Committee Member: Copeland, Joh