Protection against overflow attacks

Buffer overflow happens when the runtime process loads more data into the buffer than its design capacity. Bad programming style and lack of security concern cause overflow vulnerabilities in almost all applications on all the platforms;Buffer overflow attack can target any data in stack or heap. The current solutions ignore the overflowed targets other than return address. Function pointer, for example, is a possible target of overflow attack. By overflowing the function pointer in stack or heap, the attacker could redirect the program control flow when the function pointer is dereferenced to make a function call. To address this problem we implemented protection against overflow attacks targeting function pointers. During compiling phase, our patch collects the set of the variables that might change the value of function pointers at runtime. During running phase, the set is protected by encryption before the value is saved in memory and decryption before the value is used. The function pointer protection will cover all the overflow attacks targeting function pointers;To further extend the protection to cover all possible overflowing targets, we implemented an anomaly detection which checks the program runtime behavior against control flow checking automata. The control flow checking automata are derived from the source codes of the application. A trust value is introduced to indicate how well the runtime program matches the automata. The attacks modifying the program behavior within the source codes could be detected;Both function pointer protection and control flow checking are compiler patches which require the access to source codes. To cover buffer overflow attack and enforce security policies regardless of source codes, we implemented a runtime monitor with stream automata. Stream automata extend the concept of security automata and edit automata. The monitor works on the interactions between two virtual entities: system and program. The security policies are expressed in stream automata which perform Truncation, Suppression, Insertion, Metamorphosis, Forcing, and Two-Way Forcing on the interactions. We implement a program/operating system monitor to detect overflow attack and a local network/Internet monitor to enforce honeywall policies