DISCOVERING ANOMALOUS BEHAVIORS BY ADVANCED PROGRAM ANALYSIS TECHNIQUES

As soon as a technology started to be used by the masses, ended up as a target of the investigation of bad guys that write malicious software with the only and explicit intent to damage users and take control of their systems to perform different types of fraud. Malicious programs, in fact, are a serious threat for the security and privacy of billions of users. The bad guys are the main characters of this unstoppable threat which improves as the time goes by. At the beginning it was pure computer vandalism, then turned into petty theft followed by cybercrime, cyber espionage, and finally gray market business. Cybercrime is a very dangerous threat which consists of, for instance, stealing credentials of bank accounts, sending SMS to premium number, stealing user sensitive information, using resources of infected computer to develop e.g., spam business, DoS, botnets, etc. The interest of the cybercrime is to intentionally create malicious programs for its own interest, mostly lucrative. Hence, due to the malicious activity, cybercriminals have all the interest in not being detected during the attack, and developing their programs to be always more resilient against anti-malware solution. As a proof that this is a dangerous threat, the FBI reported a decline in physical crime and an increase of cybercrime. In order to deal with the increasing number of exploits found in legacy code and to detect malicious code which leverages every subtle hardware and software detail to escape from malware analysis tools, the security research community started to develop and improve various code analysis techniques (static, dynamic or both), with the aim to detect the different forms of stealthy malware and to individuate security bugs in legacy code. Despite the improvement of the research solutions, yet the current ones are inadequate to face new stealthy and mobile malware. Following such a line of research, in this dissertation, we present new program analysis techniques that aim to improve the analysis environment and deal with mobile malware. To perform malware analysis, behavior analysis technique is the prominent: the actions that a program is performing during its real-time execution are collected to understand its behavior. Nevertheless, they suffer of some limitations. State-of-the-Art malware analysis solutions rely on emulated execution environment to prevent the host to get infected, quickly recover to a pristine state, and easily collect process information. A drawback of these solutions is the non-transparency, that is, the execution environment does not faithfully emulate the physical end-user environment, which could lead to end up with incomplete results. In fact, malicious programs could detect when they are monitored in such environment, and thus modifying their behavior to mislead the analysis and avoid detection. On the contrary, a faithful emulator would drastically reduce the chance of detection of the analysis environment from the analyzed malware. To this end, we present EmuFuzzer, a novel testing methodology specific for CPU emulators, based on fuzzing to verify whether the CPU is properly emulated or not. Another shortcoming regards the stimulation of the analyzed application. It is not uncommon that an application exhibit certain behaviors only when exercised with specific events (i.e., button click, insert text, socket connection, etc.). This flaw is even exacerbated when analyzing mobile application. At this aim, we introduce CopperDroid, a program analysis tool built on top of QEMU to automatically perform out-of-the-box dynamic behavior analysis of Android malware. To this end, CopperDroid presents a unified analysis to characterize low-level OS-specific and high-level Android-specific behaviors