Understanding of Adversary Behavior and Security Threats in Public Key Infrastructures

Public Key Infrastructure (PKI) is designed to guarantee the authenticity and integrity of digital assets such as messages, executable binaries, etc. In PKIs, there are two representative applications: 1) the Web PKI and 2) the Code-Signing PKI. 1) The Web PKI enables entities (e.g., clients and web service providers) to securely communicate over untrusted networks such as the Internet, and 2) the Code-Signing PKI helps protect clients from executing files of unknown origin. However, anecdotal evidence has indicated that adversaries compromised and abused the PKIs, which poses security threats to entities. For example, CAs have mis-issued digital certificates to adversaries due to their failed vetting processes. Moreover, private keys that are supposed to be securely kept were stolen by adversaries. Such mis-issued certificates or stolen private keys were used to launch impersonation attacks. In this regard, we need to have a sound understanding of such security threats and adversaries' behaviors in the PKIs to mitigate them and further to enhance the security of the PKIs. In this dissertation, we conduct a large-scale measurement study in the two representative applications---the Web PKI and the Code-Signing PKI---to better understand adversaries' behaviors and the potential security threats. First, in 1) the Web PKI, we mainly focus on phishing websites served with TLS certificates. From the measurement study, we observe that certificate authorities (CAs) often fail in their vetting process and mis-issue TLS certificates to adversaries (i.e., phishing attackers). Also, CAs rarely revoke their issued TLS certificates that have been compromised. Second, in 2) the Code-Signing PKI, we characterize the weaknesses of the three actors (i.e., CAs, software publishers, and clients) that adversaries can exploit to compromise the Code-Signing PKI. Moreover, we measure the effectiveness of the primary defense, revocation, against the Code-Signing PKI abuses. We find that erroneous revocations (e.g., wrong effective revocation date setting) can pose additional security threats to clients who execute binaries because the revocations become ineffective. Such security threats stem from an inherent challenge of setting an effective revocation date in the Code-Signing PKI and CAs' misunderstanding of the PKI. These findings help Anti-Virus companies and a CA fix their flaws