Two classes of novel TCP exploits and the countermeasures

xviii, 219 p. : ill. ; 30 cm.PolyU Library Call No.: [THS] LG51 .H577P COMP 2007 LuoThis thesis presents two classes of novel TCP exploits and the countermeasures. In the first exploit, we have proposed a new breed of low-rate Denial-of-Service (DoS) attacks, referred to as pulsing DoS (PDoS) attacks, which abuse TCP congestion control mechanisms to throttle a victim's throughput. Comparing with traditional flooding-based DoS attacks, PDoS attacks use much less attack traffic to cause similar damage to TCP flows. Besides TCP, the dominant transport protocol today, the emerging SCTP and DCCP will also be vulnerable to PDoS attacks. On the defense side, we have proposed two new effective schemes to detect PDoS attacks. In the second exploit, we have proposed two novel network timing channels, TCPScript and Cloak, which facilitate stealthy communications in the Internet. By exploiting TCP's flow concept, sliding window, and acknowledgement mechanisms, TCPScript and Cloak provide much higher channel capacity, camouflage flexibility, and reliability than existing covert channels. Since the protocol features exploited by TCPScript and Cloak are widely adopted by modern transport protocols, similar covert channels could be imbedded in other protocols. We have also proposed new detection schemes to uncover TCPScript and Cloak channels. In the PDoS attacks, we have fully exploited TCP congestion control mechanisms to effectively deny TCP flows from using the available bandwidth. Unlike traditional flooding-based attacks, the PDoS attack sends out a train of attack pulses, each of which will cause packet drops at the affected routers. Due to TCP's additive-increase/multiplicative-decrease and timeout mechanisms, the periodic packet losses will cause the TCP victim's throughput to stay at a very low value. We have evaluated the effectiveness of the PDoS attack on popular TCP variants, TCP-friendly protocols, and active queue management schemes based on analytical modeling and test-bed experimentations. Since PDoS attacks could be configured in their intensity and periodicity of the attack pulses, we have also studied the tradeoff between attack damage and attack cost. Subsequently, we have optimized the PDoS attack to achieve the best tradeoff. Finally, we have generalized the PDoS attacks, other low-rate DoS attacks, and flooding-based attacks under a single framework: polymorphic DoS attacks.On the countermeasures, we have designed a two-stage detection mechanism for PDoS attacks and Vanguard for PMDoS attacks. The two-stage detection mechanism is designed to detect PDoS attacks at the network under protection. It employs a wavelet analysis to monitor the variability in the incoming TCP data traffic and outgoing TCP acknowledgment traffic. The second stage is to detect the attack based on change-point detection of the monitored statistics in the first stage. Vanguard, on the other hand, is designed to detect different forms of DoS attacks, i.e., the PMDoS attacks. As a result, Vanguard uses three anomalies for an accurate detection and for reducing false positives and false negatives. We have conducted extensive experiments on a test bed to evaluate their performance in terms of detection accuracy and computational requirement, and have compared them with several detection systems proposed by others. In the second class of TCP exploits, we have fully utilized TCP's protocol features to design more effective network timing channels. The first idea is to exploit TCP's bursty traffic for imbedding covert messages. In particular, we have designed TCPScript to imbed messages in the TCP data burst size. Moreover, we have built into TCPScript additional mechanisms based on TCP acknowledgements to increase its channel reliability. The second idea is to imbed covert messages in the packet-flow combinations which are used in the design of Cloak. Cloak possesses many outstanding properties which cannot be achieved by existing network timing channels, including high channel throughput, full reliability, and very high flexibility. For the proof of concept, we have prototyped TCPScript and Cloak, and have evaluated them in a test bed and PlanetLab. Experiment results have showed that TCPScript and Cloak enjoy better performance as compared with two other network timing channels. On the countermeasures, we have proposed new detection schemes for detecting TCP-Script and Cloak channels which are based on identifying anomalies in the TCP data and acknowledgment traffic. We have evaluated the detection rates of the schemes based on public traces.Department of ComputingPh.D., Dept. of Computing, The Hong Kong Polytechnic University, 200