Multi threaded pattern searching of large files using limited memory

Pattern searching and discovery for large files is prohibitively slow and requires large amounts of memory for processing. As the number of patterns to process increase, the amount of memory needed increases exponentially exceeding the resources in a traditional computer system. The solution to this problem involves utilizing the hard drive to save pattern information. A program was created called Pattern Finder which saves patterns, keeps track of how much memory it uses and when that threshold is reached, it dumps the information to the hard drive. The other problem inherent with pattern searching besides limited resources is the amount of processing time it takes to complete. To speed up processing, we implement a multithreaded suffix tree pattern finding algorithm that utilizes multiple processing cores. The goal is to mimic Amdahl’s law by adding more cores and therefore increasing throughput