Symbol Ranking Text Compression with Shannon Recodings

In his work on the information content of English text in 1951, Shannon described a method of recoding the input text, a technique which has apparently lain dormant for the ensuing 45 years. Whereas traditional compressors exploit symbol frequencies and symbol contexts, Shannon's method adds the concept of "symbol ranking", as in `the next symbol is the one third most likely in the present context'. While some other recent compressors can be explained in terms of symbol ranking, few make explicit reference to the concept. This report describes an implementation of Shannon's method and shows that it forms the basis of a good text compressor. Keywords text compression, Shannon, symbol ranking Category E.4 1. Introduction In 1951 C.E. Shannon published his classic paper on the information content of English text, establishing the well-known bounds of 0.6 -- 1.3 bits per letter [Shannon 51]. What is perhaps less recognised is the method by which he obtained those results, and it is that..