Compression of index term dictionary in an inverted-file-orientated database: Some effective algorithms

A new method of index term dictionary compression in an inverted-file-orientated database is discussed. A technique of word coding that generates short fixed-length codes obtained from the index terms themselves by analysis of monogram and bigram statistical distributions is described. Transformation of the index term dictionary into a code dictionary preserves a word-to-word discrimination with a rate of three synonyms per 1300 terms, at compression ratio up to 90% and at low cost in terms of the CPU time expenditure. When applied in computer network environment, it offers substantial savings in communication channel utilization at negligible response time degradation. Experimental data for 26,113 index term dictionary of the New York Times Info Bank available via a computer network are presented.