Text Ranking and Classification using Data Compression
  A well-known but rarely used approach to text categorization uses conditional
entropy estimates computed using data compression tools. Text affinity scores
derived from compressed sizes can be used for classification and ranking tasks,
but their success depends on the compression tools used. We use the Zstandard
compressor and strengthen these ideas in several ways, calling the resulting
language-agnostic technique Zest. In applications, this approach simplifies
configuration, avoiding careful feature extraction and large ML models. Our
ablation studies confirm the value of individual enhancements we introduce. We
show that Zest complements and can compete with language-specific
multidimensional content embeddings in production, but cannot outperform other
counting methods on public datasets.
