Gzip能否成为语言模型？

研究者探索了将gzip压缩算法用于语言建模的可行性1。该方案基于压缩与预测等价性的原理，即每个预测模型本质上是一个压缩器，所有压缩算法都可作为预测模型1。

具体实现中，研究者利用gzip中的DEFLATE算法通过32KiB滑动窗口存储语料库，采用束搜索算法逐字节生成文本1。评分方式为score(candidate) = len(gzip(context + candidate))，压缩长度越小表示预测效果越好1。为解决整数字节长度量化所带来的噪声问题，方案采用束搜索策略，预先查看整个跨度后再提交候选内容1。生成循环包含提示、上下文、搜索和提交四个步骤，每次仅保留最近生成的尾部字节在评分上下文中，以避免不必要地重复复制历史文本1。

虽然gzip生成的文本并不完全连贯，但该实验证实了这一压缩算法能够展现出对文本结构的理解1。

An exploration of using gzip compression as a language model has demonstrated that the compression algorithm can generate text by treating compression and prediction as equivalent tasks.1 The approach is built on the principle that every predictive model is fundamentally a compressor, and all compression algorithms function as prediction models.1

The method leverages gzip's DEFLATE algorithm, which stores data matches within a 32 KiB sliding window to compress information.1 Text generation proceeds byte-by-byte using beam search to navigate the quantization noise inherent in integer byte lengths, allowing the algorithm to preview an entire span before committing to selections.1 The scoring mechanism evaluates candidates by measuring the compressed length when each candidate is appended to the existing context—shorter compression lengths indicate better predictions.1 To prevent repetitive text generation, only the most recently generated trailing bytes are retained in the scoring context rather than recycling the full history.1

While the generated text lacks complete coherence, the results confirm that gzip demonstrates understanding of text structure through this experimental framework.1 The generation loop operates in four stages: providing a prompt, maintaining context, executing the beam search, and committing the selected output.1