数据压缩与大语言模型在本质上都是预测问题。[1]Google DeepMind在2023年发表的论文主张语言建模和压缩是同一事物的两种不同观点。[1]运行长度编码、算术编码、霍夫曼编码等经典压缩技术，与现代语言模型使用相同的数学原理——都通过建立更好的概率分布来优化预测。[1]
熵是压缩的基础，代表给定概率集合下单个符号可以达到的最小比特数下限。[1]大语言模型的交叉熵与压缩中的熵使用相同的底层公式，两者都通过降低数据的熵值来实现各自的目标。[1]以order-1模型为例，使用上下文的压缩效果可以将输出减少超过一半，相比未使用上下文的模型。[1]
然而，大语言模型在实际压缩应用中存在显著缺点。[1]由于模型本身可能达到数GB的规模，将其用于HTTP响应压缩会造成严重的计算开销。[1]
An article published on August 11, 2026, examines the fundamental connection between data compression and large language models, proposing that both are essentially prediction problems at their core [1]. The piece traces how compression techniques such as run-length encoding, arithmetic coding, and Huffman coding operate through the same mathematical principles as language models—by constructing superior probability distributions to optimize prediction and reduce data entropy [1].
The analysis reveals that large language models and compression algorithms share identical underlying mathematics, rooted in entropy as the theoretical lower bound for bits required to encode a symbol given a probability distribution [1]. A 2023 paper from Google DeepMind argued that language modeling and compression represent two perspectives on the same phenomenon [1]. The cross-entropy metric used in language models employs the same foundational formula as entropy in compression theory [1]. Practical demonstration shows that compression using order-1 context models can reduce output size by more than half compared to models operating without contextual information [1].
However, the integration of language models into compression workflows faces significant limitations [1]. The computational overhead proves prohibitive, as language models themselves can reach sizes of several gigabytes, making their deployment for HTTP response compression impractical due to substantial processing costs [1].