Cartesia Research 于2025年提出了分层网络（H-Nets）概念，旨在解决传统语言模型依赖手工设计分词器的问题 1。近期发布的《From Tokens to Bytes: Scaling Hierarchical Networks》研究系列第一部分显示，H-Nets通过动态、端到端的分块替代传统的BPE等静态分词方式，在计算效率上显著优于基线Transformer模型 1。该架构不仅在语言建模中表现突出，还展示了作为通用基础架构的潜力，其初步应用成果已扩展至DNA序列和视觉数据等非语言领域 1。
在具体的性能对比中，1阶段H-Nets与标准Tokenized Transformer相比，提供了高达3-5倍的计算效率提升 1。进入2阶段实验后，与拥有3B参数的开源匹配模型SmolLM3相比，H-Nets仅需约1/10的计算量和数据量（从11.1T tokens降至~1T tokens）即可达到相同性能 1。这种优势在编程任务中尤为明显：在训练分布内的编程语言上，H-Nets的计算倍率中位数为2.04x，而在外分布（OOD）编程语言中则升至3.10x 1。此外，H-Nets展现出“上下文内分词”等新涌现行为，能够在推理过程中动态调整分块策略 1。在生物学领域，针对DNA序列的处理实现了6-7倍的计算增益，并能发现如密码子网格等自然生物结构 1。
Cartesia Research has released the first part of a research series on hierarchical networks (H-Nets), exploring their scaling laws and qualitative behaviors in language modeling. 1 The study indicates that H-Nets, a concept proposed by the firm in 2025 to address the limitations of handcrafted tokenizers in traditional language models, offer significant computational efficiency improvements over baseline Transformer architectures by employing dynamic, end-to-end chunking instead of static methods like Byte Pair Encoding (BPE). 1
In controlled comparisons, single-stage H-Nets demonstrated up to 3–5 times greater computational efficiency than standard Tokenized Transformers. 1 Further experiments with two-stage H-Nets revealed even more substantial gains; when compared against the open-source SmolLM3 model with 3 billion parameters, the H-Net architecture achieved equivalent performance using approximately one-tenth of the compute resources and training data, reducing the required volume from 11.1 trillion tokens to around 1 trillion tokens. 1 The architecture also showed robust generalization capabilities, with a median compute multiplier of 2.04x for programming languages within the training distribution, which increased to 3.10x for out-of-distribution (OOD) programming languages. 1
Beyond traditional text processing, the research highlights emerging behaviors such as "in-context tokenization," where H-Nets dynamically adjust their chunking strategies during inference. 1 The potential for H-Nets as a universal foundation architecture was further evidenced by preliminary applications in non-language domains, including visual data and biological sequences. 1 In tests involving DNA sequences, the model achieved a 6–7 fold increase in computational efficiency and successfully identified natural biological structures, such as codon grids. 1
评论
还没有评论，欢迎留下第一条。