Manticore Search 引入了向量列的文本分块功能，旨在解决超长文档在向量搜索中的截断问题 1。用户可在表定义中通过 chunk_strategy 参数自动将长文档分割成多个块并分别嵌入 1。该功能提供了五种分块策略：truncate（默认）、mean、fixed、recursive 和 sentence 1。其中 truncate 和 mean 策略生成单个向量，而 fixed、recursive 和 sentence 三种策略则生成多个向量 1。用户还可配置 max_tokens（块大小）、overlap_tokens（相邻块重叠令牌数）和 max_chunks（每文档最大向量数上限）等参数 1。
在 Manticore 英文手册数据集上的基准测试中，该功能显著提升了检索效果 1。深层内容的召回率@5 从 55.1% 提升至 83.3%，平均倒数排名（MRR）也从 0.44 上升到 0.70 1。不过这一改进伴随着性能成本的增加：索引 RAM 消耗增长了 2.5 倍（从 4.2MB 升至 11.7MB），摄入时间延长 4 倍（从 21 秒增至 86 秒） 1。根据测试建议，块大小为 128 至 256 个令牌对技术文档表现最优，较小的块能提供更精准的匹配效果，但会增加生成的向量数量 1。
Manticore Search has introduced text chunking functionality for vector columns, enabling automatic segmentation of lengthy documents into smaller chunks that are individually embedded during table definition 1. The feature addresses a critical limitation where documents exceeding a model's input token window become partially unsearchable due to truncation.
The implementation offers five chunking strategies tailored to different use cases 1. The truncate and mean approaches generate a single vector per document, while fixed, recursive, and sentence strategies produce multiple vectors stored in array columns 1. Users can configure chunk size through max_tokens, set overlap between adjacent chunks via overlap_tokens, and cap the maximum vectors per document with max_chunks 1. Testing on the Manticore English manual—a 189-page dataset containing approximately 298,000 words with documents ranging from brief notes to 39,000-word release logs—demonstrated substantial improvements 1. Recall rates for deep content improved from 55.1% to 83.3% at rank 5, while mean reciprocal rank increased from 0.44 to 0.70 1. The optimization trade-offs include a 2.5-fold increase in index RAM (from 4.2MB to 11.7MB) and a fourfold rise in ingestion time (from 21 seconds to 86 seconds) 1.
Performance varies by embedding model: the all-MiniLM-L6-v2 model supports 512 tokens (approximately 380 English words), while text-embedding-3-small accommodates 8,192 tokens 1. Benchmarking revealed that chunk sizes between 128 and 256 tokens deliver optimal results for technical documentation, producing sharper matches through smaller chunks despite increased vector quantities, while overlaps provide minimal quality improvements for structured text 1.
评论
还没有评论，欢迎留下第一条。