华为发布了名为MACaQ(Model-agnostic quantization beyond text)的内部量化工具,旨在优化大语言模型及多模态模型的本地推理效率 1。该工具支持在llama.cpp、vLLM、Diffusers和PyTorch等框架下对文本、图像、视频和音频模型进行量化 1。
在性能指标方面,MACaQ宣称能以极低内存消耗快速完成超大参数模型的量化处理 1。具体数据显示,使用仅32.5 GB设备内存,可在64分钟内量化一个122B参数的模型 1。相比MR-GPTQ和DuQuant++,MACaQ具有更低的困惑度(perplexity),或在Llama-3-8B上以缩短11倍的量化时间达到最佳困惑度水平 1。
此外,华为还发布了辅助工具MACaQ-R,用于精修现有的GGUF模型以提高质量并保持与标准llama.cpp的兼容性 1。该项目采用Apache License 2.0许可证开源 1。相关硬件支持方面提到了CANN内核(ops-nn #12568, ops-tilelang #44),用于Ascend芯片加速 1。
Huawei has released MACaQ, a model-agnostic quantization tool designed to optimize local inference efficiency for large language models and multimodal architectures. 1 The utility supports the quantization of text, image, video, and audio models while maintaining compatibility with mainstream frameworks including llama.cpp, vLLM, Diffusers, and PyTorch. 1 It is distributed under the Apache License 2.0. 1
The tool claims high performance in resource-constrained environments, demonstrating the ability to quantize a 122-billion-parameter model within 64 minutes using only 32.5 GB of device memory. 1 In comparative benchmarks, MACaQ achieves lower perplexity than MR-GPTQ and DuQuant++, or reaches optimal perplexity levels on Llama-3-8B with a quantization time that is 11 times shorter. 1 To support hardware acceleration, the release includes CANN kernels for Ascend chips, specifically referenced in ops-nn #12568 and ops-tilelang #44. 1 Additionally, Huawei introduced MACaQ-R, a companion tool intended to refine existing GGUF models to improve quality while preserving standard llama.cpp compatibility. 1
评论
还没有评论,欢迎留下第一条。