开发者成功在iPhone 18的Apple Neural Engine上部署了Qwen 3.8-27B大型语言模型1。该实现通过向量量化和低比特权重压缩技术，将模型大小压缩至1.5比特每权重(bpw)，同时支持24K的上下文长度1。
在性能表现方面，该模型在iPhone 18的A20处理器上的解码速度达到15-70 tok/s，其中标准场景下速度为15-60 tok/s，在易预测的上下文中可峰值达到70 tok/s1。技术实现上，K/V缓存采用int8量化以节省内存，而Delta网络保持FP16精度1。此外，草稿模型dflash-2采用混合2-4向量量化方案，大小约为2.8 bpw1。
Developers have achieved the deployment of Qwen 3.8-27B, a large language model, on Apple's Neural Engine within the iPhone 18, leveraging advanced compression and quantization techniques.1 The implementation employs vector quantization and low-bit weight compression to achieve a model size of 1.5 bits per weight (bpw), while maintaining support for a 24K context length with int8 quantization applied to the K/V cache and FP16 precision retained for the Delta network.1
The model demonstrates practical inference speeds ranging from 15–60 tokens per second in standard scenarios, with peak performance reaching 70 tokens per second when processing highly predictable contexts.1 The base model, originally compressed to 1.585 bpw, was further refined to 1.5 bpw through vector quantization using 6 bits per 4 weights, while an accompanying draft model (dflash-2) operates at approximately 2.8 bpw through hybrid 2–4 vector quantization.1 The optimization was executed on the A20 processor powering the iPhone 18.1
评论
还没有评论，欢迎留下第一条。