一项在线计算器工具近日发布，可预测不同Mac机型运行各类大语言模型的能力和速度1。该工具基于两个核心公式运作：首先通过参数数量计算模型是否能装入统一内存，其次利用内存带宽除以权重大小的方式预测生成速度1。
工具开发者通过在M4和M5 Pro机型上进行实际测量，建立了预测精度达90.8%的模型1。以Llama 3.3 70B为例，该模型在M4 Pro 64GB配置下可正常运行，其权重为42.8 GB，加上8K tokens上下文约1 GB以及macOS系统占用后需用掉53.2 GB的可用内存，生成速度约为5.7 tokens/sec1。内存计算采用参数数量（十亿）×比特数÷8的公式，其中Q4_K_M量化平均4.85比特，Q8_0约8.5比特，FP16为16比特1。
从硬件支持角度看，M4基础款内存带宽为120 GB/s，而M5 Pro达到307 GB/s1。生成速度的预测公式为内存带宽÷权重大小加3.3毫秒固定开销，在测试中达到理论带宽的90.8%，预测精度误差控制在4.6%以内1。70B模型在Q4_K_M量化下需约56 GB内存，包括43 GB权重、15%运行时开销、1-2 GB KV缓存及3 GB系统预留1。对于想要体验该配置的用户，M4 Pro Mac mini 64GB的租赁价格为$197/月（年付）或$286/月（月付）1。
A developer has created an online calculator tool that predicts which Mac models can run various large language models and at what speed 1. The tool employs two core formulas: one to determine whether a model fits within a Mac's unified memory based on parameter count, and another to estimate token generation speed using memory bandwidth and weight size 1.
The calculations were validated through actual measurements on M4 and M5 Pro machines, achieving a prediction accuracy of 90.8% 1. For instance, Llama 3.3 70B can run on an M4 Pro with 64 GB of memory at approximately 5.7 tokens per second 1. The 70B model's 42.8 GB of weights, combined with roughly 1 GB for an 8K token context window and macOS system overhead, consumes approximately 53.2 GB of the available 64 GB 1.
The memory requirement formula multiplies parameter count in billions by bit depth and divides by 8, with Q4_K_M quantization averaging 4.85 bits and Q8_0 using approximately 8.5 bits compared to 16 bits for FP16 1. Generation speed is calculated by dividing memory bandwidth by weight size, plus a fixed 3.3 millisecond overhead, with M4 base models providing 120 GB/s of bandwidth and M5 Pro reaching 307 GB/s 1. The tool's predictions stay within 4.6% error margin of actual performance 1. For reference, an M4 Pro Mac mini with 64 GB costs $197 per month on annual billing or $286 per month on month-to-month rental 1.
评论
还没有评论，欢迎留下第一条。