创业公司 RunAnywhere 宣布开源名为 TrnRT(Trainium RunTime)的大语言模型推理引擎,该项目由来自 YC W26 批次的 Sanchit 发布 1。TrnRT 旨在填补当前软件栈对 AWS Trainium 芯片支持的空白,此前 vLLM Neuron 不支持 GLM 模型,NxDI 已停止开发,且 SGLang Neuron 的实现已被移除,导致缺乏针对 Trn2 的良好支持方案 1。该引擎通过优化内核和内存利用,试图挑战 Nvidia H100/H200 在成本效益方面的传统优势 1。
目前,TrnRT 已在 AWS trn2.48xlarge 实例上实现了对 GLM-5.3 Flash (FP8) 模型的运行支持 1。在半分析(SemiAnalysis)的 AgentX 基准测试中,该系统实现了每分钟 357 万 token(3.57M TPM)的高吞吐量 1。在单箱配置下,使用 8 个 trn2 芯片并配合缓存感知路由器,系统可同时服务 2 个 GLM-5.3 flash 实例;在批量为 12、拥有 32 个代理的情况下,请求速率达到每分钟 28.9 次(28.9 RPM)1。此外,在 16 个代理且命中缓存提示词的场景下,P50 首 token 延迟约为 1.24 秒,解码速度为每秒 35.9 token 1。
技术层面,开发者编写了覆盖整个层的“巨型内核”(megakernel),使得在 batch size 为 8 时解码速度提升了 25% 1。这一优化显著降低了硬件成本,数据显示 trn2.48xlarge 上的每 TB 显存成本不足 8x H200 集群的一半 1。RunAnywhere 已发布初步基准测试结果,并欢迎社区参与贡献以进一步完善该项目 1。
Startup RunAnywhere, a Y Combinator Winter 2026 (YC W26) company represented by developer Sanchit, has released TrnRT (Trainium RunTime), an open-source large language model inference engine designed specifically for AWS Trainium hardware 1. The project aims to challenge the cost-efficiency dominance of Nvidia’s H100 and H200 GPUs by optimizing kernel performance and memory utilization on AWS infrastructure 1. Currently, TrnRT supports the GLM-5.3 Flash model in FP8 precision, running on trn2.48xlarge instances 1.
The release addresses significant gaps in existing ecosystem support for Trn2 chips: vLLM Neuron does not support GLM, NxDI development has ceased, and the SGLang Neuron implementation was removed 1. By introducing this new engine, RunAnywhere seeks to provide a viable alternative for high-throughput inference on AWS chips 1.
Preliminary benchmarks shared with the Hacker News community demonstrate competitive performance metrics 1. On SemiAnalysis's AgentX benchmarks, TrnRT achieved a throughput of 3.57 million tokens per minute (TPM) 1. When serving two GLM-5.3 Flash instances on a single box using eight trn2 devices and a cache-aware router, the system handled a request rate of 28.9 requests per minute (RPM) with 32 agents at a batch size of 12 1. Latency figures indicate a P50 time-to-first-token (TTFT) of approximately 1.24 seconds, with a decode speed of 35.9 tokens per second when running 16 agents with cached prompts 1.
Technically, the engine employs a "megakernel" approach that covers entire model layers, resulting in a 25% improvement in decoding speed at batch 8 1. From a cost perspective, RunAnywhere claims that the memory cost per terabyte on a trn2.48xlarge instance is less than half that of an 8x H200 setup 1. The company invites community contributions to further develop the engine 1.
评论
还没有评论,欢迎留下第一条。