TokenTown是一款交互式可视化工具，以虚拟城市的形式直观展示大语言模型的运作机制[1]。该工具在浏览器中实时模拟了Transformer模型的完整处理流程，包括分词、嵌入、位置编码、多头注意力、前馈网络等核心环节，通过等距城市地景的比喻使复杂的计算过程变得易于理解[1]。
为了便于演示和交互，TokenTown采用了大幅缩减的模型规模：使用12维而非数千维的向量、2个注意力头而非数十个、2-12层而非80层，词汇表只有几百词而非10万以上[1]。用户首次进入每个区域时会停留9-26秒进行说明，按下Space键可无限暂停，S键支持单步进行，Speed滑块可调节0.4倍至8倍的播放速度[1]。
该工具的权重采用随机生成方式，未经过训练。最终输出逻辑混合使用真实隐藏状态投影与二元语法先验，以保持生成内容的可读性[1]。
TokenTown is an interactive visualization tool that presents the inner workings of large language models through the metaphor of a virtual city.[1] The browser-based tool provides a real-time simulation of the complete Transformer model processing pipeline, including tokenization, embedding, positional encoding, multi-head attention, and feed-forward networks, making the complex computational mechanisms of LLMs intuitive and accessible.[1]
The tool implements a full set of computational operations: tokenizers, embedding lookups, sinusoidal positional encoding, layer normalization, multi-head scaled dot-product attention, residual connections, GELU feed-forward layers, and temperature/top-p sampling.[1] To keep the visualization manageable, the model uses drastically reduced dimensions—12 dimensions instead of thousands, 2 attention heads rather than dozens, 2–12 layers compared to 80 in full models, and a vocabulary of hundreds of words instead of over 100,000.[1] The weights are randomly generated and untrained; the final output logic blends genuine hidden state projections with binary bigram priors to maintain readability.[1]
Users can navigate the virtual city with several interactive controls: the tool pauses for 9–26 seconds upon first entering each region to provide explanations, the spacebar enables infinite pausing, the S key advances step-by-step, and a speed slider allows adjustment between 0.4× and 8× playback speeds.[1]