一位开发者在Hacker News上分享了构建claude-real-video项目的经验，该项目是一个MIT许可的本地视频处理管道[1]。他指出，视觉大模型每个视频能处理约100-150张图像[1]，因此帧的选择策略成为关键——问题不在于处理多少帧，而在于选择正确的帧[1]。
为了解决帧选择问题，该工具采用了多层去重和选择算法[1]。首先使用ffmpeg的场景检测作为初步筛选[1]，随后通过多个去重通道进行精细化处理：包括16×16 RGB签名（8%阈值）用于基础去重、32×32网格用于小主体检测、以及192×192签名用于捕捉文本和UI变化[1]。这套方法避免了均匀采样导致的大量重复静态帧或关键时刻遗漏的问题[1]。
该工具还集成了Whisper进行语音识别[1]，支持MCP服务器接口，可在Claude Desktop等客户端中使用[1]，输出为JPEG、文本转录和清单文件[1]。项目已在github.com/HUANGCHIHHUNGLeo/claude-real-video上开源[1]。
An author has shared insights from developing claude-real-video, an MIT-licensed local video processing pipeline, highlighting that the critical challenge in enabling large language models to understand video content is not volume but strategic frame selection [1]. Visual LLMs can process approximately 100-150 images per video, and the approach to choosing which frames to extract fundamentally determines the quality of the model's comprehension [1].
The author explains that uniform sampling across videos produces suboptimal results, leading either to redundant static frames or missed critical moments [1]. To address this, the project employs ffmpeg scene detection as an initial filtering step, followed by multiple deduplication channels: 16×16 RGB signatures with an 8% threshold for general redundancy, 32×32 grid analysis for detecting small subjects, and 192×192 signatures to capture text and UI changes [1]. The pipeline integrates Whisper for speech recognition and outputs processed frames as JPEG files alongside text transcriptions and inventory files [1]. The tool supports the MCP server interface, enabling integration with Claude Desktop and other compatible clients [1]. The complete implementation is available at github.com/HUANGCHIHHUNGLeo/claude-real-video [1].