Google DeepMind推出了名为"视觉提示工程"（VIPE）的新范式，将提示工程应用从语言模型扩展至视觉领域 [1]。该方法通过将抽象图像转换为更真实的视觉场景来优化视频模型的推理性能 [1]。
在多项性能测试中，视觉提示工程的效果往往超越传统的文本提示工程 [1]。以Veo 3.1为例，其准确率从41.3%提升至59.3% [1]；Omni Flash则从56.3%提升到67.5% [1]。在自由形式视觉提示工程的应用中，部分任务的错误率下降超过75% [1]。不过在Sort 3 Numbers任务上呈现不同表现，最优文本提示的准确率达到86%，而最优图像提示为76% [1]。
这一新方法包含构想器、编辑器、筛选器三个步骤 [1]。
Google DeepMind has unveiled a new approach called "visual prompt engineering" (VIPE) that extends the concept of prompt optimization from language models into the visual domain. [1] Rather than relying solely on text instructions, this method transforms abstract images into more realistic visual scenes to enhance video model reasoning performance. [1]
The experimental results demonstrate that visual prompts frequently outperform text-based prompts across multiple tasks. [1] For instance, Veo 3.1's accuracy improved from 41.3% to 59.3%, while Omni Flash saw gains from 56.3% to 67.5%. [1] In some cases, freeform visual prompt engineering reduced error rates by more than 75% on particular tasks. [1] However, on the Sort 3 Numbers task, the optimal text prompt achieved 86% accuracy compared to 76% for the optimal image prompt. [1]
The visual prompt engineering methodology comprises three sequential steps: a conceptualizer, an editor, and a filter. [1] This framework allows researchers to systematically refine visual inputs to achieve superior model performance on video understanding and reasoning challenges.
