A research team led by Google, together with researchers from Stanford, Oxford, and other U.S. universities, has introduced StreamVibe, a video agent designed to decide when it should speak instead of responding only to explicit commands. The system is intended to observe activities such as cooking or assembling furniture and offer help when a user encounters difficulty, even when the request is vague.
The researchers say existing video assistants are generally designed for specific instructions, such as notifying someone when water begins to boil. As a result, they often fail to recognize implied intentions, lose track of completed steps, repeat advice, or respond too late. They can also continue following an earlier request after the user changes it during the task.
To study these problems, the team created a benchmark containing about 4,100 examples drawn from more than 183 hours of everyday activities and software recordings. The examples cover explicit instructions, vague goals, and requests that change over time. In one scenario, a user dismantling a toy excavator first asks for progress updates and later switches to requesting a quality check.
The researchers trained StreamVibe-VL to decide every second whether to remain silent or respond. reinforcement learning was used to make the assistant more selective and reduce unnecessary interruptions. A separate memory system records pending requests and completed steps, while the model itself determines whether to speak and what to say.
According to the paper, StreamVibe-VL improves response timing and quality by 10.7 percent compared with other models. The memory system is intended to provide a further 12.3 percent improvement in proactive interaction while reducing unwanted interruptions. The model generally outperformed its base model and other open-source systems, although Google's proprietary Gemini 3.8 Flash achieved higher scores in several tests.
The results also highlight the limits of current video agents. Models often make suitable decisions when they only need to evaluate individual moments, but performance falls sharply when they process a full video and must build on their previous responses. The paper reports that a key response-content metric declined by as much as 61.5 percent when systems relied on their own interaction history. Videos longer than five minutes were particularly difficult, especially when instructions changed during the task.
In an example involving a GIMP tutorial, the comparison model JoyAI incorrectly stated that a converted image had already been saved before the user opened it. StreamVibe-VL instead supplied the next relevant steps, including selecting “Colors” and then “Desaturate.” In a tutorial for making rolled toast with jam, other models mainly described visible actions, while StreamVibe-VL recommended removing the crust first.
The researchers note that real-world situations are likely to be more complex than the benchmark. Future work will examine how user preferences and the surrounding environment affect the assistant's behavior over time. StreamVibe is part of Google's broader work on video systems that analyze actions and determine which information requires closer attention.
Sources: The Decoder.