AI & ML
Beyond Success Rate: Interventional Evaluation of Memory in Vision-Language-Action Systems
arXiv cs.RO0 views
arXiv:2610.23372v1 Announce Type: new Abstract: When we say that a robot uses memory, does that memory actually play a role? Success-rate ablations alone cannot establish the use of task-relevant facts: they change information and computation together, while aggregate rates can hide offsetting gains and losses. We propose an Interventional Memory Evaluation Protocol (IMEP) for vision-language-action (VLA) systems to address four questions: (1) Does memory provide a net benefit? (2) Under what conditions does memory help? (3) Do changes to memory alter task outcomes? (4) Are these effects specific to task-relevant facts? At the retrieval interface, we compare target-fact substitutions with fact-preserving replacements, minimal numerical perturbations, and exact replicas, while assessing decision correctness separately from motor execution success. We evaluate within-episode, cross-episode, and agentic memory. In cross-episode learned-bank experiments, a one-unit-in-the-last-place (ULP) intervention, including its installation procedure, changes 11.9% of seed-paired outcomes, while changing key facts produces no clear additional effect over fact-preserving replacements. To examine when memory helps, we introduce ChainBench, where correct decisions require historical information unavailable in current observations. The full memory configuration raises task success from 39.1% to 74.3%, matching the oracle-decision condition under shared recorded executions. These results show that correct history can support robot decisions, but establishing whether a memory system realizes this benefit requires fact-specific evidence beyond aggregate success rates.
Source:https://arxiv.org/abs/2610.23372PreviousLAW: Unifying Language, Action, and World Modeling for Agentic Physical IntelligenceNextLook Before You Act: Bridging Active Perception and Vision-Language-Action Models via Observation-Aligned Data Adaptation
Related posts

The Download: the agentic web arrives and the drive for safer roads
0MIT Technology Review

Stop counting tokens, start counting what ships
0MIT Technology Review
OpenAI is finally merging ChatGPT’s chat and Work tabs
0TechCrunch

10 Open-Source Robotics Repositories for 3D Perception, VLA Policies, and Sensor Calibration
0dev.to
My self-modifying system tried to wire a gate-passing module into its own protection enforcer. It was refused three times.
0dev.to
Last night an AI agent performed brain surgery on a mouse
0Hacker News