AI & ML
What Can a VLA Learn from the Agent That Calls It? Task-Level Experience Distillation for Embodied Navigation
arXiv cs.RO0 views
arXiv:2611.16246v3 Announce Type: replace Abstract: Vision-language-action (VLA) models drive the robot directly from camera views, yet a VLA alone shows three typical failures. It walks past the goal, stops before reaching it, and rarely recovers after going off track. Pairing a VLA with a large language model (LLM) planner fixes these failures, but queries the LLM throughout every episode, adding token costs and latency. We introduce Harness-to-Executor (H2E), a task-level distillation method that pairs an LLM planner and a frozen VLA as a heterogeneous training agent, keeps only sessions that reach the goal, and then trains a VLA alone on them so no LLM is used at test time. In these sessions, the planner gives sub-instructions and moves the robot with its own tools, detours are common, and the planner decides when to stop. H2E therefore labels every motion as a VLA action for the original task, whoever made it. It masks the loss on steps moving away from the goal and assigns end labels to states near it, so the VLA learns where the task ends. H2E sets a new state of the art on REVERIE-CE and NavRAG-CE. It raises an 8B VLA's success rate by 15.6% and 11.1% respectively, surpassing a method trained on 10M samples. The distilled VLA exceeds its agent on unseen episodes with zero LLM tokens and runs 3.7 times faster.
Source:https://arxiv.org/abs/2611.16246PreviousFast Inference-Time Policy Refinement via Action-Affine World ModelsNextSALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models
Related posts

Orchestrate DuckDB analytics pipelines with Amazon MWAA
0AWS Big Data Blog

Long-term Amazon Redshift observability with Datadog and Amazon S3 Tables
0AWS Big Data Blog

Choosing the right network architecture for Amazon Bedrock AgentCore
0AWS Networking Blog

New Site-to-Site VPN connections will default to recommended cryptography
0AWS Networking Blog

How Salesforce is advancing global-scale model serving with cross-Region inference routing in Amazon SageMaker AI
0AWS Machine Learning Blog

How Airbnb uses cost per task to operate coding agents on Amazon Bedrock
0AWS Machine Learning Blog