Frontend
JoyInside-VLM-Agent: Product-Conditioned Multimodal Policy Learning for Embodied Agents
arXiv cs.RO0 views
arXiv:2610.12957v1 Announce Type: new Abstract: Embodied agents must translate multimodal observations and user requests into executable actions while adapting to product constraints, changing goals, and execution feedback. We present JoyInside-VLM-Agent, a ReAct-style multimodal agent developed through product-conditioned trajectory construction and staged policy learning for robotic platforms and smart hardware. A closed-loop sandbox generates multimodal interaction trajectories covering goal revisions and failure recovery, which are checked against execution outcomes and semantic criteria. Supervised fine-tuning and multi-teacher on-policy distillation precede reinforcement learning with DPER (Dual-Source Privileged On-Policy Self-Distillation with Token-Level Evidence Routing). DPER routes token-level evidence between a frozen privileged teacher and a privileged student sharing the rollout policy's parameters, modulating reward-derived token advantages while preserving their signs. Training rewards combine generative process evaluation, multi-judge rankings, and task-queue consistency. JoyInside-VLM-Agent achieves a BFCL v4 overall score of 77.44, exceeding the evaluated baselines; results on the other public benchmarks are mixed. On JoyInside-Devices and JoyInside-Robotics, the evaluated system, which includes domain-specific skill refinement, leads the baselines on all reported metrics. Two real-world quadruped case studies illustrate visually grounded search, replanning after execution failure, and explicit failure reporting.
Source:https://arxiv.org/abs/2610.12957PreviousFrom Mobile Tap to Autonomous Agent: A Reference Architecture for the AI-Native FinTech StackNextAdaptive Whole-Body MPC with Optimal Posture Planning for Humanoid Robots under Unknown Payloads
Related posts
The User Cannot See Which Tab Is Selected, and Their Operating System Deleted Your Gradient
0dev.to CSS
Why I Prefer Lightweight WordPress Development and Custom Code
0dev.to CSS
Constraining agent CSS to design tokens (without pasting the whole theme)
0dev.to CSS
Exporting a Solo site: fixing fonts, image paths and forms by hand
0dev.to CSS
Your AI Coding Agent Needs More Than a Good Prompt
0Hacker News New
How to Migrate WordPress to Next.js Without Losing SEO
0dev.to