Open Source
CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing
arXiv cs.SD0 views
arXiv:2610.10264v1 Announce Type: new Abstract: Multimedia editing requires generative models to interpret complex, compositional instructions and modify only the desired elements across modalities, a capability largely beyond existing systems. Current approaches are typically limited to simple, single-attribute edits within one modality, since diverse, high-quality editing pairs are hard to obtain, especially for cross-modal tasks where aligned audiovisual (AV) data is scarce. We present CrossEdit, a unified omni-modal editing model for images, audio, and video that performs AV movie scene edits zero-shot through cross-modal transfer. Our key observation is that complex instruction-following, once learned in any modality, generalizes to others. We exploit the additive nature of acoustic signals to procedurally generate large-scale audio editing pairs with compositional instructions, and show that fine-tuning on this synthetic data and self-supervised AV masked reconstruction, alongside a targeted set of curated cross-modal tasks, induces instruction-following that transfers zero-shot to unseen modality and instruction combinations. To evaluate this capability, we release CrossEditBench, a human-annotated benchmark of AV edits on movie scenes, and propose AV-FES, a metric that jointly scores instruction following and consistency. We show that our proposed techniques improve performance on zero-shot AV movie scene editing, while maintaining or improving performance on lip-synced speech editing and standard image, video, and audio editing benchmarks. Demos are available at https://wanchichen.github.io/crossedit/.
Source:https://arxiv.org/abs/2610.10264PreviousCARES: A Controlled Synthetic Benchmark of Speaker Reactions to SoundNextMIRA: A Musical Intent Refinement Agent for Aligning Text-to-Music Generation with User Intent
Related posts

Karpathy built HackerTube in 2 hours. I rebuilt it in the open and measured it
0dev.to AI
Show HN: Poinky – Claude Code you talk to, that sees your screen
0Hacker News Show
Show HN: DeskAccess – Open-source remote access for Raspberry Pi
0Hacker News Show
Show HN: The Public Eye – Agentic search for congressional bill legibility
0Hacker News Show
NASA’s Perseverance Mars rover breaks historic record after traveling 28 miles
0Science Daily Tech
A fresh wave of cyberattacks has hit Europe’s supermarket refrigerators, destroying food across countries
0TechCrunch