AI & ML
LiveFoley: Dynamic Real-Time Video-to-Audio Generation for Immersive Interactive Worlds
arXiv cs.SD0 views
arXiv:2611.05988v1 Announce Type: new Abstract: Streaming video-to-audio (V2A) generation is essential for immersive interactive applications. However, real-time constraints leave existing streaming V2A methods far behind offline models: (1) visual frontends shrink their receptive fields to cut look-ahead latency, severely degrading audio-visual alignment; (2) causal autoregressive diffusion backends denoise each audio patch in isolation, impairing audio quality. To this end, we present LiveFoley, a streaming causal autoregressive diffusion architecture that addresses both bottlenecks with imperceptible latency. For the visual frontend, we introduce a Trailing Sliding Window that converts non-causal segment-level encoders into streaming modules without retraining. By simply realigning the window, it preserves precise audio-visual alignment, while a configurable hop size reduces the frontend look-ahead well below the human perceptual threshold. Building on this tunable hop size, we propose the Unified Varhop Framework to accommodate scenarios with diverse latency and computation budgets. For the audio backend, we explore widening the denoising window of the local diffusion transformer (LocDiT) head in two temporal directions. History LocDiT adds past context to improve quality at no latency cost and serves as our default. Alternatively, Future-Warmup LocDiT breaks the autoregressive deadlock with an approximate future condition, further improving generation quality at the cost of an acceptable backend look-ahead. Extensive experiments demonstrate that LiveFoley matches offline models in temporal alignment and leads streaming systems in generation quality. Project page: https://dalision.github.io/livefoley-page/
Source:https://arxiv.org/abs/2611.05988PreviousVIPBench: A Human-Aligned Benchmark for Voice Identity Perception in the Age of Voice CloningNextGraphSeal: Graph-Augmented Speech Deepfake Detection via Temporal-Spectral Artifact Propagation
Related posts
Meta Quest Promo Codes and Coupons for November 2026
0Wired
BullMQ in NestJS: Retries, Idempotency and Dead-Letter Queues
0dev.to
80.6% at Chores, 12% at Real Work: AI's Benchmark Week
0dev.to
Efficient estimation of regression models for recurrent events in presence of terminal events
0arXiv stat.CO
Local Whittle estimation for distributional time series: A regularization framework in Bayes Hilbert space
0arXiv stat.CO
Inventory and Assortment Optimization under General Choice: A Unified Approximation Framework
0arXiv stat.CO