Backend
Truncated backpropagation through time induces a two-sided gradient turnpike property
arXiv cs.SY0 views
arXiv:2611.07415v1 Announce Type: new Abstract: Truncated backpropagation through time reduces the computational burden of training recurrent models by restricting gradient computations to shorter temporal contexts. This induces two distinct boundary effects: missing past information perturbs the forward state evolution, while truncating the backward recursion omits future information. This motivates decoupling the interval over which gradient contributions are averaged for the parameter update from the surrounding temporal context over which they are computed. Under exponential decay of the state-transition Jacobians and mild regularity conditions, this leads to a two-sided gradient turnpike property: the error between truncated- and full-context gradient contributions decays exponentially with the distance from both context boundaries. We derive bounds for the averaged update gradient, yielding guarantees on gradient alignment and approximate full-context stationarity; under unbiased interval sampling and a Polyak-Lojasiewicz condition, these further imply a bound on the resulting optimality gap. Numerical experiments on the Wiener-Hammerstein benchmark illustrate the role of past and future context and the trade-off between predictive performance and computational cost.
Source:https://arxiv.org/abs/2611.07415PreviousA Novel Asymmetric Distributed Voltage Regulation Scheme to Enhance the Utilization of Distributed Energy ResourcesNextSafe Output-Regulation Flows for Constrained Optimization via Control Barrier Functions
Related posts
Stochastic Derivative Estimation for Discontinuous Sample Performances: A Leibniz Integration Perspective
0arXiv stat.CO
Efficiently Estimating Cycle Statistics on Graphs via Spanning Trees
0arXiv stat.CO
I Audited 26 Episodes of a 59,000-Episode Open Robot Dataset. The Data Is Fine — the Metadata Isn't.
0dev.to Data
Is Google Maps accurate for closed businesses? We checked 400 places
0dev.to Data
WARN Act and Your FSA: What Happens to Your Flexible Spending Account After a Layoff
0dev.to News
The Week of November 27: 5,991 Workers, a Thanksgiving-Week Layoff Wave
0dev.to News