AI & ML
Optimistic Actor-Critic with Parametric Policies for Linear Markov Decision Processes
arXiv stat.ML0 views
arXiv:2603.28595v3 Announce Type: replace-cross Abstract: Although actor-critic methods have been successful in practice, their theoretical analyses have several limitations. Specifically, existing theoretical work either sidesteps the exploration problem by making strong assumptions or analyzes impractical methods with complicated algorithmic modifications. Moreover, the actor-critic methods analyzed for linear MDPs often employ natural policy gradient and define "implicit" policies through the accumulated critic estimates, rather than learning an explicitly parameterized actor. Such policies can be computationally expensive to sample from, making the environment interactions inefficient. To that end, we focus on the finite-horizon linear MDPs and propose an optimistic actor-critic framework that uses parametric log-linear policies. In particular, we introduce a tractable logit-matching regression objective for the actor. For the critic, we use approximate Thompson sampling via Langevin Monte Carlo to obtain optimistic value estimates. We prove that, when the policy features are sufficiently expressive, the resulting algorithm achieves $\widetilde{O}(\epsilon^{-4})$ and $\widetilde{O}(\epsilon^{-2})$ sample complexity in the on-policy and off-policy settings, respectively. Our results match the state-of-the-art dependence on $\epsilon$ in prior theoretical work, while our algorithm is more aligned with practice.
Source:https://arxiv.org/abs/2603.28595PreviousKinetic Langevin Splitting Schemes for Constrained SamplingNextInverse Design for Conditional Distribution Matching
Related posts
Generalizing self-normalized importance sampling with couplings
0arXiv stat.CO
Bayesian Computation of Copula-based Multivariate Models with an Application to Doping Detection
0arXiv stat.CO
WAIVES: Weak-form Asymptotically-unbiased Instrumental-Variable Estimation for SDEs
0arXiv stat.CO
Treat Your Honeypot Analysis Like Code: Git, Pull Requests and a Human Gate for AI [Guest Diary], (Wed, Nov 18th)
0SANS ISC
ISC Stormcast For Thursday, November 19th, 2026 https://isc.sans.edu/podcastdetail/10186, (Thu, Nov 19th)
0SANS ISC
Thống kê hiệu suất Al Ahli vs Al Nassr: Phép thử cực đại cho hàng công chủ nhà
0dev.to Data