Stochastic Gradient Methods in Infinite-Dimensional Hilbert Spaces
We develop an end-to-end, fully rigorous theory of stochastic gradient methods in infinite-dimensional Hilbert spaces. After assembling the minimal functional-analytic and measure-theoretic toolkit (including the key inequalities used throughout), we show that the many ways practitioners define a “stochastic gradient” in function spaces in fact agree under mild assumptions. On this foundation we establish well-posedness of discrete- and continuous-time dynamics and make the continuum link to gradient-flow PDEs precise. The quantitative core of the work gives non-asymptotic convergence guarantees—with explicit constants, not just big-O—across a spectrum of regimes: convex and strongly convex objectives, PL/KL-type nonconvex landscapes, heavy-tailed noise, and composite (proximal) models. We separate and compare weak versus strong convergence, build the necessary martingale toolkit from first principles, and resolve measurability issues that arise only in infinite dimensions. A spectral analysis of the linearized dynamics clarifies mode-by-mode behavior and explains slow directions via the operator spectrum. Beyond the base theory, we treat Gaussian/RKHS settings, extensions to Hilbert manifolds, and what provably breaks (and what survives) in general Banach spaces. From a numerical perspective, we analyze five practical discretizations, proving stability+consistency ⇒ convergence, and provide pseudocode with cost that tracks mesh size, step-size, and accuracy ε; fully discrete schemes are shown to converge to the infinite-dimensional limit with explicit error constants. Four case studies—quantum ground states (imaginary-time flows), elasticity, optimal control (Pontryagin principle in function spaces), and Bayesian inverse problems (posterior concentration rates)—demonstrate the theory in action. The manuscript closes with a curated list of open problems that map a path for future work on stochastic optimization in infinite dimensions.
