When Higher Order Hurts: Pre-Asymptotic Order Collapse in Generative ODE Sampling — A Theory of Discretization–Learning Interaction

Higher-order ODE solvers are widely believed to improvesample quality in diffusion and flow-based generative models at a fixednumber of function evaluations (NFE). We show that this assumptioncan fail: at practical step sizes, higher-order methods can systematicallyunderperform first-order Euler. The mechanism is a multiplicative in-teraction between discretization and learning error. Applying backwarderror analysis to a learned field fˆ = f +ε reveals interaction terms hk ∇k εthat dominate classical truncation error at practical NFEs in diffusionand flow models, causing a solver of nominal order p to behave as lowerorder—pre-asymptotic order collapse. Our analysis yields a predictivecrossover step size h⋆ characterizing when higher order helps or hurts.Validation on toy manifolds and pretrained diffusion and flow modelsconfirms the predicted hierarchy: Euler is best at low NFE, Heun over-takes at moderate budgets, and RK4 can surpass Heun at higher NFE.Building on this theory, we propose an adaptive sampler that dynami-cally selects solver order based on the local roughness score R = h/h⋆ ,estimated at zero extra NFE, and often improves sample quality acrossmodels and datasets.