Bridging Online and Offline Handwriting via Differentiable Physical Rendering

Realistic handwritten text generation plays an important rolein numerous applications, such as font design, biometric authentication,and robotic calligraphy. Existing methods are typically divided into twoindependent paradigms: online approaches that estimate handwritingtrajectories and offline approaches that synthesize realistic handwritingimages. While online models capture structural and temporal dynamics,they often lack fine-grained textures, whereas offline models reproducerealistic appearance but discard stroke order. However, unifying onlineand offline models remains challenging due to (1) the lack of an explicitphysical model linking stroke kinematics to pixel-level appearance and(2) the absence of paired trajectory–image datasets. Moreover, enablingend-to-end learning requires a differentiable rendering process acrossmotion and appearance domains. To address these challenges, we proposea compact physical brush model that bridges stroke dynamics and visualappearance, together with a differentiable rendering module that convertsstroke trajectories into stylized images. By integrating these components,we propose a unified online–offline handwriting generation frameworkvia differentiable brush rendering. The proposed framework consistsof four core modules: 1) a text-to-stroke generator that predicts thetarget stroke conditioned on the given text and style image, 2) a brushparameter observer that extracts brush model parameters from stylereferences, 3) a differentiable brush renderer that maps a stroke sequenceand physical brush parameters into a handwritten image, and 4) a zero-shot image refiner that refines rendered images via diffusion models.Extensive experiments and real-world robotic calligraphy demonstrationsvalidate our approach, achieving both structural and visual fidelity.