Teaching an Agent to Sketch One Part at a Time

We develop a method for producing vector sketches one partat a time. To do this, we train a multi-modal language model-based agentusing a novel multi-turn process-reward reinforcement learning follow-ing supervised fine-tuning. Our approach is enabled by a new datasetwe call ControlSketch-Part, containing rich part-level annotations forsketches, obtained using a novel, generic automatic annotation pipelinethat segments vector sketches into semantic parts and assigns paths toparts with a structured multi-stage labeling process. Our results indicatethat incorporating structured part-level data and providing agent withthe visual feedback through the process enables interpretable, control-lable, and locally editable text-to-vector sketch generation. Project page:https://xiaodan.io/teaching-an-agent-to-sketch/