Training a coding model to paint watercolours with TRL and OpenEnv

使用 TRL 和 OpenEnv 训练一个能绘制水彩画的编程模型

On 23 August, Surya Narreddi posted a beautiful video of watercolours painted by a language model. The model writes JavaScript through p5.brush, a library that “adds natural drawing tools to p5.js”. The video went viral fast, over 1.5M views at the time of writing. 8 月 23 日，Surya Narreddi 发布了一段精美的视频，展示了由语言模型绘制的水彩画。该模型通过 p5.brush 编写 JavaScript 代码，这是一个为 p5.js “添加自然绘画工具”的库。这段视频迅速走红，截至撰写本文时，观看次数已超过 150 万。

The video came with a blog post explaining the training behind an earlier and narrower stage of the project, close-up flowers rather than the full compositions in the video, sadly without open artifacts yet. His site says a full technical report is coming, so ensure you follow him. The original idea is his, coming from the art and design side, where his skills are way beyond mine. My attempt is on the engineering side, reproducing the recipe in the open with every piece published. 视频附带的一篇博客文章解释了该项目早期阶段的训练过程，当时主要集中在特写花卉而非视频中完整的构图，遗憾的是目前尚未开源相关产物。他的网站显示完整的技术报告即将发布，请务必关注他。最初的创意来自他，源于艺术与设计领域，他在这些方面的造诣远超于我。我的尝试侧重于工程层面，旨在以开源方式复现这一方案，并公开所有相关组件。

Note: for the context behind the project, told by Surya himself, watch this video of his thesis. In this article I try and reproduce his idea with TRL and OpenEnv. The reference pool dataset, the RL environment, the training scripts and the trained models, all open. The whole pipeline runs on Hugging Face, end to end: training on Jobs, the RL environment and the scorer model as Spaces, the pairwise judge through Inference Providers and every artifact on the Hub, gathered in one collection. 注：关于该项目的背景，请观看 Surya 本人讲述的毕业论文视频。在本文中，我尝试使用 TRL 和 OpenEnv 复现他的想法。参考池数据集、强化学习环境、训练脚本以及训练好的模型均已开源。整个流程在 Hugging Face 上端到端运行：在 Jobs 上进行训练，将强化学习环境和评分模型部署为 Spaces，通过 Inference Providers 调用成对判别器，所有产物均汇集在 Hub 的一个合集中。

Once the two Spaces are up, the recipe is one command. Duplicate the environment and the scorer model, set two environment variables for the reward mix, and launch: 一旦两个 Spaces 运行起来，只需一条命令即可执行方案。复制环境和评分模型，设置两个用于奖励组合的环境变量，然后启动：

hf jobs uv run train/watercolour_grpo.py --flavor h200 --timeout 48h --secrets HF_TOKEN -- \
  --env-url https://<you>-watercolour-env.hf.space \
  --model Qwen/Qwen3.5-35B-A3B --lora --all-linear --bf16 --gradient-checkpointing \
  --subject 'a peach hibiscus' --references 4 \
  --top-p 0.95 --top-k 20 \
  --lr 5e-5 --lr-scheduler constant_with_warmup --warmup-steps 5 \
  --scale-rewards none \
  --steps 110 --n-episodes 240 --num-generations 8 \
  --per-device-batch-size 1 --gradient-accumulation-steps 8 \
  --max-completion-length 8192 \
  --run-tag my-run --out <you>/watercolour-grpo --push-to-hub

The rest of this article is the story of getting there, and every piece is in the repo. I have followed the original blog step by step, and only changed something when strictly needed. Every idea of my own went into a list instead of into the experiment, and that list became “What I would try next” at the end, next to the full list of published artifacts. 本文的其余部分讲述了实现这一过程的故事，所有组件均可在仓库中找到。我按照原始博客的步骤一步步操作，仅在必要时进行了修改。我个人的每一个想法都被列入了一个清单，而不是直接放入实验中，该清单最终成为了文末的“下一步尝试”，并附带了所有已发布产物的完整列表。

Why people loved it: The paintings look loose, imperfect, handmade, at a moment when image models produce perfect (statistically average) pictures. My guess is that this contrast is a big part of why the video went viral. It reminded me of the early days of generative AI art, when the point was to explore the medium. 人们喜爱它的原因：在图像模型生成完美（统计学平均意义上）图片的当下，这些画作看起来松散、不完美且具有手工感。我猜这种反差是视频走红的重要原因。它让我想起了生成式 AI 艺术的早期阶段，那时人们的重点在于探索媒介本身。

In his thesis, Surya describes the path that led here. He started by prompting text-to-image models, where the prompt is the only lever you can pull, and more detail buys more control only up to a point. Training the model itself goes further. The other half of the idea is the medium. The model writes a program of about 150 lines of JavaScript that paints the image. That model output is code. You can read it, edit it and run it again, and the decision behind each brushstroke is visible. 在论文中，Surya 描述了通往此处的路径。他最初从提示词生成图像模型开始，提示词是唯一可控的杠杆，但增加细节所带来的控制力有限。直接训练模型则更进一步。该创意的另一半在于媒介。模型编写约 150 行 JavaScript 代码来绘制图像。模型的输出是代码。你可以阅读、编辑并再次运行它，每一笔画背后的决策都是可见的。

RL over taste: Most of the recent RL work on language models uses rewards you can verify. For example, math problems with a known answer, code that passes tests, or graders that are right or wrong and cheap to run. This project is closer to the older exception, RLHF, where the model learns a reward model from human preferences. Here the reward is aesthetic preference. There is no correct answer. The real question of the project is whether you can do RL over taste. 基于品味的强化学习：近期大多数关于语言模型的强化学习工作都使用可验证的奖励。例如，有已知答案的数学题、通过测试的代码，或者对错分明且运行成本低的评分器。本项目更接近于较早的例外情况——RLHF，即模型从人类偏好中学习奖励模型。这里的奖励是审美偏好。没有标准答案。该项目的真正问题在于：你是否可以对“品味”进行强化学习？

The reward, as his blog defines it and as the RL environment I built implements it: 正如他的博客所定义以及我所构建的强化学习环境所实现的那样，奖励机制如下：

term	weight	what it measures
gate	0.05	the sketch compiles, paints something, does not cheat
length	0.05	a soft push towards longer code snippets
pairwise judge	0.60	style, compared against references drawn from a pool
HPSv3	0.30	aesthetic preference on the render

项	权重	衡量指标
门控 (gate)	0.05	草图是否可编译、是否绘图、是否作弊
长度 (length)	0.05	对更长代码片段的轻微引导
成对判别器 (pairwise judge)	0.60	风格，与从池中抽取的参考图进行对比
HPSv3	0.30	渲染图的审美偏好

HPSv3 is an open 7B preference model. Give it an image and a text description, and it returns a score for how much a person would prefer that image. It was trained on a large set of human choices between pairs of images, so its score is an average of many people’s taste. The pairwise judge is Qwen3-VL-30B-A3B-Instruct, a general vision model called through HF Inference Providers. The pairwise judge sees the candidate painting next to four references randomly selected from the pool, guided by a written description of what to weigh (bleeds, translucent washes, soft edges), each comparison in both… HPSv3 是一个开源的 7B 参数偏好模型。给它一张图片和一段文字描述，它会返回一个分数，代表人类对该图片的偏好程度。它是在大量人类对成对图像的选择数据上训练的，因此其分数代表了许多人的平均品味。成对判别器是 Qwen3-VL-30B-A3B-Instruct，这是一个通过 HF Inference Providers 调用的通用视觉模型。成对判别器将候选画作与从池中随机选取的四张参考图放在一起观察，并根据书面描述（如渗色、半透明水洗、柔和边缘）进行权衡，每次比较都在……