RL 1: Biological foundations and the “Law of Effect” (1898–1949)

RL 1：生物学基础与“效果律”（1898–1949）

Before GPUs. Before neural networks. Before transistors were even a rumor. There was a hungry cat in a wooden crate, trying to get to a piece of fish. The main character of this era isn’t an engineer. It’s a loose group of psychologists — some of whom actively disliked each other’s ideas — poking at animals to answer one question: why do living things do the things they do? 在 GPU 出现之前，在神经网络出现之前，甚至在晶体管还只是传闻的时候，有一只饥饿的猫被关在木箱里，试图吃到一块鱼。这个时代的主角不是工程师，而是一群松散的心理学家——他们中有些人甚至非常反感彼此的观点——他们通过研究动物来回答一个问题：生物为什么会做出它们所做的行为？

They never wrote a line of code. They were not trying to build AI. And yet, in the fifty years between 1898 and 1949, they collectively reverse-engineered the core loop that every modern reinforcement learning system still runs on. This post is that story. No math yet — I promise. Just the loop, and the four people who found its pieces. 他们从未写过一行代码，也并非试图构建人工智能。然而，在 1898 年到 1949 年这五十年间，他们共同逆向工程出了现代强化学习系统至今仍在运行的核心循环。这篇文章讲述的就是这段故事。我保证，目前还没有数学公式，只有这个循环，以及发现其组成部分的四个人。

First, the thing we’re building. Let me give you the shape of the answer before the story, so you know what you’re collecting. Reinforcement learning describes any system that learns by doing stuff and seeing what happens. That’s it. Not by being shown correct answers — by trying, failing, and adjusting. 首先，我们要构建的是什么？在进入故事之前，先让我为你勾勒出答案的轮廓，这样你就知道你在收集什么了。强化学习描述的是任何通过“做事并观察结果”来学习的系统。仅此而已。它不是通过被告知正确答案来学习，而是通过尝试、失败和调整来学习。

We’ll call the learner the agent (the cat, the rat, the robot, the model). Everything outside it is the environment (the box, the room, the game). The loop runs like this: The agent observes its state — the situation it’s currently in. It picks an action. The environment responds with a reward — a number saying “that was good” or “that was bad.” The agent lands in a new state, and the loop repeats. 我们将学习者称为“智能体”（agent，可以是猫、老鼠、机器人或模型）。智能体之外的一切都是“环境”（environment，可以是盒子、房间或游戏）。循环过程如下：智能体观察其“状态”（state）——即它当前所处的境况；它选择一个“动作”（action）；环境以“奖励”（reward）作为回应——一个表示“好”或“坏”的数值；智能体进入一个新的状态，循环往复。

Two more, and they’re the ones that do the heavy lifting later. The agent needs a policy: a strategy for choosing actions. Not a fixed script — a rule of thumb that says in this kind of situation, do this kind of thing. And it needs a sense of value: how good a state is, based not on the reward sitting in it right now, but on everything it tends to lead to. An empty hallway is worth nothing on its own. An empty hallway with the kitchen at the end of it is worth quite a lot. 还有两个概念，它们在后续工作中起着至关重要的作用。智能体需要一个“策略”（policy）：一种选择动作的策略。它不是固定的脚本，而是一条经验法则，告诉你在这种情况下该做什么。它还需要一种“价值”（value）感：评估一个状态有多好，这不仅基于当前状态下的即时奖励，还基于它可能导向的所有后续结果。一条空走廊本身毫无价值，但如果走廊尽头是厨房，那它就非常有价值。

Six words: agent, state, action, reward, policy, value. Every single one of them was discovered by someone studying animals, decades before anyone could program a computer. Let’s go get them. 六个词：智能体、状态、动作、奖励、策略、价值。每一个词都是在有人能编写计算机程序之前的几十年里，由研究动物的人发现的。让我们去看看它们是如何被发现的。

1898 — Thorndike’s Cats and the Law of Effect

1898年 — 桑代克的猫与“效果律”

In 1898, a graduate student named Edward Thorndike was annoyed. The animal literature of his day was full of charming anecdotes — dogs who understood latches, cats who figured out doorknobs. Thorndike’s suspicion was that people were flattering their pets. Animals weren’t reasoning. They were doing something else, and nobody had bothered to measure it. So he built the puzzle box. 1898年，一位名叫爱德华·桑代克（Edward Thorndike）的研究生感到很恼火。当时关于动物的文献充斥着迷人的轶事——比如能理解门闩的狗，能弄懂门把手的猫。桑代克怀疑人们是在美化他们的宠物。动物并不是在进行推理，它们在做别的事情，而没有人去测量过。于是，他制造了“迷箱”（puzzle box）。

The setup is almost comically simple. A hungry cat goes in a small wooden crate. The door opens only if the cat pulls a loop of string, or presses a lever, or steps on a pedal. A piece of fish sits outside, in plain view. Then Thorndike started a stopwatch. 这个装置简单得近乎滑稽。一只饥饿的猫被关进一个小木箱里。只有当猫拉动一根绳圈、按下杠杆或踩下踏板时，门才会打开。一块鱼放在外面，清晰可见。然后，桑代克按下了秒表。

Trial one is chaos. The cat claws the bars. It shoves its paws through every gap. It bites things, yowls, thrashes at anything loose. After a few minutes of pure flailing, it hits the lever by accident. The door swings open. Fish. Trial two: slightly less flailing. Trial twenty: the cat walks in, hits the lever, and strolls out. 第一次尝试是一片混乱。猫抓挠栅栏，把爪子伸进每一个缝隙，啃咬东西，嚎叫，拍打任何松动的东西。在几分钟的胡乱挣扎后，它意外地碰到了杠杆。门开了。鱼到手了。第二次尝试：挣扎稍微减少。第二十次尝试：猫走进去，按下杠杆，然后悠闲地走出来。

Here’s what Thorndike noticed, and it’s the part that matters: there was never a lightbulb moment. No trial where the cat suddenly got it. If cats were reasoning about levers, you’d expect the escape time to fall off a cliff — one long trial, then instantly fast ones. Instead, when he plotted escape time against trial number, he got a slow, noisy, gradual curve. The useless behaviors quietly faded. The useful one got faster. 桑代克注意到了这一点，这正是关键所在：猫从来没有过“灵光一现”的时刻。没有哪一次尝试是猫突然“开窍”了。如果猫是在对杠杆进行推理，你会预期逃脱时间会断崖式下跌——经历一次漫长的尝试后，立刻变得非常快。然而，当他绘制逃脱时间与尝试次数的关系图时，得到的是一条缓慢、嘈杂且渐进的曲线。无用的行为悄然消失，有用的行为变得越来越快。

That curve is the first real evidence in this story. Learning wasn’t insight. It was selection — good behaviors surviving, bad ones dying off. Thorndike wrote it up as the Law of Effect: If an action is followed by a satisfying state of affairs, the bond between that situation and that action is strengthened. If followed by discomfort, it is weakened. 这条曲线是这个故事中第一个真正的证据。学习不是洞察力，而是选择——好的行为得以保留，坏的行为逐渐消亡。桑代克将其总结为“效果律”（Law of Effect）：如果一个动作之后紧跟着令人满意的结果，那么该情境与该动作之间的联系就会加强；如果紧跟着不适，联系就会减弱。

Read that again with our vocabulary. Situation is the state. Action is the action. Satisfying is the reward. Bond strengthened is the policy update. Thorndike, in 1898, wrote pseudocode for a policy gradient. He just used the word “bond.” 用我们现在的词汇再读一遍：情境就是“状态”，动作就是“动作”，令人满意就是“奖励”，联系加强就是“策略更新”。1898年的桑代克实际上写出了策略梯度的伪代码，他只是用了“联系”（bond）这个词。

🎒 In the backpack: trial-and-error learning. Rewards make actions more likely. Learning is gradual, not sudden. 🎒 背包里的收获：试错学习。奖励使动作发生的可能性增加。学习是渐进的，而非突发的。

But something’s missing

但还缺少点什么

Thorndike’s cat only learns after it gets the fish. The reward does all the teaching. But watch a real animal for five minutes and you’ll notice it reacting to things that aren’t rewards at all. Your dog loses its mind at the sound of a cupboard opening. Nothing has been eaten yet. The cupboard predicts food. Thorndike had no account of that. Rewards were where his story started and ended. The person who noticed the signal was, characteristically, not looking for it. 桑代克的猫只有在吃到鱼之后才会学习。奖励承担了所有的教学工作。但如果你观察一只真实的动物五分钟，你会发现它会对那些根本不是奖励的东西做出反应。你的狗听到橱柜打开的声音就会兴奋不已，尽管还没吃到任何东西。橱柜预示着食物。桑代克对此无法解释。奖励是他故事的起点，也是终点。而那个注意到信号的人，正如其性格所体现的那样，他原本并不是在寻找这个。

The 1900s–1920s — Pavlov and the Problem of Prediction

1900年代–1920年代 — 巴甫洛夫与预测问题

Ivan Pavlov was a Russian physiologist studying digestion. He won a Nobel Prize for it in 1904. He was, by most accounts, mildly irritated that history remembers him for the dogs. His actual research required measuring saliva. And his dogs kept ruining the measurements by salivating early — at the sight of the lab assistant who usually brought food, at the sound of footsteps in the corridor. Before any food existed. 伊万·巴甫洛夫（Ivan Pavlov）是一位研究消化的俄罗斯生理学家。他于1904年因此获得了诺贝尔奖。据大多数人所说，他对自己因狗而被历史铭记感到有些恼火。他真正的研究需要测量唾液，但他的狗总是因为提前流口水而破坏测量结果——看到通常送食物的实验助手时，听到走廊里的脚步声时，甚至在食物出现之前就会流口水。

Most researchers would call that noise and try to control for it. Pavlov called it the experiment. He ran it systematically. Present a neutral signal — a metronome, a buzzer, a light, a tone — then deliver food, a second or two later. Repeat. (Quick myth-correction, since it costs one sentence: the famous bell is mostly folklore. Pavlov’s lab notes lean heavily on metronomes, buzzers, and tuning forks. The bell survived because it’s easier to draw.) 大多数研究人员会称之为“噪声”并试图消除它，但巴甫洛夫将其视为实验本身。他系统地进行了实验：呈现一个中性信号——节拍器、蜂鸣器、灯光或音调——然后在一两秒后提供食物。重复此过程。（快速纠正一个误区：著名的铃铛大多是民间传说。巴甫洛夫的实验记录更多使用的是节拍器、蜂鸣器和音叉。铃铛之所以流传下来，是因为它更容易画。）

After enough pairings, the signal alone made the dogs salivate. Nothing about the food had changed. What changed was that the dog had learned the signal carries information about the future. This is the second piece, and it’s a big one. Thorndike’s cat learned what to do. Pavlov’s dog learned what’s coming. In RL terms, that’s… 经过足够多次的配对后，仅凭信号就能让狗流口水。食物本身没有任何变化，改变的是狗学会了信号携带关于未来的信息。这是第二个关键部分，而且非常重要。桑代克的猫学会了“做什么”，巴甫洛夫的狗学会了“预知未来”。用强化学习的术语来说，这就是……