The Celery Lifecycle: How a Task Gets Registered, Queued, and Run

Celery 生命周期：任务是如何被注册、排队和执行的

If you have ever needed to send an email, process a payment, or generate a report without making your user wait, you have probably run into Celery. Celery is a tool that lets you run jobs in the background, away from your main app. This article breaks down how it works, step by step, in plain language. 如果你曾经需要发送电子邮件、处理付款或生成报告，同时又不想让用户等待，那么你很可能接触过 Celery。Celery 是一个允许你在后台运行任务，使其脱离主应用程序的工具。本文将用通俗易懂的语言，逐步解析其工作原理。

What Is Celery, In Simple Terms

简单来说，什么是 Celery？

Think of Celery like a restaurant kitchen. Your app (the waiter) takes an order from a customer. Instead of cooking the food itself, the waiter drops the order into a queue (the kitchen order rail). A cook (the worker) picks up the order from the rail and prepares it. When the food is ready, it goes to a pickup counter (the result backend) where anyone can come check if it’s done. 把 Celery 想象成一家餐厅的厨房。你的应用程序（服务员）接收顾客的订单。服务员不会亲自下厨，而是将订单放入队列（厨房的订单栏）。厨师（工作进程/Worker）从订单栏取走订单并进行烹饪。当食物准备好后，它会被送到取餐台（结果后端/Result Backend），任何人都可以去查看订单是否完成。

Celery has four main players: Celery 有四个主要角色：

• The Producer - your app, the one that creates tasks.
• 生产者 (Producer) - 你的应用程序，负责创建任务。
• The Broker - the message queue that holds tasks until a worker is free.
• 代理 (Broker) - 消息队列，负责暂存任务，直到有 Worker 空闲。
• The Worker - the process that picks up and runs the tasks.
• 工作进程 (Worker) - 负责获取并执行任务的进程。
• The Result Backend - where results are stored, if you need them later.
• 结果后端 (Result Backend) - 如果你需要后续获取结果，这里就是存储结果的地方。

In short: your app sends a task message to the broker. The broker holds it until a worker is free. The worker picks it up, runs the actual function, and (if you set one up) writes the result to the result backend. Your app can then go back and check that result backend to see what happened. Now let’s go through each part. 简而言之：你的应用程序向 Broker 发送任务消息。Broker 将其暂存，直到有 Worker 空闲。Worker 获取任务、执行实际函数，并将结果写入结果后端（如果你配置了的话）。随后，你的应用程序可以回头检查结果后端，查看任务执行情况。现在，让我们逐一解析每个部分。

---

1. How Tasks Get Registered

1. 任务是如何被注册的

Before Celery can run a task, it needs to know the task exists. This is called registration, and it happens the moment your Python code is imported - not when the task runs. 在 Celery 运行任务之前，它必须先知道任务的存在。这被称为“注册”，它发生在 Python 代码被导入的那一刻，而不是任务运行的时候。

The @app.task decorator

@app.task 装饰器

You create a Celery app instance, then decorate any function with @app.task. That decorator does not run the function immediately. Instead, it wraps the function and adds it to a task registry - basically a dictionary that Celery keeps internally, mapping a task name to the actual function. 你创建一个 Celery 应用实例，然后用 @app.task 装饰任何函数。该装饰器不会立即运行函数，而是将函数包装起来并添加到任务注册表中——这本质上是 Celery 内部维护的一个字典，用于将任务名称映射到实际的函数。

from celery import Celery
app = Celery("myproject")

@app.task
def send_welcome_email(user_id):
    # logic to send an email
    print(f"Sending welcome email to user {user_id}")

The moment Python imports this file, send_welcome_email gets registered under the name myproject.tasks.send_welcome_email (module path + function name, by default). 当 Python 导入此文件时，send_welcome_email 就会以 myproject.tasks.send_welcome_email（默认情况下为模块路径 + 函数名）的名称被注册。

Why registration matters

为什么注册很重要

Here is the key thing people miss: the producer and the worker are often two separate processes, sometimes on two separate machines. When your app calls send_welcome_email.delay(5), it does NOT run the function. It just creates a message like this: 这里有一个人们常忽略的关键点：生产者和 Worker 通常是两个独立的进程，有时甚至位于两台不同的机器上。当你的应用程序调用 send_welcome_email.delay(5) 时，它并不会运行该函数，而是创建一条如下所示的消息：

{
  "task": "myproject.tasks.send_welcome_email",
  "args": [5],
  "kwargs": {},
  "id": "a1b2c3d4-uuid"
}

That message is just a name and some arguments - a string, basically. The actual Python function only needs to exist on the worker’s side, because the worker is the one that looks up the name in its own registry and calls the real function. This is why, if a task is not registered on the worker (say, you forgot to import that module in the worker’s app), you will get a KeyError or NotRegistered error even if the producer sent the message fine. The message got sent, but nobody on the worker side knew what myproject.tasks.send_welcome_email meant. 这条消息本质上只是一个名称和一些参数组成的字符串。实际的 Python 函数只需要存在于 Worker 端，因为是 Worker 在自己的注册表中查找该名称并调用真正的函数。这就是为什么如果任务没有在 Worker 端注册（例如，你忘记在 Worker 的应用中导入该模块），即使生产者成功发送了消息，你也会收到 KeyError 或 NotRegistered 错误。消息虽然发送成功了，但 Worker 端没人知道 myproject.tasks.send_welcome_email 是什么意思。

Autodiscovery

自动发现

In bigger apps (like Django), you don’t manually import every task file. Celery gives you app.autodiscover_tasks(), which scans your installed apps for a tasks.py file and imports them automatically, registering every @app.task function it finds inside. 在大型应用（如 Django）中，你不需要手动导入每个任务文件。Celery 提供了 app.autodiscover_tasks()，它会扫描你已安装的应用中的 tasks.py 文件并自动导入它们，注册在其中找到的每一个 @app.task 函数。

So the registration flow, in words, goes like this: Python imports the file that contains your task → the @app.task decorator fires → the function’s name and a reference to it get added to the Celery app’s internal registry → from that point on, the app knows this task exists and can hand it work whenever it’s called. 因此，注册流程可以概括为：Python 导入包含任务的文件 → @app.task 装饰器触发 → 函数名称及其引用被添加到 Celery 应用的内部注册表中 → 从那一刻起，应用就知道该任务存在，并可以在调用时为其分配工作。

---

2. The Broker URL - The Middleman

2. Broker URL - 中间人

The broker is a message queue. Its only job is to hold task messages until a worker is ready to grab one. Celery does not process anything itself - it just passes messages through the broker. You configure it with a single connection string, the broker URL: Broker 是一个消息队列。它唯一的任务就是暂存任务消息，直到有 Worker 准备好获取它们。Celery 本身不处理任何任务——它只是通过 Broker 传递消息。你只需通过一个连接字符串（即 Broker URL）进行配置：

app = Celery(
    "myproject",
    broker="redis://localhost:6379/0"
)

Common broker choices: 常见的 Broker 选择：

Broker	Example URL	Notes
Redis	redis://localhost:6379/0	Fast, simple, good for most small-to-medium apps
Redis	redis://localhost:6379/0	快速、简单，适用于大多数中小型应用
RabbitMQ	amqp://guest:guest@localhost:5672//	More robust, built for messaging, heavier to run
RabbitMQ	amqp://guest:guest@localhost:5672//	更稳健，专为消息传递设计，运行开销较大
Amazon SQS	sqs://	Managed, good if you’re already on AWS
Amazon SQS	sqs://	托管服务，如果你已经在 AWS 上，这是个好选择

What the broker URL actually controls Broker URL 实际上控制了什么

The broker URL tells Celery: Broker URL 告诉 Celery：

• Which service to talk to (Redis, RabbitMQ, etc.)
• 要连接的服务（Redis、RabbitMQ 等）
• Where it lives (host and port)
• 服务的位置（主机和端口）
• How to authenticate (username/password, if needed)
• 如何进行身份验证（如果需要，包括用户名/密码）
• Which database/vhost to use (the /0 at the end of a Redis URL picks database 0, for example)
• 使用哪个数据库/虚拟主机（例如，Redis URL 末尾的 /0 表示选择 0 号数据库）

Nothing more. The broker does not know what a task “means.” It just stores and forwards messages, like a mailbox. This is why Celery can swap Redis for RabbitMQ without changing a single line of your task code - only the broker URL changes. 仅此而已。Broker 不知道任务“意味着”什么。它就像一个邮箱，只负责存储和转发消息。这就是为什么 Celery 可以在不修改一行任务代码的情况下将 Redis 替换为 RabbitMQ——只需更改 Broker URL 即可。

Picture it this way: your app pushes a task message into the broker’s queue. The broker just sits there holding it. On the other side, one or more workers are constantly listening to that same queue. Whichever worker is free at that moment grabs the next message in line. If you have five workers all pointed at the same broker URL, they are all pulling from the same line - so Celery scales horizontally just by adding more workers behind the same broker. 想象一下：你的应用程序将任务消息推送到 Broker 的队列中。Broker 只是静静地持有它。在另一端，一个或多个 Worker 持续监听同一个队列。任何空闲的 Worker 都会获取队列中的下一条消息。如果你有五个 Worker 都指向同一个 Broker URL，它们都会从同一个队列中拉取任务——因此，Celery 只需在同一个 Broker 后添加更多 Worker 即可实现水平扩展。

---

3. Execution - What Happens When a Task Runs

3. 执行 - 任务运行时发生了什么

Once a worker pulls a task message off the broker, here is what happens, step by step: 一旦 Worker 从 Broker 中拉取到任务消息，以下是逐步发生的流程：

1. Deserialize - the worker reads the JSON (or whatever serializer you’re using) message and pulls out the task name, args, and kwargs.
2. 反序列化 (Deserialize) - Worker 读取 JSON（或你使用的任何序列化器）消息，并提取出任务名称、参数 (args) 和关键字参数 (kwargs)。
3. Lookup - it checks its local task registry for a function matching that name.
4. 查找 (Lookup) - 它在本地任务注册表中检查是否有与该名称匹配的函数。
5. Acknowledge (ack) - by default, Celery acknowledges the message either right before or right after running it, telling the broker “I’ve got this, you can remove it from the queue.” This matters for reliability - if a worker crashes before acking, the broker can redeliver the task to another worker.
6. 确认 (Acknowledge/ack) - 默认情况下，Celery 会在运行任务之前或之后确认消息，告诉 Broker：“我已经收到任务了，你可以将其从队列中移除。”这对可靠性至关重要——如果 Worker 在确认前崩溃，Broker 可以将任务重新分发给另一个 Worker。
7. Run - the actual Python function executes.
8. 运行 (Run) - 执行实际的 Python 函数。