Free Endpoints Are a Contract, Not a Gift: A Fit Test for Agent Workloads

免费端点是一份合同，而非一份礼物：智能体工作负载的适配性测试

Free model access is not a gift. It is a contract with someone else’s rate limits, queueing policy, and maintenance schedule. Self-hosting inverts that contract: you own the latency, the GPU, and the 2 a.m. page. Most teams choose between the two by comparing price per token, and that is exactly how they end up with a production agent that stalls at 9:15 every morning.

免费的模型访问权限并非一份礼物。它实际上是一份合同，受制于他人的速率限制、排队策略和维护计划。而自托管则颠覆了这种合同：你需要自行负责延迟、GPU 以及凌晨两点的系统报警。大多数团队在两者之间做选择时，往往只比较 Token 的单价，而这正是导致他们的生产环境智能体每天早上 9:15 准时卡死的原因。

Agent workloads are moving from demos to production, and the conversation has shifted from what models can do to what they cost to operate. The problem is that agent traffic does not look like chat traffic. A coding agent emits bursts of small requests — a tool call, a diff review, a short completion — separated by long idle gaps. That shape punishes endpoints optimized for steady throughput. A cost-per-token benchmark measures unit price, not whether the endpoint survives your burst pattern. The only honest test is to probe the endpoint the way your agent will actually call it.

智能体工作负载正从演示阶段转向生产环境，讨论的焦点也从“模型能做什么”转向了“运行成本是多少”。问题在于，智能体的流量模式与聊天流量截然不同。编程智能体会发出突发性的小请求（如工具调用、差异审查、简短补全），中间夹杂着长时间的空闲间隔。这种流量形态对那些针对稳定吞吐量优化的端点来说是一种考验。基于 Token 的成本基准测试衡量的是单价，而非端点能否承受你的突发流量模式。唯一可靠的测试方法是按照智能体实际调用的方式去探测端点。

Three questions decide the fit before any pricing math. First, what is your traffic shape: steady, bursty, or spiky? Second, what happens to your data when it crosses a third-party boundary? Third, how much operational slack do you have — can you babysit a self-hosted model, or does the endpoint need to be someone else’s problem? Free hosted tiers win when the answers are steady, non-sensitive, and no-slack; self-hosting wins when they are spiky, sensitive, and you have the time.

在进行任何价格计算之前，有三个问题决定了适配性。第一，你的流量形态是什么：稳定、突发还是尖峰？第二，当数据跨越第三方边界时会发生什么？第三，你有多少运维余力——你是能亲自维护自托管模型，还是必须将端点问题甩给别人？当答案是“稳定、非敏感、无运维余力”时，免费托管层级是赢家；而当答案是“尖峰、敏感、有时间维护”时，自托管则是更好的选择。

Consider a concrete case. A background job that summarizes a few documents an hour is steady and forgiving; a free tier is almost certainly fine. An interactive coding agent that fires eight parallel tool calls while a developer waits is spiky and latency-sensitive; the same free tier can feel like a different product.

考虑一个具体的案例。一个每小时总结几份文档的后台任务是稳定且宽容的，免费层级几乎肯定没问题。而一个在开发者等待时同时触发八个并行工具调用的交互式编程智能体，则是尖峰且对延迟敏感的；同样的免费层级在此时可能会表现得判若两物。

Here is a probe you can run against any OpenAI-compatible endpoint. It fires a fixed number of requests at a fixed concurrency, retries once after a 429, and reports success rate, rate-limit events, and latency percentiles. Run it twice: once at concurrency 1 for a baseline, once at the concurrency your agent actually uses. The difference between those two runs is the real cost of the endpoint.

以下是一个你可以针对任何兼容 OpenAI 接口的端点运行的探测脚本。它以固定的并发数发送固定数量的请求，在遇到 429 错误时重试一次，并报告成功率、速率限制事件和延迟百分位。请运行两次：第一次以并发 1 作为基准，第二次以你智能体实际使用的并发数运行。这两次运行之间的差异就是该端点的真实成本。

# probe_endpoint.py — fit test for a free or cheap model endpoint.
# Usage:
# export ENDPOINT_URL='https://...'
# export ENDPOINT_KEY='your-key'
# export PROBE_MODEL='model-name'
# python probe_endpoint.py --requests 60 --concurrency 4

import argparse
import asyncio
import os
import statistics
import time
import httpx

async def call_once(client, url, headers, payload):
    t0 = time.perf_counter()
    try:
        r = await client.post(url, headers=headers, json=payload, timeout=60)
        return r.status_code, (time.perf_counter() - t0) * 1000
    except Exception as exc:
        return 0, (time.perf_counter() - t0) * 1000

async def worker(client, url, headers, payload, sem, results):
    async with sem:
        status, ms = await call_once(client, url, headers, payload)
        if status == 429:
            # one recovery attempt: wait, then resend
            await asyncio.sleep(2)
            status, ms = await call_once(client, url, headers, payload)
            results.append(('recovered', status, ms))
        else:
            results.append(('direct', status, ms))

async def main():
    ap = argparse.ArgumentParser()
    ap.add_argument('--requests', type=int, default=60)
    ap.add_argument('--concurrency', type=int, default=4)
    ap.add_argument('--prompt', default='Reply with the single word ok.')
    args = ap.parse_args()

    url = os.environ['ENDPOINT_URL']
    key = os.environ['ENDPOINT_KEY']
    model = os.environ['PROBE_MODEL']
    headers = {'Authorization': f'Bearer {key}', 'Content-Type': 'application/json'}
    payload = {
        'model': model,
        'messages': [{'role': 'user', 'content': args.prompt}],
        'max_tokens': 8,
    }

    sem = asyncio.Semaphore(args.concurrency)
    results = []
    async with httpx.AsyncClient() as client:
        tasks = [
            worker(client, url, headers, payload, sem, results)
            for _ in range(args.requests)
        ]
        await asyncio.gather(*tasks)

    ok = [ms for _, status, ms in results if status == 200]
    limited = [ms for kind, _, ms in results if kind == 'recovered']
    failed = [ms for _, status, ms in results if status not in (200, 429)]

    print(f'requests={len(results)} ok={len(ok)} rate_limited={len(limited)} failed={len(failed)}')
    if ok:
        ok.sort()
        p95 = ok[min(len(ok) - 1, int(len(ok) * 0.95))]
        print(f'latency_ms p50={statistics.median(ok):.0f} p95={p95:.0f} max={ok[-1]:.0f}')
    if limited:
        print('429s observed; the probe waited 2s and retried once. If recovered entries are 429 again, the endpoint needs a longer cooldown than your agent timeout.')

if __name__ == '__main__':
    asyncio.run(main())

Read the output like this. If p95 latency sits close to p50, the endpoint queues fairly under load. If p95 is three times p50, you are seeing contention, and your agent’s timeouts will fire at the worst possible moment. If 429s appear at your expected burst size, the free tier’s contract does not match your traffic shape. No retry logic fixes that; it only converts a rate limit into a token incinerator.

请这样解读输出：如果 p95 延迟接近 p50，说明端点在负载下排队公平。如果 p95 是 p50 的三倍，说明存在资源争用，你的智能体超时将在最糟糕的时刻触发。如果在预期的突发流量规模下出现了 429 错误，说明免费层级的合同与你的流量形态不匹配。任何重试逻辑都无法解决这个问题，它只会将速率限制变成 Token 的“焚化炉”。

Record four numbers for each run: success rate, 429 count, p50, and p95. That is your endpoint’s signature. Compare the signature at concurrency 1 and concurrency 8; if p95 doubles while success rate drops, the endpoint is not built for your agent’s parallel tool calls. Some agents fan out ten tool calls at once, and a free tier that handles one request gracefully can still fail that pattern. The probe results feed directly into the decision table below. A high 429 count at your working concurrency moves you to the “risky” row regardless of how cheap the tokens are.

记录每次运行的四个数值：成功率、429 次数、p50 和 p95。这就是你端点的“签名”。比较并发 1 和并发 8 下的签名；如果 p95 翻倍且成功率下降，说明该端点并非为你的智能体并行工具调用而设计。有些智能体会同时发出十个工具调用，一个能优雅处理单个请求的免费层级在面对这种模式时仍可能失败。探测结果直接决定了下方的决策表。在你的工作并发数下，如果 429 次数过高，无论 Token 多便宜，你都处于“高风险”行。

Workload shape	Free hosted tier	Self-hosted
Dev/test, low volume	Fits	Overkill
Steady production, non-sensitive	Fits with a fallback	Predictable but costly
Bursty, latency-sensitive	Risky	Better control
Regulated or private data	Avoid	Required
No operational slack	Fits	Do not attempt

工作负载形态	免费托管层级	自托管
开发/测试，低流量	适用	过度设计
稳定生产，非敏感	适用（需备选方案）	可预测但昂贵
突发，延迟敏感	高风险	更好的控制
受监管或私有数据	避免	必须
无运维余力	适用	请勿尝试

Notice what is missing from the table: price. Price decides which self-hosted option you pick, not whether you self-host. The free tier’s real cost is coupling — your agent’s availability inherits someone.

注意表中缺少了什么：价格。价格决定了你选择哪种自托管方案，而不是决定你是否应该自托管。免费层级的真实成本是“耦合”——你智能体的可用性继承了某人的（服务质量）。