Stop letting your AI agent test Android apps with screenshots

别再让你的 AI 智能体通过截图来测试 Android 应用了

I shipped an Android app mostly by pairing with Claude Code. Writing features went fine. Testing them on a real phone did not. 我发布了一款 Android 应用，开发过程主要是与 Claude Code 结对编程。编写功能一切顺利，但在真机上进行测试时却遇到了麻烦。

The problem: When an AI coding agent tests an app on a device, the usual loop is: take a screenshot, look at it, decide where to tap, tap, take another screenshot. Every step sends an image to the model. A 20-step login-and-checkout flow can cost thousands of image tokens, fill the context window, and the agent still sometimes taps the wrong thing. 问题在于：当 AI 编程智能体在设备上测试应用时，通常的循环是：截屏、查看截图、决定点击位置、点击、再截屏。每一步都会向模型发送一张图片。一个 20 步的登录和结账流程可能会消耗数千个图像 Token，填满上下文窗口，而且智能体有时还是会点错地方。

Most of that work doesn’t need eyes. “Tap Sign in, wait, check that Home is on screen” is a text problem. 大部分工作其实根本不需要“视觉”。“点击登录，等待，检查主页是否显示”本质上是一个文本处理问题。

The idea: Android can already describe the screen as text: uiautomator dump returns every visible node with its text and bounds. So instead of the agent driving the phone step by step, it writes the steps once as JSON: 思路是：Android 本身就能以文本形式描述屏幕：uiautomator dump 命令可以返回每个可见节点的文本及其边界信息。因此，与其让智能体一步步操作手机，不如让它一次性写好 JSON 格式的步骤：

{
  "name": "Login works",
  "app": "com.example.app",
  "steps": [
    {"app": "restart"},
    {"type": ["Email", "test@example.com"]},
    {"type": ["Password", "secret123"]},
    {"tap": "Sign in", "exact": true},
    {"expect": ["Home"], "wait": 3}
  ]
}

…and runs it with one command: ……然后通过一条命令运行它：

$ python scenario.py scenarios/login.json PASS Login works (5 steps, 12 s)

One line back. If a step fails, the runner prints the failing step, what was on screen, and saves a screenshot, which is usually enough for the model to fix either the test or the app: 只需一行代码。如果某一步失败，运行器会打印出失败的步骤、屏幕上的内容，并保存一张截图，这通常足以让模型修复测试脚本或应用本身：

✗ 5 expect: expected ['Home'] (found []); screen: ['Sign in', 'Wrong password', ...] FAIL Login works step 5/5 (9 s)

The model only opens a screenshot when a visual decision is actually needed (layout, colours). 模型只有在真正需要视觉判断（如布局、颜色）时才会查看截图。

What real devices taught me

真机测试教会我的事

Most of the code is not the happy path; it’s the stuff that broke on real phones: 大部分代码逻辑并非“理想路径”，而是那些在真机上容易出错的地方：

Stale dumps. When uiautomator dump fails during an animation, the old ui.xml is still on the device and gets pulled again, so the test “sees” the previous screen. The runner deletes the file on both sides before every dump. 陈旧的转储文件。 当 uiautomator dump 在动画过程中失败时，旧的 ui.xml 仍留在设备上并被再次拉取，导致测试“看到”的是上一个屏幕。运行器会在每次转储前删除两端的文件。
The navigation bar. Text drawn under the translucent system bar is “visible”, but tapping it presses Home. Those nodes are ignored. 导航栏。 在半透明系统栏下方绘制的文本虽然是“可见”的，但点击它会触发 Home 键。这些节点会被忽略。
Password fields drop characters when adb input text sends the whole string at once. Type character by character. 密码字段。 当 adb input text 一次性发送整个字符串时，密码字段会丢失字符。必须逐个字符输入。
Other apps steal the foreground (ads, OEM notification managers). The runner notices and brings the app back. 其他应用抢占前台（如广告、厂商通知管理器）。运行器会检测到这种情况并将应用切回前台。
Switches on settings rows with a multi-line description sit ~180 px below the title. Matching by distance fails; matching the smallest container that holds both the text and one Switch works. 设置行中的开关。 带有长描述的设置行，开关通常位于标题下方约 180 像素处。通过距离匹配会失败；匹配同时包含文本和开关的最小容器则可行。
Airplane mode lies. Android remembers Wi-Fi you turned on manually while in airplane mode, so an “offline” test can silently run online. The runner turns Wi-Fi off explicitly and waits until wlan0 has no IP. 飞行模式会撒谎。 Android 会记住你在飞行模式下手动开启的 Wi-Fi，因此“离线”测试可能会在不知不觉中联网运行。运行器会显式关闭 Wi-Fi 并等待直到 wlan0 没有 IP 地址。
Airplane mode kills wireless adb, stranding the phone. It’s refused unless you’re on USB. 飞行模式会断开无线 ADB，导致手机失联。除非使用 USB 连接，否则无法恢复。

Using it from an agent

在智能体中使用

Add a few lines to your CLAUDE.md / AGENTS.md: 在你的 CLAUDE.md 或 AGENTS.md 中添加几行：

## Device testing
- Never drive the phone step by step with screenshots. Write a scenario in scenarios/<name>.json and run: python scenario.py scenarios/<name>.json | tail -3
- To see the current screen as text: python ui.py dump

I ran ~90 scenarios this way on my own app before extracting the runner into its own repo. It’s Python standard library + adb, nothing to install on the phone, MIT licensed: 我在自己的应用上通过这种方式运行了约 90 个场景，随后将这个运行器提取到了独立的仓库中。它仅依赖 Python 标准库和 adb，无需在手机上安装任何东西，采用 MIT 协议开源：

https://github.com/sinangumuskabak-sys/android-scenario-runner

It isn’t a replacement for Espresso or Maestro. It’s a single file you can hand to an agent so it tests cheaply. Issues and PRs welcome, especially from people on OEM skins I haven’t tried. 它不是 Espresso 或 Maestro 的替代品。它只是一个你可以交给智能体的单文件工具，让它能以低成本进行测试。欢迎提交 Issue 和 PR，特别是那些使用我尚未测试过的厂商定制系统（OEM skins）的用户。