Sloth Roll: an on-device vision model that turns my screenshots into calendar events, places and expenses. I only swipe.

本文为原文前 6,000 字符的节选翻译，完整内容请查看原文。

Title: Sloth Roll: an on-device vision model that turns my screenshots into calendar events, places and expenses. I only swipe.

Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady Submission 🦥 This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady What I Built My camera roll is full of screenshots of things I meant to do later. A flyer for a tournament on Saturday. A restaurant someone recommended. A receipt I should have written down. I take the screenshot, and that’s where it ends. Later never comes. So I built an app for that later.

Hacktoberfest 开源 AI 挑战赛第三周：稳扎稳打提交 🦥 这是我为 Hacktoberfest 开源 AI 挑战赛第三周“稳扎稳打”所做的提交。我构建了什么？我的相机胶卷里塞满了打算以后处理的截图。比如周六比赛的传单、别人推荐的餐厅，还有我本该记下来的收据。我截了图，然后就没有然后了，“以后”永远不会到来。所以我为这些“以后”开发了一个应用。

Sloth Roll reads my screenshots on the iPhone with an open-weight vision model and turns each one into a card: an event goes to Calendar, a place opens in Apple Maps, a receipt is saved to an expense list inside the app. I don’t type, copy, paste, or switch apps. I only swipe: right to do it, left to skip it. Tap a card to fix the title, date, place or amount first.

Sloth Roll 使用开源视觉模型在 iPhone 上读取我的截图，并将每一张截图转化为一张卡片：活动会被存入日历，地点会在 Apple 地图中打开，收据则会被保存到应用内的费用清单中。我无需打字、复制、粘贴或切换应用。我只需要滑动：向右滑动执行，向左滑动跳过。点击卡片可以先修改标题、日期、地点或金额。

It also does the cleanup I keep putting off. Screenshots that are blurry, near-duplicates, or that the model calls not useful end up in a Clean up tab, and I delete them in one tap. They go to Recently Deleted, through iOS’s own confirmation, so nothing disappears by surprise. Everything runs on the phone. After a one-time model download, it works in airplane mode.

它还能处理我一直拖延的清理工作。模糊的、近乎重复的，或者模型认为无用的截图会被归入“清理”标签页，我只需轻轻一点即可删除。它们会进入“最近删除”，并经过 iOS 系统的确认流程，因此不会有任何东西意外消失。所有操作都在手机上运行。在一次性下载模型后，它可以在飞行模式下工作。

The lazy payoff. In one scan of my real library, the app went through 50 new images, read 43 of them with the model, and found 19 actions, in about 3.7 minutes, while the phone sat on the table. My part was 19 swipes. If adding one event or expense by hand takes about a minute, that’s roughly 19 minutes of copying turned into under a minute of swiping. That’s my idea of Slow and Steady: the app is the slow one. It reads one image at a time, remembers what it already read, and waits for me. I swipe whenever I feel like it.

懒人的回报。在对我真实图库的一次扫描中，该应用处理了 50 张新图片，模型读取了其中的 43 张，并发现了 19 个操作，整个过程耗时约 3.7 分钟，期间手机就放在桌子上。我所做的只是 19 次滑动。如果手动添加一个活动或费用大约需要一分钟，那么这相当于将 19 分钟的复制工作缩短到了不到一分钟的滑动。这就是我心目中的“稳扎稳打”：应用本身是“慢”的那一个。它一次读取一张图片，记住已读内容，并等待我的指令。我可以随时随地进行滑动。

Code Martin-EG / Sloth-Roll Sloth Roll You took the screenshot. Sloth Roll does the rest. Overview An iPhone app that reads your screenshots with an open-weight vision model running on the phone and turns each one into a card: an event for Calendar, a place for Maps, or a receipt for your expense list. You only swipe. Nothing is uploaded, and after a one-time model download it works in airplane mode.

代码 Martin-EG / Sloth-Roll Sloth Roll 你负责截图，Sloth Roll 负责剩下的工作。概述：这是一款 iPhone 应用，它使用在手机上运行的开源视觉模型读取你的截图，并将每一张转化为卡片：日历活动、地图地点或费用清单中的收据。你只需要滑动。没有任何数据会被上传，且在一次性下载模型后，它可以在飞行模式下工作。

It’s a SwiftUI app (Swift 6, iOS 18+). The model is mlx-community/Qwen3-VL-2B-Instruct-4bit, a 4-bit conversion of Qwen3-VL 2B. It runs on the iPhone’s GPU with MLX Swift and mlx-swift-lm. The download is about 1.8 GB, and it takes about 2 GB of memory while loaded. I tested on an iPhone 17 Pro.

这是一个 SwiftUI 应用（Swift 6，iOS 18+）。模型采用的是 mlx-community/Qwen3-VL-2B-Instruct-4bit，这是 Qwen3-VL 2B 的 4 位量化版本。它通过 MLX Swift 和 mlx-swift-lm 在 iPhone 的 GPU 上运行。下载大小约为 1.8 GB，加载时占用约 2 GB 内存。我在 iPhone 17 Pro 上进行了测试。

Triage first, model second. A vision model is slow and heavy, so a scan works in two passes. First, with no model loaded, every new image gets cheap checks. Camera photos are skipped (they have camera make, model and lens info in the file). Apple’s Vision text recognition must find at least 20 characters, or the image is skipped. The shortest real action in my test set had 40. The same pass measures blur and takes a Vision feature print for finding duplicates, so each image is opened only once.

先筛选，后模型。视觉模型运行缓慢且沉重，因此扫描分为两步。第一步，在不加载模型的情况下，对每张新图片进行低成本检查。相机拍摄的照片会被跳过（因为文件里包含相机品牌、型号和镜头信息）。Apple 的 Vision 文字识别必须至少找到 20 个字符，否则图片会被跳过。我的测试集中最短的实际操作包含 40 个字符。同一阶段还会测量模糊度并提取 Vision 特征指纹以查找重复项，因此每张图片只需打开一次。

Then the model loads once, reads only the images that passed, newest first, saves each answer right away, and unloads at the end. Dates relative to the screenshot, not to today. My first prompt resolved “hoy” (“today”) against the day I ran the scan, so a cinema ticket from July came back dated October. Now the prompt includes the day the screenshot was taken, plus the next seven days spelled out with weekday names, in Spanish and English. “Mañana”, “este sábado” and “hoy” land on the right day.

随后模型加载一次，仅读取通过筛选的图片（从最新开始），立即保存每个答案，并在最后卸载。日期是相对于截图时间而非今天。我最初的提示词将“hoy”（今天）解析为我运行扫描的那一天，导致七月的电影票被标记为十月。现在，提示词包含了截图拍摄的日期，以及接下来的七天（用西班牙语和英语拼写出星期几）。“Mañana”（明天）、“este sábado”（这个周六）和“hoy”（今天）都能对应到正确的日期。

The model’s answer is untrusted. It answers with one JSON object: type, title, date, time, location, amount, currency, confidence. In practice it wraps the JSON in code fences, adds chatter, leaves trailing commas, and once answered with {“error”: ”…”}. The parser cuts the answer down to the JSON object, checks every field (a real date, a real 24-hour time, an amount that isn’t negative), and turns unknown values into safe defaults. A bad answer gets one stricter retry, then the card says “couldn’t read this one”. It never crashes on bad output.

模型的回答是不可信的。它会返回一个 JSON 对象：类型、标题、日期、时间、地点、金额、货币、置信度。实际上，它会用代码块包裹 JSON，添加闲聊内容，留下多余的逗号，甚至有一次返回了 {“error”: ”…”}。解析器会将答案裁剪为纯 JSON 对象，检查每个字段（有效的日期、有效的 24 小时制时间、非负金额），并将未知值转换为安全默认值。错误的回答会触发一次更严格的重试，如果依然失败，卡片会显示“无法读取此项”。它绝不会因为错误的输出而崩溃。

One small prompt change mattered a lot. One version of the prompt made the model drop the “type” key on 23 of 24 images, so almost every image needed a retry. Listing “type” as a required first field brought that to zero retries. Order numbers aren’t money. The model kept reading order and booking numbers as amounts. The prompt now says amounts must be money (a price or a total) and calls out folio, order and ticket numbers as not amounts.

一个小小的提示词改动产生了巨大影响。旧版本的提示词导致模型在 24 张图片中有 23 张丢失了“type”键，几乎每张图片都需要重试。将“type”列为必需的首个字段后，重试次数降为零。订单号不是金额。模型曾不断将订单号和预订号误读为金额。现在的提示词明确指出金额必须是货币（价格或总计），并特别说明流水号、订单号和票号不是金额。

Clean up reuses the same model. Blur is a Laplacian variance score, with a contrast check so a plain blank screen doesn’t count as blurry. Duplicates are screenshots whose Vision feature prints are very close; the newest one is kept. For everything else, the model that’s already loaded gets one question: “Is this screenshot still useful to keep? Answer only yes or no.”

清理功能复用了同一个模型。模糊度通过拉普拉斯方差评分确定，并带有对比度检查，以确保纯空白屏幕不会被误判为模糊。重复项是指 Vision 特征指纹非常接近的截图；系统会保留最新的一张。对于其他所有内容，已加载的模型会被问一个问题：“这张截图还有保留价值吗？只回答 yes 或 no。”