A single function Jev-like wrapper for LLMs, including vision models

一个用于大语言模型（含视觉模型）的单函数 Jev 类封装器

I was intrigued by Jev and the self-hostable projects appearing around it, such as OpenJev and SemIf. Reading about them introduced me to a neat trick: reading an LLM’s token probabilities. Apparently this is an old trick for some people. See e.g. OpenAI’s logprobs cookbook. But it was new to me. 我对 Jev 以及围绕它出现的自托管项目（如 OpenJev 和 SemIf）很感兴趣。阅读这些项目让我学到了一个巧妙的技巧：读取大语言模型（LLM）的 Token 概率。显然，这对某些人来说是个老把戏了，例如可以参考 OpenAI 的 logprobs 指南，但这对我来说是全新的。

I believe the basic idea is to write a prompt like this: 我认为其基本思路是编写如下提示词：

State: My order arrived broken and I want a refund. Question: Which team should handle this? [A] billing [B] shipping [C] returns Answer with the letter of the best option only. 状态：我的订单送达时已损坏，我想要退款。 问题：哪个团队应该处理这个问题？ [A] 账单 [B] 运输 [C] 退货 仅回答最佳选项的字母。

Then add a few JSON request parameters to a compatible Chat Completions request: 然后向兼容的 Chat Completions 请求中添加几个 JSON 请求参数：

{
  "max_completion_tokens": 1,
  "logprobs": true,
  "top_logprobs": 20
}

The LLM API will return the letter plus the model’s log probabilities for alternative tokens. Repeat for each question. Forcing it to generating only one token avoids a lengthy answer and is super quick, though processing the input still costs time. Though for each of the questions a shared state prefix can be KV-cached if the backend supports it. LLM API 将返回该字母以及模型对替代 Token 的对数概率。对每个问题重复此操作。强制其仅生成一个 Token 可以避免冗长的回答，且速度极快，尽管处理输入仍需时间。不过，如果后端支持，每个问题的共享状态前缀可以进行 KV 缓存。

The fun part: this works with vision models too. Jev’s documented request format currently describes only text/JSON state. I added an attachments field for images for my local experiments. My example captures webcam frames, sends base64 JPEGs, and prints a table: is a person visible, are we indoors or outdoors, and how bright is the scene? 有趣的部分在于：这也适用于视觉模型。Jev 目前记录的请求格式仅描述了文本/JSON 状态。我在本地实验中添加了一个用于图像的 attachments 字段。我的示例会捕获摄像头帧，发送 Base64 编码的 JPEG，并打印一个表格：是否有人可见、是在室内还是室外、场景亮度如何？

With Gemma 4 12B on my RTX 3090, I get around 1 frames per second, with three questions per frame. I also ran it against OpenAI gpt-6-luna and got around 0.2 FPS. Presumably because I didn’t make any effort to avoid the cost of a separate connection through their system per question per frame. Specialized computer vision models surely are much more efficient, but what I like here is the flexibility: change a condition by describing it in plain text. 在我的 RTX 3090 上运行 Gemma 4 12B 时，每秒大约能处理 1 帧，每帧包含三个问题。我也在 OpenAI 的 gpt-6-luna 上运行过，大约是 0.2 FPS。这大概是因为我没有采取任何措施来避免每帧每个问题都要通过其系统建立单独连接的开销。专业的计算机视觉模型肯定效率更高，但我喜欢的是这里的灵活性：只需用纯文本描述即可更改条件。

Here’s the standalone Python example (OpenCV is just used for convenient access to the webcam, not for any actual computer vision): 以下是独立的 Python 示例（OpenCV 仅用于方便访问摄像头，而非进行任何实际的计算机视觉处理）：

(Code block omitted for brevity, as requested)