cactus-compute / needle

查看原文 / View Original 编辑 / Edit

Needle 2

Needle 2 is an open 45M-parameter model for tool calling, device use and structured extraction. The whole model is a single 14MB binary that runs a full session in about 28MB of RAM. It is built on our Simple Attention Network findings, compressed to CQ2-bit with Cactus Quants, and baked into its own engine. On the benchmarks below, Needle 2 trades wins with other small models like FunctionGemma 270M, LFM2.5 230M and Apple FM, at 5x to 70x smaller, and 2 bits against their f16.

Needle 2 是一个开源的 45M 参数模型，专为工具调用、设备使用和结构化数据提取而设计。整个模型是一个 14MB 的单一二进制文件，在运行完整会话时仅需约 28MB 内存。它基于我们“简单注意力网络”（Simple Attention Network）的研究成果构建，通过 Cactus Quants 压缩至 CQ2-bit，并集成在专属引擎中。在下方的基准测试中，Needle 2 与 FunctionGemma 270M、LFM2.5 230M 和 Apple FM 等其他小型模型互有胜负，但体积缩小了 5 到 70 倍，且仅使用 2-bit 量化，而对比模型则为 f16。

This repository is the Python package: inference, LoRA fine-tuning, and export. pip install cactus-needle, describe your tools, and call them from Python. The inference engine is fetched once from Hugging Face and cached; there is nothing else to build.

本仓库提供了 Python 包，支持推理、LoRA 微调和导出功能。通过 pip install cactus-needle 安装后，即可在 Python 中描述并调用你的工具。推理引擎会从 Hugging Face 下载一次并缓存，无需进行其他构建。

  • Self-contained: weights baked into a single 14MB engine; no separate model files to manage, and inference does no network.

  • Simple contract: tool calls come back as structured data, text in, JSON out; a byte-level grammar compiled from your schemas constrains every token.

  • Confidence-gated: every response carries a calibrated confidence score from a learned head; set a threshold, act above it, escalate below it.

  • Tool retrieval: declare a large catalogue and a built-in retrieval head renders only the top five tools per turn, with the grammar constrained to that subset.

  • Bounded memory: a 256-token sliding window with the tools pinned as KV sinks, so total memory stays near 28MB no matter how long the conversation runs.

  • 自包含：权重被集成在 14MB 的单一引擎中；无需管理单独的模型文件，推理过程不依赖网络。

  • 简单契约：工具调用以结构化数据返回，输入文本，输出 JSON；通过从你的 Schema 编译出的字节级语法来约束每一个 Token。

  • 置信度门控：每个响应都带有由学习头（learned head）计算出的校准置信度分数；你可以设置阈值，高于阈值则执行，低于则升级处理。

  • 工具检索：即使声明了庞大的工具库，内置的检索头每轮也仅呈现前五个工具，并将语法约束在该子集内。

  • 内存受限：采用 256-token 的滑动窗口，并将工具固定为 KV 槽（KV sinks），因此无论对话运行多久，总内存占用始终保持在 28MB 左右。

Weights: huggingface.co/Cactus-Compute/needle2 · Source: github.com/cactus-compute/needle

权重：huggingface.co/Cactus-Compute/needle2 · 源码：github.com/cactus-compute/needle

Simple Attention Network

Needle 2 is a Simple Attention Network, our dense small-model recipe: a Hadamard MLP in place of the FFN, GQA attention, engram key-value memory, and multi-lane hyper-connections. See the paper for the design and ablations: arXiv:2607.18363.

简单注意力网络 (Simple Attention Network)

Needle 2 采用了我们的密集型小型模型方案——简单注意力网络：用 Hadamard MLP 替代 FFN，使用 GQA 注意力机制、Engram 键值内存以及多通道超连接。设计细节与消融实验请参阅论文：arXiv:2607.18363。

Each block carries its update rule. Here x̂ is the RMS-normalised flattening of the four residual streams, H the orthonormal Walsh-Hadamard transform (a fixed matrix, applied in n log n time with no weights to read), (kₜ, vₜ) rows gathered from hashed n-gram tables, and P the doubly-stochastic normalisation of the routing logits A, computed by Sinkhorn iteration; a, b, g and all σ-gates are learned and input-dependent. Both attention and MLP residuals are sandwich-normed and gated, the engram sites fire at two layers, and decoding is constrained by a byte-level grammar compiled from the declared schemas.

每个模块都包含其更新规则。其中 x̂ 是四个残差流的 RMS 归一化展平，H 是正交 Walsh-Hadamard 变换（一个固定矩阵，以 n log n 时间复杂度应用，无需读取权重），(kₜ, vₜ) 行是从哈希 n-gram 表中收集的，P 是路由 Logits A 的双随机归一化（通过 Sinkhorn 迭代计算）；a、b、g 和所有 σ-门控都是可学习且依赖于输入的。注意力残差和 MLP 残差均经过夹层归一化（sandwich-normed）和门控，Engram 位点在两层触发，解码过程则受到从声明的 Schema 编译出的字节级语法的约束。

Quickstart

pip install cactus-needle

Needle reads your tool descriptions to decide what to call and how to fill arguments, so describing them well is the whole game. You can do it three ways, from least to most control.

快速开始

pip install cactus-needle

Needle 通过读取你的工具描述来决定调用什么以及如何填充参数，因此写好描述是关键。你可以通过以下三种方式实现，控制力由弱到强：

Simple: decorate a function. The signature gives the argument types, the docstring is the tool description, and run() completes the loop: model picks the call, Needle executes your function, feeds the result back, and returns the final response with the executed tool results attached as results.

简单模式：装饰函数。 函数签名提供参数类型，Docstring 作为工具描述，run() 完成整个循环：模型选择调用，Needle 执行你的函数，反馈结果，并返回最终响应，其中包含已执行的工具结果。

import needle

@needle.tool
def get_weather(city: str):
    "Get the current weather for a city."
    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
print(agent.run("what's it like in Lagos right now?")["results"])
# [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

Medium: describe each argument and offer choices. Needle reads a Google-style Args: block for per-parameter descriptions; a default makes an argument optional; a Literal becomes a fixed set the model must choose from (it cannot emit anything else).

中级模式：描述每个参数并提供选项。 Needle 会读取 Google 风格的 Args: 块以获取参数描述；设置默认值使参数变为可选；Literal 则定义了模型必须从中选择的固定集合（模型无法输出其他内容）。

from typing import Literal

@needle.tool
def set_thermostat(temperature: int, mode: Literal["heat", "cool", "auto"] = "auto"):
    """Set the thermostat.
    Args:
      temperature: target temperature in Celsius
      mode: heating strategy to use
    """
    return {"temperature": temperature, "mode": mode}

agent = needle.Needle(tools=[set_thermostat])
agent.run("make it 21 and cool the room")

Advanced: constrain the values with needle.Field, attached inline via Annotated. Ranges, patterns, lengths, and item counts are compiled into the decode grammar, so the model can only ever emit values that satisfy them.

高级模式：使用 needle.Field 约束值，通过 Annotated 内联附加。 范围、模式、长度和项目计数会被编译进解码语法中，因此模型只能输出满足这些条件的值。

from typing import Annotated

@needle.tool
def send_money(
    amount: Annotated[float, needle.Field(gt=0, le=10000, description="USD, up to 10,000")],
    to: Annotated[str, needle.Field(pattern=r"^@[a-z0-9_]+$", description="recipient handle")],
    memo: Annotated[str, needle.Field(max_length=80)] = "",
):
    "Send money to a handle."
    return {"sent": amount, "to": to}

Field supports description, enum, const, ge/le/gt/lt, multiple_of, min_length/max_length, pattern, format, min_items/max_items, and unique_items.

Field 支持 description、enum、const、ge/le/gt/lt、multiple_of、min_length/max_length、pattern、format、min_items/max_items 和 unique_items。

Extraction: to pull structured data out of text, declare the shape and call extract(). Pass a Pydantic model and you get a typed object back.

提取模式：若要从文本中提取结构化数据，声明形状并调用 extract()。传入一个 Pydantic 模型，你将获得一个类型化的对象。

from pydantic import BaseModel

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

invoice = needle.extract("Invoice from Acme Corp, $1,200.00, due 2026-09-01", Invoice)
print(invoice.vendor, invoice.total) # -> Acme Corp 1200.0

By hand - the decorator just builds a JSON schema; you can pass that schema directly, which is exactly what Needle consumes.

手动模式 - 装饰器本质上只是构建了一个 JSON Schema；你可以直接传入该 Schema，这正是 Needle 所消费的内容。

← 返回新闻列表 / Back to news list