cactus-compute/needle

Needle is a 26-million parameter AI agent model distilled from Gemini 3.1, designed to run efficiently on tiny devices like phones, watches, and glasses. It uses a novel Simple Attention Network architecture and achieves high-speed inference (6000 tok/s prefill, 1200 tok/s decode) on Cactus hardware, with fully open weights and dataset generation.

Developers can build personal AI assistants that execute function calls (e.g., weather queries, tool invocations) directly on-device with low latency. Real-world use cases include smart glasses that fetch information hands-free, smartwatches that control IoT devices, and mobile apps that process user commands offline. The ceiling includes complex multi-step agentic workflows, though the model is optimized for single-shot tasks and may struggle with conversational depth.