AntOS AI Subsystem

AntOS can call out to language models, and does so in two visible places: the console's ? natural-language helper (see AntOS) and the ai command (a full chat client). Both sit on one small subsystem — a set of configured providers, split by use-type, reached through a couple of Lua bindings (ai.ask / ai.askSimple; the concise binding reference is luau_ai). This document describes that subsystem, including the MCP client — how a model reaches external tools — which is now built across all provider styles.

Status: built. Everything here is implemented and compiles: the C core (all three API styles), the Lua bindings, the mcp library/command, and the ai tool-use loop. What remains is a real round-trip against a live MCP server to confirm the wire formats — the openai-style path (Groq/Ollama/OpenAI/Kimi) is the most exercised; the gemini and anthropic tool formats are emitted from spec and want on-hardware confirmation.

The shape of it

The AI subsystem is deliberately thin. It is not a model, and it is not a fixed assistant — it is a way for scripts to ask a model something, with the provider, model name, and key kept in config rather than baked in. Like the rest of AntOS, the interesting behaviour (the ? helper, the ai chat app) is Luau on top of a small C core, so it can be read, changed, or replaced.

Two entry points do the actual work, both C-side bindings:

overrides carries the per-call settings — provider, model, key, timeout, max_tokens, temperature, system. Everything else (provider catalogue, config storage, history) is built around these two calls.

Providers

A provider is a model endpoint AntOS knows how to talk to. ai.providers() returns the catalogue; each entry looks like:

ai.fetchModels{ provider, key } pulls a provider's current model list over the network when you're configuring, so the menu reflects what the account can actually reach rather than a stale built-in list.

The shipping catalogue is six providers across three API styles — Groq, Ollama (local), OpenAI, and Kimi (Moonshot AI) are all openai-style, Google Gemini is gemini-style, and Anthropic is anthropic-style. The C core dispatches on the style, not the individual provider, so any OpenAI-compatible endpoint (Groq, Ollama, OpenAI, Kimi, …) shares one code path.

New providers are added to the catalogue in the C core — but because callers only ever name a provider by id in config, adding one doesn't touch any script.

Use-types

A use-type is what you're asking for, not who answers. The subsystem keys configuration by use-type so different jobs can route to different providers and models — a cheap fast model for the ? helper, a stronger one for long-form chat, a code-tuned one for programming help — without the calling script caring.

Config is stored per use-type as ai.<type>.<key> (e.g. ai.chat.provider, ai.chat.model, ai.chat.key). ai config walks you through setting a provider and model for each use-type you want, and tests the connection with a one-word round-trip.

This is the same declare-truth-once pattern used across AntOS — the wiki's document providers, the radio-capability model on Phreak, and the os.arguments command specs all work this way: a registry of typed entries that UI and scripts both read, rather than behaviour hard-coded in one place.

The two front ends

? — the natural-language console helper. Start a console line with ? and ask in plain English. It answers from the live system first — finding files, reading program descriptions, checking running tasks, searching by tag — and for open-ended questions hands off to the configured model, grounded with a list of your actual installed programs so the answer fits your machine. The whole thing is a Luau script; it's meant to be read and changed.

ai — the chat client. A full multi-turn chat app. It keeps a persisted conversation history per use-type (stored via the config keys above, encoded so messages survive round-trips), sends the whole history through ai.ask so the model has context, and renders replies as markdown in the console. ai config manages providers; ai with a question starts or continues a chat.

Scripts can also call the model directly through the ai library — the same ai.ask / ai.askSimple the front ends use — so any script can be AI-assisted.

Tool use and MCP

Today ai.ask returns prose: you send messages, you get text back. The next step is tool use — letting the model do things mid-answer (read a file, call an API, query the system) by returning a structured "call this tool" request that AntOS executes and feeds back. The chosen standard for exposing tools is MCP (Model Context Protocol) — an open protocol where a server advertises tools and a client lets a model call them.

AntOS will adopt this client-first: the ai subsystem connects out to MCP servers to gain abilities. Making AntOS an MCP server — so an external AI on a PC can drive the machine — is a deliberate later step (see the end of this section).

Why it belongs inside the ai subsystem

Tool use isn't a separate feature bolted alongside AI — it's a property of the request/response cycle. The tool list must ride with the request, and the model's reply must be able to say "call tool X" instead of answering. So the work extends the existing ai.ask path rather than sitting beside it. The payoff: a model's tools can come from a remote MCP server or from AntOS's own os.arguments-specced scripts through the same mechanism — remote and local tools normalised into one vocabulary, the same "unify one layer in" approach the Phreak BLE HID input layer uses for input sources.

The three seams

1. overrides.tools — sending the tool list. A new optional field on the ai.ask overrides: a list of tool definitions, each a { name, description, parameters } (parameters as a JSON schema), in the internal vocabulary. When present, the active provider's adapter translates them into that provider's tool-calling format (see Provider-agnostic tool calling below). The list AntOS builds is the same regardless of provider; only the adapter differs.

2. A three-value return — getting a tool call back. ai.ask gains a third return value: reply, err, tool_calls, where tool_calls (when non-nil) is a list of { name, args } the model wants run. Existing callers that read only reply, err keep working unchanged — Lua ignores extra returns — so this is backward-compatible and nothing in today's ai library breaks.

3. An mcp helper beside ai — talking to servers. A small library, not a rival to ai:

It's JSON-RPC over HTTP, reusing the HTTP client the get / download commands already use. The minimal useful subset of MCP is just three methods — initialize, tools/list, tools/call. Resources, prompts, and sampling are optional protocol surface to skip until wanted.

Provider-agnostic tool calling

The subsystem already routes different use-types to different providers, and it already dispatches on an API style rather than per-provider: each provider in the catalogue declares a style, and the C core sends the request through the matching handler. Today those handlers are ask_openai, ask_gemini, and ask_anthropic, and the shipping providers map onto them like this:

That's the adapter layer already — three styles, not six providers. Tool use extends the style handlers, so adding tool support means touching three code paths, and every provider that shares a style comes along for free (native tool use for Groq/Ollama/OpenAI/Kimi lands in one place). AntOS defines one internal tool vocabulary and each style handler translates to and from its own dialect — the same declare-once-then-adapt pattern as the LED driver abstraction (one set-colour/fade API over PCA9685 or KTD2026).

The internal vocabulary is provider-neutral and small:

Each style handler implements three translations against that vocabulary:

1. outgoing tools — turn the internal tool list into this style's request field:
   • openai: a top-level tools array of { type: "function", function: {...} }; tool calls come back in choices[].message.tool_calls.
   • anthropic: a top-level tools array of { name, description, input_schema }; tool calls come back as tool_use blocks in the content array.
   • gemini: a tools array of functionDeclarations; calls come back as functionCall parts in the response content.
2. incoming tool calls — parse that style's "call a tool" response into the internal { id, name, args } list (each style names and nests it differently, per the above).
3. outgoing tool results — format an executed result as the follow-up message the style expects, keyed to the call id:
   • openai: a message with role: "tool" and the matching tool_call_id.
   • anthropic: a user message containing a tool_result block with tool_use_id.
   • gemini: a functionResponse part.

Adding a new provider that reuses an existing style needs no tool-use work at all — it inherits its style handler. A genuinely new style (a future provider that's neither OpenAI-, Gemini-, nor Anthropic-shaped) adds a fourth handler with these same three translations; the tool-use loop, the mcp helper, and every calling script stay untouched.

For a provider whose style has no native tool calling, the handler uses a generic text fallback: describe the tools in the system prompt, ask the model to emit a tool call as a small JSON block, and parse that back into the internal { id, name, args }. Less reliable than native tool calling, but it means even a plain chat model drives tools — the capability degrades gracefully rather than vanishing. (All three current styles support native tool calling, so the fallback is a safety net for future additions, not a present need.)

Capability flag. A provider entry should carry a tools level (native / text-fallback / none), set when the provider is added and ideally confirmed at ai config test time. The tool-use loop reads it to choose the native path or the fallback, so a use-type pointed at a tool-capable provider gets the clean path and one pointed at a plain chat model still works.

Doc vs. code note. The catalogue above is the shipping provider table (sys_ai.c). If the provider set changes — adding or removing an endpoint, or a new API style — update this table to match; the canonical source is the code, not this document.

The tool-use loop

The loop that ties it together is short, lives in Lua, and runs inside the ai flow:

tools = gather from connected MCP servers        -- overrides.tools
reply, err, calls = ai.ask(history, overrides)
while calls do                                   -- model wants tools
    for each c in calls do
        result = mcp.call(conn, c.name, c.args)
        append tool result to history
    end
    reply, err, calls = ai.ask(history, overrides) -- ask again, now with results
end
-- calls == nil  ->  reply is the final answer

The same loop drives local tools (an os.arguments script exposed as a tool) and remote MCP tools — they differ only in how mcp.call vs a local dispatch runs the action.

Provider support — already covered by the style split. All three current API styles (openai, gemini, anthropic) support native tool calling, so every shipping provider can do the clean path. The style-handler design means the work is three implementations, not six, and future providers inherit a style for free. A future style without native tool calling would use the text fallback, so the feature stays as broadly compatible as possible — quality scales with the provider, but it never simply vanishes.

Later: AntOS as an MCP server

The mirror image of the above is exposing AntOS's own capabilities as an MCP server, so an external AI — Claude on a PC, an IDE agent — can inspect and drive the machine: list tasks, read a DBFS file, post a notification, check the heap. It's the same relationship the GDB server already offers a debugger, but for AI. The mapping is nearly mechanical, because an AntOS command's os.arguments spec (name, description, typed parameters) is already the same shape as an MCP tool definition — the script registry is a tool catalogue waiting to be exposed.

It's deferred deliberately, not because it's hard: a control surface deserves careful thought about what to expose and with what guardrails, and building the client first means learning the protocol from the consuming side before opening the machine to an outside driver.

Examples

Worked examples of the pieces. Everything here is implemented; the note on each is about how you'd actually invoke it, not whether it exists. The final loop is what ai <question> runs internally once tools are cached — you don't call it by hand.

The everyday workflow

For a user who just wants their AI to use tools, the whole thing is three steps:

mcp add weather https://host/mcp     -- register an MCP server (persists)
ai -tools                            -- gather its tools into the AI's cache
ai do I need a coat in Rochdale?     -- chat; tools are used automatically

Once the cache exists (built by ai -tools), every ai chat offers those tools to the model automatically — no flag needed. Manage it with:

• ai -tools — (re)build the cache from all configured servers
• ai tools list — show what's cached and which server each tool came from
• ai tools clear — turn tools off (delete the cache); chat goes back to plain

The server list itself is managed with the mcp command (mcp to list, mcp add NAME URL, mcp del NAME, mcp test NAME).

Ask a question from a script

local ai = require("ai")

-- one-shot
local reply, err = ai.askSimple("What's the Lua pattern for a trailing newline?")
if reply then print(reply) else print("error: " .. err) end

-- multi-turn: pass a message history
local reply, err = ai.ask({
    { role = "system",    content = "You are terse." },
    { role = "user",      content = "Name three sorting algorithms." },
})
print(reply or err)

ai.ask returns reply, err, tool_calls. The third value is nil here because no tools were offered — existing two-value callers are unaffected.

Route by use-type

-- Use whichever provider/model the user configured for "code"
-- (falls back to "general" if code isn't set — see Use-types).
local reply = ai.ask(
    { { role = "user", content = "Refactor this loop: ..." } },
    { use = "code" }
)

Talk to an MCP server directly (mcp.lua, standalone)

The mcp library is standalone: you can connect, list tools, and call them with no AI involved. Useful for bringing up or debugging a server.

local mcp = require("mcp")

local conn, err = mcp.connect("https://host/mcp")
assert(conn, err)

for _, t in ipairs(mcp.tools(conn)) do
    print(t.name .. " — " .. t.description)
end

-- call a tool; args is a table, result comes back as text
local result = mcp.call(conn, "get_weather", { city = "Rochdale" })
print(result)

From the console, the same thing without writing a script:

mcp add weather https://host/mcp     -- register it (persists)
mcp test weather                     -- connect and list its tools
mcp                                  -- list configured servers

Offer tools to the model by hand (low-level)

You can drive one tool-exchange turn yourself. This is what the loop below automates — shown here so the seam is concrete.

local ai  = require("ai")
local mcp = require("mcp")

local conn  = mcp.connect_named("weather")     -- a configured server
local tools = mcp.tools(conn)                   -- already shaped for overrides.tools

local history = { { role = "user", content = "Do I need a coat in Rochdale?" } }

local reply, err, calls = ai.ask(history, { tools = tools })

if calls then
    -- the model wants to run tools rather than answer yet
    for _, c in ipairs(calls) do
        print("model wants: " .. c.name .. "(" .. c.args .. ")")
        local args   = require("data").json.decode(c.args)
        local result = mcp.call(conn, c.name, args)
        print("  -> " .. result)
    end
else
    print(reply)
end

The full tool-use loop (what ai <question> runs internally)

Putting it together: offer the tools, run whatever the model calls, feed the results back, and repeat until it answers with text. This is what typing ai <question> does once tools are cached (via ai -tools) — you won't call any of this by hand; it's shown so the mechanism is legible.

local ai   = require("ai")
local mcp  = require("mcp")
local json = require("data").json

-- Gather tools from every configured server, remembering which conn owns each.
local conns, tools, owner = {}, {}, {}
for _, s in ipairs(mcp.servers()) do
    local conn = mcp.connect_named(s.name)
    if conn then
        conns[#conns + 1] = conn
        for _, t in ipairs(mcp.tools(conn)) do
            tools[#tools + 1] = t
            owner[t.name] = conn
        end
    end
end

local history = { { role = "user", content = "Do I need a coat in Rochdale?" } }

while true do
    local reply, err, calls = ai.ask(history, { tools = tools })
    if err then print("error: " .. err); break end

    if not calls then
        print(reply)          -- model answered with text: done
        break
    end

    -- Record the assistant's tool-call turn, then run each call and append
    -- its result as a role="tool" message keyed by id (see "The three seams").
    history[#history + 1] = { role = "assistant", content = reply or "", tool_calls = calls }
    for _, c in ipairs(calls) do
        local conn   = owner[c.name]
        local result = conn and mcp.call(conn, c.name, json.decode(c.args)) or "no such tool"
        history[#history + 1] = {
            role = "tool", tool_call_id = c.id, content = result,
        }
    end
end

The loop is provider-agnostic (the style handlers translate the wire format) and source-agnostic (a local os.arguments script exposed as a tool would run through the same calls handling — only the dispatch differs from mcp.call).

Summary

• Today: ai.ask / ai.askSimple over configured providers, split by use-type, surfaced as the ? helper and the ai chat client — all Luau on a small C core.
• Built — tool use via MCP: folded into ai.ask through three seams — overrides.tools, a tool_calls return, and an mcp helper — with a Lua loop (in ai.lua) that treats local and remote tools alike. Managed with the mcp command and ai -tools.
• Provider-agnostic: one internal tool vocabulary translated by the existing API-style handlers — ask_openai (Groq/Ollama/OpenAI/Kimi), ask_gemini, ask_anthropic — so tool use is three implementations covering all six shipping providers, with a text fallback reserved for any future style that lacks native tool calling.
• Later (server): expose AntOS's os.arguments scripts as MCP tools so an external AI can drive the machine, like the GDB server but for AI.