What's TZRO

A zero-dependency, local-first agentic execution engine. It compiles natural language prompts into topologically-sorted DAG (Directed Acyclic Graph) workflows, executes them durably with checkpointing, and provides relational memory, knowledge graph, and micro-skill extraction.

Intelligent Task Offloading

Today's agentic coding workflows are fragile. A cloud-only "generate-and-run" chat loop means every directory listing, every file parse, every grep runs through a frontier API — burning tokens, hitting daily rate limits, and freezing your workflow mid-task. This is token-maxing: feeding a $15/MTok model with work a local 3B can handle in milliseconds.

tzro implements Inference Arbitrage — the principle that the right model for the job is rarely the most expensive one. By routing execution-level work and even code generation to a grammar-locked local engine (via tzro_code), you preserve your cloud budget for what actually requires frontier reasoning: architectural planning, complex judgment, and interactive user dialogue.

The Cloud Plans, Local Executes

The division of labor is explicit and non-negotiable:

The 5-Minute Plug-and-Play Quickstart

Zero habit change. tzro plugs directly into your existing client tools — Claude Code, Cursor, Windsurf, GitHub Copilot, OpenCode, or Antigravity IDE — as an MCP server. You keep using your tools exactly as before; the sidecar silently intercepts execution-level work and routes it locally.

curl -fsSL https://get.tzro.ai | sh

  ⚙  AI Editor Configuration

  ✔ Claude Code
  ✔ Cursor
  ○ Windsurf (not detected)
  ○ GitHub Copilot (not detected)
  ○ OpenCode (not detected)
  ✔ Gemini / Antigravity IDE

  Select interfaces to configure:
  Press Enter to accept detected defaults

{
  "mcpServers": {
    "tzro": {
      "command": "~/.tzro/bin/tzro-mcp",
      "args": [],
      "env": {
        "TZRO_DIR": "/path/to/your/project"
      }
    }
  }
}

# From Claude Code, Cursor, or any MCP-compatible client:
"Use tzro to build markdown documentation for this entire directory."

# Or directly via CLI:
tzro chat "Explore this codebase and explain its architecture."

Local Security & Sandbox Transparency

We assume you're skeptical of running a local executable that intercepts AI tool calls. Good. Here's the complete security architecture — no hand-waving, no "trust us" appeals. Least-Privilege Security-by-Design at every layer.

Data Sovereignty

100% of code parsing, file edits, and tool manipulations remain local behind the loopback interface. Sensitive corporate data never leaves the premises.

Containerized Execution

When tasks require code execution beyond simple file operations, TZRO dynamically orchestrates short-lived, task-scoped container sandboxes leveraging local Podman socket routes.

• Ephemeral containers — spun up per-task, destroyed on completion. No persistent state leakage between executions.
• Rootless Podman — no Docker daemon, no root access. Runs entirely in userspace via podman --rootless.
• Resource-capped — each container is CPU/memory-limited to prevent runaway processes from impacting your development environment.

# How tzro spawns a sandboxed execution container:
podman run --rm --read-only \
  --network=none \
  --memory=512m --cpus=1.0 \
  -v /project:/workspace:ro \
  tzro-sandbox:latest \
  sh -c "$TASK_COMMAND"

Grammar Gating (GBNF)

For structured output calls — tool dispatch, classification, parameter extraction, and validator passes — the local LLM is constrained by logit-level GBNF grammar rules that mathematically prevent arbitrary command injection. These are the security-critical call sites where output format matters.

On constrained calls, the model's output is restricted strictly to predefined JSON schemas. It cannot produce output outside the grammar — this is enforced at the logit sampling level, not by post-hoc filtering. If the grammar says the output must be a JSON object with keys tool, args, and reasoning, then that is the only structure the model can physically produce. Free-form generation calls (synthesis, reasoning text) are not grammar-constrained.

# Example: Tool dispatch grammar constraint
root   ::= "{" ws "\"tool\":" ws tool-name "," ws
             "\"args\":" ws json-object "," ws
             "\"reasoning\":" ws string ws "}"
tool-name ::= "\"read_file\"" | "\"list_dir\"" | "\"search_files\"" | "\"web_search\""

Network Isolation

Mounted target repositories default to read-only parameters, and network interfaces inside execution containers are entirely decoupled (network=none) to block supply-chain leaks.

• Read-only mounts — project directories are mounted with :ro flag by default. Write access requires explicit opt-in via task configuration.
• Network decoupled — containers run with --network=none. No DNS resolution, no HTTP requests, no exfiltration vector.
• No package installation — containers have no access to package registries. No npm install, no pip install, no supply-chain attacks.

Local Hardware Optimization

Running a local LLM sidecar will not freeze your laptop or disrupt your multitasking workflow. Here's exactly how we ensure that.

Speculative Decoding

tzrod uses n-gram speculative decoding forward-projections to speed up local 3B–4B generation velocities by 1.8× on standard CPUs.

The technique works by maintaining a rolling n-gram cache of recently generated token sequences. When the model begins generating a new sequence, the speculative decoder proposes multiple candidate continuations in parallel based on n-gram frequency statistics. The main model then verifies these candidates in a single forward pass — accepting correct predictions for free and rejecting mismatches with zero wasted compute.

Warm KV Prefix-Sharing

System prompts and tool schemas are pin-locked into KV Cache slot 0, eliminating redundant compilation on every request. The result: compilation times consistently under <100ms.

When tzrod starts, it pre-compiles the system prompt, all registered tool schemas, and the GBNF grammar into a persistent KV Cache prefix. This prefix is shared across all subsequent inference calls without re-processing. Only the user-specific context (task prompt, file contents, intermediate outputs) needs fresh computation.

┌─────────────────────────────────────────────────────────┐
│  KV Cache Slot 0 (pin-locked, persistent)               │
│  ├── System Prompt          ~420 tokens                  │
│  ├── Tool Schemas (26 tools) ~1,800 tokens               │
│  └── GBNF Grammar           ~200 tokens                  │
├─────────────────────────────────────────────────────────┤
│  KV Cache Slot 1 (dynamic, per-request)                 │
│  ├── Task Prompt                                        │
│  ├── File Contents / Context                            │
│  └── Intermediate Outputs                               │
└─────────────────────────────────────────────────────────┘

Zero-Latency Attention Preemption

The AttentionScheduler instantly pauses background processing tasks and dumps their state to disk ($slot_0.bin) the exact millisecond a user triggers a foreground prompt.

This ensures an interactive foreground Time-to-First-Token (TTFT) of ≤ 450ms — regardless of how many background DAG nodes are currently executing. Your interactive coding session is never degraded by background work.

Debugging & Visibility

Cooperative execution trades wall-clock latency for extreme cost savings — pennies vs. multi-dollar cloud bills. We don't pretend this is free. What we do provide is total visibility into what the background engine is doing, so you always know it isn't hanging.

Fullscreen TUI Dashboard

Launch the interactive terminal dashboard to visualize parallel Kahn Topological Sort execution levels, node states, and real-time progress:

tzro tui

The Bubble Tea TUI shows:

• DAG visualization — nodes rendered by topological level with dependency edges
• Node status — pending, running, completed, failed, or skipped (with color coding)
• Live output streaming — real-time stdout from actively executing nodes
• Resource metrics — CPU, memory, and token generation throughput

Token Tracking & Benchmark Comparison

Every task execution carries a TokenTracker embedded in its Go context, separately accumulating local and cloud token usage (prompt, completion, duration, tokens/sec) across all inference calls within that task. This infrastructure powers the offline benchmark comparison suite.

# Run the comparison suite against a specific task category
tzro compare --category docgen --output ./results/

# Example per-task output:
  ✓ 3500 cloud tokens, 8200 local tokens, $0.0525, 14200ms, 12 tool calls

# Summary output:
Cloud token savings (ReAct → Cooperative): 72%
Cost savings: $0.5250 → $0.1470 (saved $0.3780)

The comparison harness runs each benchmark task under multiple conditions and attributes savings into three independently measured buckets:

Logs & Compliance Audit

For debugging execution issues or generating compliance documentation:

# Live engine logs — see every tool dispatch, node transition, and error
tail -f ~/.tzro/logs/engine.log

# Generate a compliance audit report (HTML) for security review
tzro audit --compliance-html

# Inspect a specific task execution trace
tzro task status <taskId> --offline

The compliance audit generates a standalone HTML document covering:

• Data flow attestation — proof that no data left loopback during the session
• Tool execution log — every tool invoked, arguments passed, and outputs returned
• Grammar constraint verification — confirmation that all LLM outputs matched GBNF schemas
• Container lifecycle audit — creation, execution, and teardown timestamps for all sandboxes

v1.0.0 — Recent Engine Advances

The v1.0.0 release introduces major architectural improvements that significantly increase local execution quality, reduce token waste, eliminate hallucinations, and enable concurrent inference workloads.

Dual-Sidecar Inference Architecture

Previous versions ran a single llama-server process for all local inference. This forced a tradeoff: use a fast small model (great for routing, bad at code) or a quality model (great at code, too slow for 30-step Probe chains). The hot-swap mechanism blocked all inference during model switches (~1-1.5s downtime per swap).

v0.9.0 replaces this with two independent llama-server processes running concurrently:

┌──────────────────────────────────────────────────────────┐
│                     tzrod daemon                         │
│                                                          │
│  GlobalRouterModel              GlobalWorkerModel       │
│  ┌──────────────────┐          ┌──────────────────┐     │
│  │ Model: Gallium-350M│         │ Model: Agents-A1  │     │
│  │ Port:  (dynamic)   │          │ Port:  (dynamic)  │     │
│  │ Ctx:   2,048 tok   │          │ Ctx:   16,384 tok │     │
│  │ Slots: 1           │          │ Slots: 1          │     │
│  └──────────────────┘          └──────────────────┘     │
│       ↑ fast, routing               ↑ quality, gen    │
└──────────────────────────────────────────────────────────┘

Inference routing is explicit at the call site. The inference package exposes CallRouter() and CallWorker() functions:

{
  "ggufModelPath":    "Agents-A1-4B-Q4_K_M.gguf",
  "routerModelPath":  "Gallium-350M.Q8_0.gguf"
}

Structured Content-Aware Compaction

Previous compaction pipelines treated all content identically — dumping code, tool outputs, and reasoning text into a single LLM "compress these steps" prompt. This caused the model to hallucinate type names and lose function signatures during compression.

The Structured Compactor (internal/compactor/) classifies content into typed segments and applies the appropriate compaction strategy for each:

// Before compaction: 1207 lines of raw Go source
// After Code Skeleton extraction:

package inference

type LocalModelManager struct { ... }

func (m *LocalModelManager) Start(ctx context.Context) error {
    // [body: 89 lines, calls: m.ensureBinary(), m.spawnProcess(), m.healthCheck()]
}

func (m *LocalModelManager) CallRouter(ctx context.Context, ...) (*InferenceResult, error) {
    // [body: 42 lines, calls: m.routerClient.Chat(), m.fallbackToWorker()]
}

func (m *LocalModelManager) CallWorker(ctx context.Context, ...) (*InferenceResult, error) {
    // [body: 38 lines, calls: m.workerClient.Chat()]
}

This replaces the legacy monolithic compaction prompt and serves all compaction call sites: probe thought chains, recall refined context, accumulated context, and synthesis truncation.

Multi-Branch MCTS Evaluation

Edge Thoughts are compact reasoning states generated when the executor traverses a DAG edge whose target node has a non-zero Activation Threshold. In v0.9.0, Edge Thought evaluation gains a second mode: multi-branch evaluation, inspired by Monte Carlo Tree Search.

The Speculation Fence gates speculative candidate evaluation by Tool Proactivity Level:

Two-Tier Context Budget

Accumulated Context — the structured collection of completed upstream node outputs injected into each DAG node's prompt — previously had no budget controls. Large Probe outputs or verbose tool responses would consume the entire context window, drowning downstream nodes in noise.

v0.9.0 introduces tiered per-node output budgets with a dynamic ceiling:

Recall Nodes & Map-Reduce Synthesis

When a Probe Node completes a 15-30 step exploration, its accumulated thought chain can contain 20K+ tokens of interleaved tool outputs, reasoning, and code. Feeding this raw history directly into a synthesis node caused two failure modes:

• The Synthesis Cliff — one-shot synthesis on massive input produced hallucinated types and lost function signatures
• Prefill Latency — processing 20K+ tokens of raw history in a single inference call caused 15-30 second delays

Recall Nodes solve this. The Kahn Compiler automatically injects a Recall Node after every Probe Node, separating discovery from synthesis:

┌─────────────────────────────────────────────────────────┐
│  Probe Node (explores)                                   │
│  └── 15-30 steps of read_file, list_dir, search_files   │
│  └── Output: 20K+ tokens of raw thought chain              │
├─────────────────────────────────────────────────────────┤
│  Recall Node (synthesizes) — auto-injected               │
│                                                          │
│  Step 1: MAP                                             │
│    Classify each chunk as "Signal" or "Sludge"            │
│                                                          │
│  Step 2: REDUCE                                          │
│    Extract key facts from Signal chunks only              │
│                                                          │
│  Step 3: SYNTHESIZE                                      │
│    Produce aligned, goal-directed output                  │
├─────────────────────────────────────────────────────────┤
│  Downstream Nodes — receive clean, ~2K token summary     │
└─────────────────────────────────────────────────────────┘

AST-Based Symbol Extraction & Anchor Verification

Docgen benchmarks revealed that the local 4B model hallucinated 67% of type names in documentation outputs — the LLM judge scored these 4.25/5.0 while manual audit scored 2.65/5.0, a 38% quality inflation rate. The root cause: the model lacked sufficient context to identify real public symbols in the source files it had read.

The Symbol Extractor (internal/symbols/) solves this with deterministic, AST-verified symbol identification. Every file read during a Probe Node's Thought Chain is parsed using pure-Go tree-sitter (no CGO required), emitting structured tuples:

type Symbol struct {
    Name      string     // e.g. "CompactContent"
    Kind      SymbolKind // func | type | interface | method | class | ...
    Signature string     // e.g. "func CompactContent(content string, ...) string"
    File      string     // source file path
    Line      int        // declaration line number
    Exported  bool       // visibility flag
}

These tuples are persisted in a SQLite Symbol Index table per Probe execution. The Probe's context window carries only a count ("N symbols extracted"), not the full index — keeping token usage minimal during exploration.

After the Recall Node produces its synthesis, the Symbol Anchor Check diffs referenced identifiers against the index:

Directory Preloading

Probe Nodes explore codebases through iterative Thought Chain steps — the local model decides which file to read next based on previous observations. But a 4B model frequently makes poor navigation decisions: it fixates on a single directory, misses key files, or wastes steps on irrelevant paths. The result: incomplete exploration and missing context in synthesis.

Directory Preloading (internal/executor/probe_preload.go) eliminates this "routing problem" by injecting source context before the Thought Chain begins. The preloader walks target directories and concatenates their readable files into the prompt:

Repo Mapping & Direct Synthesis

Not every task needs a 15-30 step Probe exploration. When a task's target files are known upfront (e.g., "document the internal/executor/ package"), the full Thought Chain is overhead: the model navigates to the directory, reads each file sequentially, then synthesizes — burning steps on pure routing rather than reasoning.

Repo Mapping (internal/repomap/) generates a deterministic, AST-based codebase architecture map: every Go source file's exported structs, interfaces, and functions extracted into a structured markdown document. Combined with Direct Synthesis, the Probe can satisfy the task with a single-shot inference against this pre-compiled context.

Standard Probe (Thought Chain):
  Step 1  → list_dir("/internal/executor/")         ~2s
  Step 2  → read_file("executor.go")                ~3s
  Step 3  → read_file("probe.go")                   ~3s
  ...     → 15-30 steps of exploration               ~45-90s total
  Recall  → Map-Reduce synthesis                    ~5s

Direct Synthesis:
  RepoMap → AST-extract all symbols deterministically  ~200ms
  Synth   → Single-shot inference against repo map     ~3s
  Total: ~3s (vs. 50-95s)