OpenAI-compatible and local models

For air-gapped vaults, privacy-sensitive content, or just experimenting, Lettuce supports any OpenAI-compatible endpoint — including local model runners.

Supported tooling

Configuration

Providers are configured through environment variables (or the GUI under Settings → AI providers), not a config file. To point Lettuce at a local OpenAI-compatible runner, set:

export LETTUCE_LLM_PROVIDER=openAICompatible
export LETTUCE_LLM_BASE_URL="http://localhost:11434/v1"
export LETTUCE_ENHANCE_MODEL="qwen2.5-coder:7b"
# Ollama ignores the key, but OpenAI-compatible clients expect one:
export OPENAI_API_KEY="ollama"

Embeddings can point at the same runner (or a different one) via LETTUCE_EMBED_BASE_URL and LETTUCE_EMBED_MODEL.

For cloud-hosted aggregators, set the base URL and the real key:

export LETTUCE_LLM_PROVIDER=openAICompatible
export LETTUCE_LLM_BASE_URL="https://openrouter.ai/api/v1"
export LETTUCE_ENHANCE_MODEL="anthropic/claude-opus-4-7"
export OPENAI_API_KEY="sk-or-..."

Tool use with local models

Lettuce passes MCP tools as OpenAI function-call schema. Whether the local model uses them well depends on the model. Generally:

• Good: Llama 3.3 70B, Qwen 2.5 Coder 32B+, Mixtral 8x22B.

• OK: Llama 3.1 70B, Qwen 2.5 14B.

• Poor: smaller than 7B, ancient base models without function-calling fine-tunes.

For the smaller / older case, agents may hallucinate tool calls or ignore the schema. Restrict what the connected agent can do by issuing it a read-only scoped key — lettuce keys add --label local-triage --scope read — so even a confused model can only read and search, never write.

Performance

A local 7B is fine for tag suggestions and triage; not great for nuanced rewrites.

Privacy properties

Local models never leave your machine, and the MCP server listens on localhost only (lettuce serve, default port 51234). Combined: an air-gapped vault with a local model and a local agent is fully offline.

Practical config

The provider is a process-wide setting, so the runner you point LETTUCE_LLM_PROVIDER at is the one that handles every directive — including ones spliced by a schedule. A daily local triage looks like this:

# .lettuce/schedules.yml
- daily: "07:00"
  target: "Inbox/index.md"
  prompt: |
    Triage everything in Inbox/...

At 07:00 the SchedulePoller (running under lettuce serve, every 60s) splices > [!ai] Triage everything in Inbox/... into Inbox/index.md. Whatever MCP agent is connected — here, your local Ollama-backed agent — picks the directive up via pending_directives and does the work. Set your env vars before launching lettuce serve so the directive is handled by the local model.

Mixing

There is only one active provider at a time, so you don’t mix providers per-schedule. Instead, point Lettuce at the model that fits how you run it: a local 7B for cheap, frequent triage and tag suggestions, or Anthropic for nuanced weekly reviews. A schedule can also run a maintenance pass directly instead of asking an agent:

# .lettuce/schedules.yml
- weekly: "Fri 17:00"
  target: "Reviews/index.md"
  action: "lint-semantic"

The action: form runs the built-in maintenance pass (using whatever provider is configured) with no agent required.

Related

• AI providers/01 Anthropic Claude

• Vault automation/02 Schedules