--- name: signal-llm-gateway description: How to call LLMs through the Signal Labs LLM gateway (https://llm-gateway.dev.sine.signallabs.ai). Use whenever code, an agent or a harness needs a model call at Signal Labs - OpenAI-compatible and Anthropic-compatible endpoints, one API key, per-person budgets. --- # Signal Labs LLM gateway One endpoint for every model Signal Labs runs. It speaks the OpenAI API and the Anthropic Messages API, so any OpenAI or Anthropic SDK, LangChain, LiteLLM, Claude Code or a custom harness works by changing the base URL and the key. Built on LiteLLM; models are served from AWS Bedrock (US regions). ## Get a key 1. A human opens https://llm-gateway.dev.sine.signallabs.ai and signs in with their Microsoft work account (signallabs.ai only). 2. They click **Create key**. The key starts with `sk-` and is shown once. 3. Store it in the environment as `SIGNAL_LLM_KEY`. Never hard-code it or commit it. If you are an agent and `SIGNAL_LLM_KEY` is not set, stop and ask the user to create a key at https://llm-gateway.dev.sine.signallabs.ai. Do not try to create keys yourself: key creation requires the user's browser sign-in. ### Personal key or app key? A personal key belongs to one person: it spends their budget, and it stops working if they are suspended or leave. Use one for your own work, scripts and agents you run yourself. An app or service that runs in production, or that other people depend on, needs an **app key** instead. An app key belongs to a group, not a person: it has its own budget (which also counts toward the group's budget), its spend shows under the app's name, and it keeps working whoever joins or leaves. Only admins create app keys. To get one, ask an admin with: the app's name (e.g. `signalos-qna-prod`), the group that owns it, a contact person, a monthly budget, and optionally which models it may use and its rate limits. To rotate an app key, ask for a second key, switch the app over, then ask for the old one to be revoked. ## Endpoints | What | Base URL | Auth header | |---|---|---| | OpenAI-compatible | `https://llm-gateway.dev.sine.signallabs.ai/v1` | `Authorization: Bearer $SIGNAL_LLM_KEY` | | Anthropic Messages | `https://llm-gateway.dev.sine.signallabs.ai` (SDK appends `/v1/messages`) | `x-api-key: $SIGNAL_LLM_KEY` or `Authorization: Bearer ...` | | List models | `GET https://llm-gateway.dev.sine.signallabs.ai/v1/models` | Bearer | | This document | `GET https://llm-gateway.dev.sine.signallabs.ai/skill` | none | Supported OpenAI routes: `/v1/chat/completions` (streaming, tools, JSON mode, images), `/v1/models`. Supported Anthropic route: `/v1/messages` (streaming, tools, thinking). ## Models Use the name in the left column as `model`. Prices are USD per million tokens and are what your budget is charged. | model | input | output | cache read | cache write | capabilities | |---|---:|---:|---:|---:|---| | `claude-opus-5-5` | $4.40 | $22.00 | $0.22 | $5.50 | Frontier reasoning · 1M context · agentic coding and computer use | | `claude-sonnet-5-5` | $2.20 | $11.00 | $0.11 | $2.75 | 1M context · text and image in · 5 thinking levels · near-Opus coding | | `claude-sonnet-4-6` | $3.30 | $16.50 | $0.33 | $4.125 | 1M context · text and image in · previous-generation Sonnet | | `claude-haiku-5-5` | $0.11 | $0.55 | $0.011 | $0.1375 | 1M context · adjustable thinking · fast, cheap sub-agent work | | `gpt-6.1-sol` | $2.20 | $11.00 | $0.11 | $2.75 | OpenAI frontier · long-running agents · large-codebase refactors | | `gpt-6.1-terra` | $1.10 | $5.50 | $0.055 | $1.375 | Visual reasoning · computer use · writing · adjustable reasoning | | `gpt-6-luna` | $0.11 | $0.55 | $0.011 | $0.1375 | ~1M context · text and image in · reasoning none to max · classification | | `kimi-k3` | $3.30 | $16.50 | $0.33 | $4.125 | Open weights · 1M context · text, image and video in · always max reasoning | | `deepseek-v3.2` | $0.62 | $1.85 | – | – | Open weights · sparse attention for long context · tool calls with thinking | | `qwen3-coder-next` | $0.50 | $1.20 | – | – | Open weights · 80B MoE, 3B active · 256K context · coding agents, FIM | | `qwen3-next-80b` | $0.14 | $1.20 | – | – | Open weights · 80B MoE, 3B active · 262K context · text only | | `gpt-oss-120b` | $0.15 | $0.60 | – | – | Open weights · 131K context · reasoning low/medium/high · tools, text only | | `nova-2-lite` | $0.33 | $2.75 | $0.0825 | $0.00 | 1M context · text, image, video and documents in · 3 thinking levels | Defaults if you have no reason to choose: `auto` (below) lets the gateway pick per request. Otherwise `claude-sonnet-5-5` for agents and coding, `claude-haiku-5-5` or `gpt-6-luna` for high-volume cheap calls, `claude-opus-5-5` or `gpt-6.1-sol` only for the hardest steps. A person can read a research card for each model (strengths and weaknesses from Hacker News discussions, each linked to its source) on the Models page at https://llm-gateway.dev.sine.signallabs.ai/catalog. ## Let the gateway pick: `auto`, `auto-perf`, `auto-cost` Send one of these as `model` instead of a model name. The gateway reads the query, works out its task type (maths, coding, science, factual recall, agent task, ...) and how hard it is, and sends it to the model that does that kind of query best for the price. Same endpoints, same SDKs, nothing else changes. | model | what it optimises | typically picks | |---|---|---| | `auto` | balance of quality and cost | `gpt-6-luna` for easy and routine queries, `gpt-oss-120b` for cheap reasoning, `gpt-6.1-sol` for hard ones | | `auto-cost` | lowest cost that is still likely to be right | mostly `gpt-6-luna` and `gpt-oss-120b`; strong models only for the hardest queries | | `auto-perf` | quality first | mostly `gpt-6.1-sol` and `gpt-6.1-terra`; `gpt-6-luna` when an easy query does not need more | - It chooses among `gpt-6-luna`, `gpt-oss-120b`, `claude-haiku-5-5`, `gpt-6.1-terra`, `gpt-6.1-sol` and `claude-sonnet-5-5`, and you are billed at the price of the model it picks. - The response headers say what happened: `x-auto-router-model` (the model that answered), `x-auto-router-preset` (the name you sent), `x-auto-router-reason` (`routed`, `cached`, or `fallback-...`), `x-auto-router-domain` and `x-auto-router-difficulty` (0 = trivial to 4 = frontier). The body's `model` field echoes the name you sent. - It adds about 150 ms to a request (one Jev call and one embedding, in parallel). A repeat of the same query adds nothing. In a multi-turn agent loop it routes on the latest human instruction, not on tool results, so every turn of one task goes to the same model and keeps that model's prompt cache. - If its signals take longer than 0.5 s or fail, the request still goes through, to `gpt-6-luna` (`auto`, `auto-cost`) or `gpt-6.1-sol` (`auto-perf`). - A request with no text from the user at all (for example only an image) goes to `gpt-6.1-sol`, whichever of the three you sent. It has no `x-auto-router-*` headers; `x-litellm-complexity-router-cause: default_fallback` says why. - Requests with images never go to the text-only `gpt-oss-120b`; very long inputs skip it too. - Its routing table is learned from benchmark results (about 3,600 scored queries: maths, science, coding, factual questions and agent tasks), with each benchmark model's role mapped to one of ours. It has not yet learned from our own traffic. Where you have measured which model works for your task, name it. - Keep Claude Code on a named Claude model (`claude-sonnet-5-5`). `auto` may answer a turn with a GPT model, and Claude Code relies on Claude-specific behaviour. ```python raw = client.chat.completions.with_raw_response.create( model="auto", messages=[{"role": "user", "content": "Hello"}]) print(raw.headers["x-auto-router-model"]) # the model that answered r = raw.parse() # the usual ChatCompletion ``` ### Prompt caching and long prompts - **Claude models** cache only when you ask: add `"cache_control": {"type": "ephemeral"}` to the last block of the stable prefix (system prompt, tool definitions, long documents). Claude Code does this for you. A cache write costs 1.25x input; a cache read costs 0.1x. - **GPT-6.x models** cache automatically. The first call with a long prompt is billed as a cache write (1.25x input); repeats of the same prefix are billed at the cache-read price. - `deepseek-v3.2`, `qwen3-coder-next`, `qwen3-next-80b` and `gpt-oss-120b` have no prompt caching here. - **Long-context price tiers** reprice the whole request once the prompt crosses a threshold: `claude-haiku-5-5` above 100k input tokens costs 5x; the GPT-6.x models above 272k cost 2x input and cache, 1.5x output. `claude-sonnet-5-5` and `claude-opus-5-5` have no long-context tier. ## Examples ### curl ```bash curl https://llm-gateway.dev.sine.signallabs.ai/v1/chat/completions \ -H "Authorization: Bearer $SIGNAL_LLM_KEY" \ -H "Content-Type: application/json" \ -d '{"model": "claude-sonnet-5-5", "messages": [{"role": "user", "content": "Hello"}]}' ``` ### Python, OpenAI SDK ```python import os from openai import OpenAI client = OpenAI(base_url="https://llm-gateway.dev.sine.signallabs.ai/v1", api_key=os.environ["SIGNAL_LLM_KEY"]) r = client.chat.completions.create( model="claude-sonnet-5-5", messages=[{"role": "user", "content": "Hello"}], ) print(r.choices[0].message.content) ``` Streaming: pass `stream=True` and iterate. Tool calling: pass `tools=[...]` in the OpenAI format; the gateway translates it for every model. ### Python, Anthropic SDK ```python import os import anthropic client = anthropic.Anthropic(base_url="https://llm-gateway.dev.sine.signallabs.ai", api_key=os.environ["SIGNAL_LLM_KEY"]) r = client.messages.create( model="claude-sonnet-5-5", max_tokens=1024, messages=[{"role": "user", "content": "Hello"}], ) print(r.content[0].text) ``` ### TypeScript, OpenAI SDK ```ts import OpenAI from "openai"; const client = new OpenAI({ baseURL: "https://llm-gateway.dev.sine.signallabs.ai/v1", apiKey: process.env.SIGNAL_LLM_KEY }); const r = await client.chat.completions.create({ model: "claude-sonnet-5-5", messages: [{ role: "user", content: "Hello" }], }); ``` ### Claude Code ```bash export ANTHROPIC_BASE_URL="https://llm-gateway.dev.sine.signallabs.ai" export ANTHROPIC_AUTH_TOKEN="$SIGNAL_LLM_KEY" export ANTHROPIC_MODEL="claude-sonnet-5-5" export ANTHROPIC_DEFAULT_HAIKU_MODEL="claude-haiku-5-5" claude ``` ### LangChain ```python from langchain_openai import ChatOpenAI llm = ChatOpenAI(base_url="https://llm-gateway.dev.sine.signallabs.ai/v1", api_key=os.environ["SIGNAL_LLM_KEY"], model="claude-sonnet-5-5") ``` ## Building an agent harness on the gateway - Treat the gateway as a plain OpenAI endpoint. Keep the base URL and key in config, not in code. - Pick the model per step, or send `auto` / `auto-cost` and let the gateway pick: a cheap model (`claude-haiku-5-5`, `gpt-6-luna`, `qwen3-next-80b`, `gpt-oss-120b`) for classification, extraction and sub-agents; `claude-sonnet-5-5` for the main loop; escalate to `claude-opus-5-5` only when the cheaper model fails. - Keep one model for a whole multi-turn session where you can. Switching models mid-conversation throws away the provider's prompt cache and costs more. - Use one key per project or agent, so spend is easy to attribute. A person may have up to 10 keys. A harness that runs as a production service uses an app key (see "Get a key"), not someone's personal key. - Set `max_tokens` on every call. Reasoning models spend tokens on thinking before the answer: `kimi-k3` always reasons at max, and the GPT-6.x models and `gpt-oss-120b` reason by default. Give them room or you get empty content. - Set a client timeout of at least 300 s for long generations, and retry 429 and 5xx with backoff. - Cap your own concurrency below your limit (see below). Parallel sub-agents share it with everything else on all your keys. ## Budgets, rate limits and errors Every person has a monthly budget shared by all their keys (default $25 unless an admin raised it). Spend so far is on https://llm-gateway.dev.sine.signallabs.ai. Every person also has rate limits, shared by all their keys (defaults: 1,000 requests per minute, 10,000,000 tokens per minute, 200 requests running at once). Your own limits are on https://llm-gateway.dev.sine.signallabs.ai. A request to `auto`, `auto-perf` or `auto-cost` counts once. Going over a limit returns **429** with a message naming the limit and a `retry-after` header (seconds). Handle it like this: - Wait `retry-after` seconds (or 2 s if it is missing), then retry the same request. Back off exponentially, with jitter, if it happens again; give up after about 5 tries. - Requests and tokens per minute reset each minute. "Concurrency limit exceeded" means too many requests are running at once: retry as soon as one of yours finishes, and lower your parallelism. - The OpenAI and Anthropic SDKs already retry 429 (`max_retries`, default 2). Raise it for batch jobs. - If normal work hits the limit again and again, ask an admin to raise yours. | Status | Meaning | What to do | |---|---|---| | 401 | Missing, wrong or revoked key | Check `SIGNAL_LLM_KEY`; make a new key at https://llm-gateway.dev.sine.signallabs.ai | | 422, `"type": "budget_exceeded"` | Monthly budget used up (for an app key: the key's or its group's) | Ask an admin, or wait for the reset date shown in the portal | | 400 "Invalid model name" | Model not served here | `GET /v1/models` for the list | | 429 "Rate limit exceeded" or "Concurrency limit exceeded" | Your per-person limit (message says which) | Wait `retry-after`, then retry with backoff; see above | | 429 from the provider | Bedrock is busy for everyone on that model | Retry with exponential backoff, or try another model | | 5xx | Upstream provider error | Retry; if it persists try another model | ## Rules - Do not send client data (Optum, Norfolk Southern, etc.) unless the engagement allows AWS Bedrock in US regions. - The gateway logs who called which model, tokens and cost. It does not store prompt or response text. For `auto`, `auto-perf` and `auto-cost` it also logs how it routed: the task type and difficulty it judged, the query's embedding vector, and a hash and the length of the query text (not the text itself). - Never print, log or commit keys.