Lowest-cost routing · Smartflow + Halo

Most turns don't need
the expensive model.

Smartflow and Halo look at the prompt before it hits a provider, then send easy work to a cheaper lane and keep the hard stuff on the model you asked for. If the cheap lane fails or refuses, one retry goes back up. No extra model sits in the middle judging the request.

On for hosted Smartflow On for Halo Route auto · local · frontier One quality retry

What you get

Cheap

Greetings, lookups, short asks

Frontier

Errors, analysis, long work

1 hop

Retry if the cheap lane fails

0 LLMs

No judge model in the path

How it decides

Score the prompt. Don't call another model.

same scorer in both products

Before the provider call, the gateway already knows what kind of turn this is: greeting vs traceback, a definition vs a comparison, a short ask vs a long one. Lowest-cost routing uses those signals. One hint isn't enough. Two pointing the same way is. If it's a toss-up, the cheap lane wins. A tool error or an explicit failure skips cheap and goes straight to the capable model.

Three modes

auto — score the prompt (this is the live default)
local — always the cheap lane
frontier — always the model you asked for

A request header beats a policy. A policy beats the gateway default. Set off if you want the old behavior for a key, a team, or a single call.

Quality retry

If the cheap lane returns a 5xx, an empty body, or a refusal, Smartflow retries once on the original model. Halo Route already has failover for transport / 502 / 503 / 504 / 529. You don't eat a failed cheap call as the final answer.

agent / app │ ▼ Smartflow or Halo score prompt → cheap or frontier │ ├─ easy → same-provider mini / key local pool └─ hard → the model on the request └─ cheap failed? one retry on frontier

Smartflow (gateway)

Hosted: already on

cluster1.langsmart.app

The hosted gateway scores every non-streamed chat turn. Easy work stays on the same provider's cheap model (mini on OpenAI, haiku on Anthropic) or hops to the virtual key's local pool if you pinned one. Hard work stays on the model in the request. Streamed calls skip scoring so playground SSE still works. Pin a call to frontier with a header if you don't want the downgrade:

curl https://cluster1.langsmart.app/v1/chat/completions \
  -H "Authorization: Bearer $SMARTFLOW_KEY" \
  -H "X-Smartflow-LCR: frontier" \
  -d '{"model":"gpt-4o","messages":[{"role":"user","content":"hi"}]}'

Your own cluster or on-prem box

Same binary. Set the env on the proxy, point the cheap lane at whatever you actually run — OpenAI mini, a local Ollama, vLLM, a GPU box. Restart the proxy.

SMARTFLOW_LCR=auto
SMARTFLOW_LCR_THRESHOLD=0.5
SMARTFLOW_LCR_EFFICIENT_PROVIDER=openai    # or ollama / vllm / openrouter
SMARTFLOW_LCR_EFFICIENT_MODEL=gpt-4o-mini
# leave CAPABLE unset to keep the caller's model for hard turns

Kubernetes:

kubectl -n smartflow set env deployment/smartflow-proxy -c proxy \
  SMARTFLOW_LCR=auto \
  SMARTFLOW_LCR_EFFICIENT_PROVIDER=openai \
  SMARTFLOW_LCR_EFFICIENT_MODEL=gpt-4o-mini

Docker / compose: add the same keys to the proxy service environment. OpenRouter-edition installs that already have a local GPU should set SMARTFLOW_LCR_EFFICIENT_PROVIDER=ollama (or vllm) instead of OpenAI.

Per team, per key, per request

A policy attachment can set the mode and the two models. Same scoping you already use for model rewrite (directory group, virtual key, team). The request header still wins, so a debugger can force frontier without changing policy.

{
  "lcr_mode": "auto",
  "effort_threshold": 0.5,
  "efficient_model": "gpt-4o-mini",
  "capable_model": "gpt-4o"
}

Headers: X-Smartflow-LCR: auto|local|frontier. Open the trace on that turn — Request → Routing shows lcr:efficient, lcr:capable, or lcr:skipped stream. FinOps has the Efficient share from the same stamp.

Halo (home / laptop agents)

Install

curl -fsSL https://halo-get.aperion.ai | sh

Route license

halo license apply
halo license show

Cheap agent

halo agent add glm --provider openai
halo serve

Route turns it on. Name the cheap agent glm.

$100 / mo

Free and Cut never hop. Route scores the prompt and, when the turn looks easy, uses the glm agent's provider and key. Hard turns stay on the inbound agent (the one whose virtual key the runtime holds). Spend still counts against that inbound cap.

If first-run config has no routing.effort block, the binary is older than this feature. Re-run curl -fsSL https://halo-get.aperion.ai | sh and restart halo serve.

First run already writes this. If you named the cheap agent something else, change efficient_agent. If glm isn't registered, nothing hops — the call stays on the original agent.

routing:
  effort:
    mode: auto
    threshold: 0.5
    efficient_agent: glm
    efficient_model: glm-4.7
    capable_agent: researcher   # optional; omit to keep the inbound agent

Force a lane from the runtime: X-Halo-LCR: auto|local|frontier. That header wins over YAML. Restart halo serve after you edit config or apply a license.

OpenClaw: halo openclaw apply. Hermes: halo hermes apply. Same as the rest of Halo — don't hand-edit three runtime files if you can avoid it.

Modes at a glance

Mode	Smartflow	Halo Route
`auto`	Score the prompt. Hosted default. Easy → cheap model, hard → caller's model.	Score the prompt. Default. Easy → `glm` agent, hard → inbound agent.
`local`	Always the cheap provider/model you configured.	Always the `efficient_agent`.
`frontier`	Always the model on the request (or `capable_model` if set).	Always the inbound / `capable_agent`.
`off`	Leave routing alone. On-prem default until you set the env.	Free and Cut. Route with `mode: off` if you want YAML but no hops.

What this is not

It does

Cut spend on greetings, lookups, and short asks
Keep analysis, troubleshooting, and errors on a capable model
Retry once if the cheap lane fails
Let a header or a policy pin a team or a single call
Stamp the trace so you can see which lane ran

It does not

Call a judge LLM. There's no second bill for the decision
Move Halo keys off the machine
Hop on Free or Cut Halo. Safety and the bill-cut stay as they were
Replace your provider catalogue. It picks between two lanes you already have
Silence a pinned local provider. An explicit local target still wins

Quick checks

You want…	Do this
See it on hosted Smartflow	Send a "hi, what is Rust?" and a production traceback. The first should land on the cheap lane; the second stays on the model you named. Trace Request → Routing shows `lcr:efficient` or `lcr:capable`. Streamed playground chat stays on the model you asked for (`lcr:skipped stream`).
Never downgrade this call	`X-Smartflow-LCR: frontier` or `X-Halo-LCR: frontier`.
Always use the GPU box	Smartflow: `SMARTFLOW_LCR=local` and the Ollama/vLLM provider. Halo: `mode: local`.
Turn it off for one tenant	Policy attachment `lcr_mode: "off"` on that key or group.
Halo hops not firing	`halo license show` should read Route. `halo agent` list should include `glm` (or whatever you set). Restart `halo serve`.

Most turns don't needthe expensive model.

It does

It does not

Most turns don't need
the expensive model.