Smartflow is an on-premises AI governance control plane. LiteLLM is an MIT-licensed AI gateway for platform teams. Both put enterprise AI traffic behind one endpoint. They diverge on what happens to that traffic once it is there: Smartflow enforces policy inline and produces regulatory evidence from it, while LiteLLM concentrates on routing, spend control and observability.
The short version
LiteLLM is the faster path to a working routing layer and has one of the broadest provider catalogs in the market. Smartflow is the choice when the governance layer itself has to be auditable, when an examiner, not an engineer, is the eventual reader of the logs. Many Smartflow deployments begin as LiteLLM deployments and move when a regulator enters the room.
Feature Comparison
| Capability | Smartflow | LiteLLM |
|---|---|---|
| Semantic cache | MetaCache resolves a lookup in four phases, first hit wins: intent signature, intent near-miss, exact key, then a VectorLite BERT semantic match. Hit rates and token savings are workload-dependent. | Exact-match caching in memory (the default), Redis, Valkey, S3 or GCS, with IAM and Entra ID auth documented for ElastiCache, Memorystore and Azure Redis. Semantic caching runs on Redis, Valkey or Qdrant. |
| Enterprise SSO | Entra ID and Okta over OIDC; composes with SAML and Kerberos single sign-on. Per-user audit trails. | SSO for the admin UI is free for up to five users; beyond five it requires an enterprise license. JWT auth and SCIM provisioning are enterprise-licensed. |
| Policy engine | Visual editor, PII detection, topic restriction, jailbreak detection, output moderation. No code required. | Guardrails framework in the open-source proxy, with Presidio PII masking and a content filter built in, plus integrations with roughly forty third-party guardrail vendors. Secret detection and redaction are enterprise-licensed, as are per-key, per-model and tag-based guardrail control, dynamic guardrail parameters and team-level guardrail restrictions. |
| Regulatory evidence | Examination packages and model inventory mapped to FINRA 3110, HIPAA, SOX and the EU AI Act, generated from runtime records. | Vendor certifications published (SOC 2 Type 2, ISO 27001). Audit logging covers create, update, delete and regenerate actions on keys, teams, users and models, with S3 export, and is enterprise-licensed. Regulator-facing evidence generation is not a documented feature. |
| MCP and agent governance | MCP JSON-RPC gateway with tool caching, plus A2A agent orchestration under AIDA agent identity. | MCP gateway across Streamable HTTP, SSE and STDIO, with OAuth 2.0, per-key and per-team permissions, tool filtering and per-invocation cost tracking. A separate A2A agent gateway with agent cards and permission controls. |
| Runtime | Rust binary. | Python, with a Rust gateway announced in June 2026 and documented in beta. |
| Deployment | Docker and Kubernetes Helm. On-premises. Air-gapped capable. | Self-hosted by default. Helm on EKS, GKE and AKS, with official Terraform modules for AWS and GCP. Multi-region over a shared database runs on the open-source proxy; the admin and worker endpoint split and air-gapped deployment are enterprise options. |
| Provider support | OpenAI, Anthropic, Google, xAI, Mistral, Cohere, DeepSeek, Groq, OpenRouter, Meta, GLM and Kimi, local models through Ollama or vLLM, and any OpenAI-compatible endpoint. | 140+ providers and 1,800+ models, per LiteLLM's homepage. |
| License | Enterprise product. Source available on request. | MIT, with the enterprise/ directory under a separate license. |
Four rows in this table changed in September 2026. LiteLLM shipped semantic caching, an MCP gateway, an A2A agent gateway and air-gapped enterprise deployment, and published SOC 2 Type 2 and ISO 27001. Earlier versions of this page said otherwise and were wrong. In October 2026 we corrected the caching, SSO, guardrail, audit logging, deployment and provider rows against LiteLLM's current documentation. Sources are listed at the foot of the page.
What each one is built to do
LiteLLM describes itself as the AI gateway for platform teams, and the product follows that description honestly: put the full AI stack behind one key, see who is driving spend, cap it before it runs, and route each request to the model that should handle it. The center of gravity is the platform team's problem, fragmentation, cost, and visibility across a growing model estate.
Smartflow starts from a different question. Not who spent what, but what authority was exercised, against which policy version, and whether the record of it survives an examination three years later. That reframing is what produces the architectural differences below. It is not a claim that one problem matters more than the other; most enterprises have both, and a good many run something like LiteLLM underneath a governance obligation it was never designed to discharge.
Deployment and the data path
Both products self-host, and both reach air-gapped environments: LiteLLM lists air-gapped deployment among its enterprise options, and documents that no telemetry is stored on its servers when self-hosted. On this axis the two are closer than they were a year ago, and a comparison written before 2026 would mislead you.
The remaining difference is what the control plane is for. Smartflow's control plane is the policy decision point: every request is evaluated against a policy version, and that version identifier is written into the record alongside the decision. The question a regulated buyer asks is not whether data left the perimeter but whether the enforcement decision is reconstructible after the fact. That is an evidence design problem, and it is the one Smartflow is organized around.
Agent and MCP governance
This row used to be the clearest line between the two products. It is not any more. LiteLLM ships a full MCP gateway with three transports, OAuth 2.0 including PKCE flows, permissions scoped by key, team or organization, semantic tool search, and cost tracking per tool invocation. It also ships an A2A agent gateway with agent cards, per-team access control and iteration budgets. Anyone claiming LiteLLM has no agent story has not read the documentation.
Where Smartflow still differs is identity. LiteLLM governs agent access through keys, teams and organizations: the same primitives that govern human access. AIDA issues a cryptographic credential to the agent itself, scoped and revocable independently of whoever launched it, so the audit record names the agent as a principal rather than inheriting a human's key. In a supervision conversation under FINRA Rule 3110 that distinction is the conversation. Elsewhere it may be an abstraction you do not need.
Evidence for examiners
LiteLLM's audit logging records create, update, delete and regenerate operations against keys, teams, users and models, capturing the acting user, the action, previous values and updated values. That is a competent administrative audit trail, and it is enterprise-licensed. Its compliance posture is expressed the way most infrastructure vendors express it, through SOC 2 Type 2 and ISO 27001 attestations of the vendor's own controls.
Smartflow's Regulatory Examination Suite answers a different question. Not "is the vendor well run" but "can this institution produce, on demand, the inventory of models in use, the policy in force on a given date, and the record of what each agent was permitted to do." That artifact is what a bank examiner asks for, and assembling it by hand after the fact is where most of the cost of an examination sits.
Where LiteLLM excels
LiteLLM has one of the broadest provider catalogs in the market, one of the largest open-source communities in this category, and the fastest path from zero to a working routing layer. The free tier is useful rather than a trial in disguise: virtual keys, spend tracking, budgets, rate limits, teams, load balancing, fallbacks and guardrails are all in the MIT-licensed proxy. Its 2026 engineering record is strong (a Rust migration in beta, an auto-router in early access, a managed agents platform in public preview) and it responded to a March 2026 supply-chain incident with a published post-mortem, engaged forensics and a rebuilt CI pipeline, which is better disclosure than the category norm.
For an engineering team building internal tooling without a regulator in the picture, LiteLLM is a strong choice and Smartflow is likely more machinery than the problem needs.
When to choose Smartflow
- Regulatory obligations: HIPAA, SOX, FINRA, the EU AI Act, or any framework requiring per-user audit trails and reproducible evidence
- Enterprise identity as a governance boundary: the identity provider you already run, Entra ID or Okta over OIDC, as the control point for AI access
- Agent identity: a scoped, revocable credential held by the agent itself, where the audit record still names the person who authorized it
- Examination readiness: evidence generated from runtime records rather than assembled by hand when the request arrives
Migrating from LiteLLM
Both expose an OpenAI-compatible interface, so the application-side change is usually a base URL and a key. The work that is not trivial is policy translation: LiteLLM guardrail configuration and per-key controls do not map one-to-one onto a policy model that versions its rules and writes the version into every decision record. Plan that as a design exercise rather than a config conversion, and run both in parallel while the policy set is validated. Most teams keep the provider catalog and virtual-key structure they already have.
Common questions
Does LiteLLM support semantic caching? Yes. LiteLLM documents both exact-match and semantic caching in the open-source proxy. Semantic caching runs on Redis, Valkey or Qdrant; exact-match caching also runs in memory, on managed cloud Redis services and on object storage.
Is LiteLLM free for enterprise use? The proxy is MIT-licensed and free to self-host. SSO beyond five users, SCIM, audit logs, secret-manager integrations, air-gapped deployment and several guardrail controls require a commercial license, priced annually by gateway request capacity, deployment architecture and support needs rather than per seat.
Can the two run together? Yes, and some do: LiteLLM as the provider abstraction, Smartflow as the policy and evidence layer in front of it. Whether that is worth the extra hop depends on how much of the LiteLLM provider catalog you use.
What changed in LiteLLM in 2026? Semantic caching, an MCP gateway, an A2A agent gateway, a Rust runtime in beta, SOC 2 Type 2 and ISO 27001, and a March 2026 supply-chain incident affecting two PyPI releases for roughly forty minutes, with official Docker images unaffected.
Related reading
Enterprise AI gateway guide · Semantic caching · Policy as code · Examination readiness · All comparisons · Model allowlist and denylist · AI guardrails · Agentic AI · MCP security
Verified against each vendor’s published documentation in October 2026. Vendor capabilities change. If something here is out of date, tell us and we will correct it.
LiteLLM sources: litellm.ai, litellm.ai/pricing, litellm.ai/enterprise, docs.litellm.ai (caching, enterprise, mcp, a2a, scim, data_security, deploy), docs.litellm.ai/blog (security-update-march-2026, soc2-type-2-report), github.com/BerriAI/litellm LICENSE.
Put this in the path of your own agents.
Policy enforced inline between your agents and every model and tool they reach, with a record bound to the human who owns it.
Request a Demo Read the docs