Editor’s note, November 2026. This post first carried figures from an April 2026 evaluation. They are retired. The numbers below come from the October 8, 2026 run published with its full method on the Smartflow latency page.
AI gateways are at a crossroads. The industry is grappling with a lack of transparency. While marketing claims of “20x speedup” are common, the technical reality of running these systems in production, especially within highly regulated on-premises environments, is rarely discussed in hard numbers.
At APERION, we judge enterprise infrastructure by measured performance. That is why we are publishing our latency numbers with the method behind them, and issuing a challenge to the market: Show me the p95.
The Method
- Build: smartflow-enterprise:v1.7.132, fast path on. Two proxy pods, one per node, TLS terminated at the ingress.
- Load: NVIDIA AIPerf 0.6.0.post1, one pass, from a generator in the same region. 60 seconds at each rate, 20 warm-up requests discarded.
- Workload: gpt-4o-mini, 64 max tokens, distinct short prompts. Cache off. Routing off.
- Baseline: direct calls to OpenAI with the same key, minutes apart.
The Results
| Requests per second | Gateway p50 / p95 | Direct p50 / p95 | Added at p50 |
|---|---|---|---|
| 30 | 1,092 / 1,861 ms | 1,081 / 1,670 ms | +11 ms |
| 40 | 1,069 / 1,777 ms | 1,074 / 1,765 ms | -5 ms |
| 60 | 1,081 / 1,795 ms | 1,028 / 1,657 ms | +53 ms |
| 80 | 1,097 / 1,850 ms | 1,019 / 1,623 ms | +78 ms |
Concurrency matched to rate. The model call is about a second. Through 80 requests a second, the gateway’s share of the median is between zero and 78 milliseconds.
What This Run Does Not Show
No p99. No cache-hit latency. Hit rate depends on the workload, so we measure it on the customer’s own traffic. The full tables, the error counts and the address for reporting a mistake are on the latency page.
Why the Control Point Has to Be Yours
For banking, insurance, and healthcare, the governance layer cannot be a third-party cloud. As we like to say, “You cannot govern what you don’t own.” If your AI gateway routes prompts through an external cloud to perform the governance check, you haven't secured your data – you've just added another point of failure.
Smartflow operates as a network-layer control plane within your own infrastructure. No data leaves the network, yet the performance exceeds cloud-hosted alternatives.
The “Show Me the p95” Challenge
The AI gateway is critical infrastructure. In any other infrastructure category – databases, load balancers, firewalls – vendors are expected to provide reproducible benchmarks. Currently, the AI gateway market is an outlier.
We are calling on all vendors to stop hiding behind “up to” marketing claims and disclose:
- p95 and p99 Latency – What does the system look like under load?
- Hardware Specs – Was this run on a supercomputer or a standard server?
- Similarity Thresholds: what was the hit rate at the similarity threshold used?
Transparency shouldn't be a feature; it should be the standard. We’ve shown our numbers. It’s time for the rest of the market to do the same, and enterprises to hold them accountable.
Interested in seeing how Smartflow performs on your own infrastructure? Contact our team and try it for yourself.
Put this in the path of your own agents.
Policy enforced inline between your agents and every model and tool they reach, with a record bound to the human who owns it.
Request a Demo Read the docs