Show Me the p95: Our AI Gateway Latency Numbers, With the Method

Show Me the p95: AI gateway latency numbers, with the method

Editor’s note, November 2026. This post first carried figures from an April 2026 evaluation. They are retired. The numbers below come from the October 8, 2026 run published with its full method on the Smartflow latency page.

AI gateways are at a crossroads. The industry is grappling with a lack of transparency. While marketing claims of “20x speedup” are common, the technical reality of running these systems in production, especially within highly regulated on-premises environments, is rarely discussed in hard numbers.

At APERION, we judge enterprise infrastructure by measured performance. That is why we are publishing our latency numbers with the method behind them, and issuing a challenge to the market: Show me the p95.

The Method

  • Build: smartflow-enterprise:v1.7.132, fast path on. Two proxy pods, one per node, TLS terminated at the ingress.
  • Load: NVIDIA AIPerf 0.6.0.post1, one pass, from a generator in the same region. 60 seconds at each rate, 20 warm-up requests discarded.
  • Workload: gpt-4o-mini, 64 max tokens, distinct short prompts. Cache off. Routing off.
  • Baseline: direct calls to OpenAI with the same key, minutes apart.

The Results

Requests per secondGateway p50 / p95Direct p50 / p95Added at p50
301,092 / 1,861 ms1,081 / 1,670 ms+11 ms
401,069 / 1,777 ms1,074 / 1,765 ms-5 ms
601,081 / 1,795 ms1,028 / 1,657 ms+53 ms
801,097 / 1,850 ms1,019 / 1,623 ms+78 ms

Concurrency matched to rate. The model call is about a second. Through 80 requests a second, the gateway’s share of the median is between zero and 78 milliseconds.

What This Run Does Not Show

No p99. No cache-hit latency. Hit rate depends on the workload, so we measure it on the customer’s own traffic. The full tables, the error counts and the address for reporting a mistake are on the latency page.

Why the Control Point Has to Be Yours

For banking, insurance, and healthcare, the governance layer cannot be a third-party cloud. As we like to say, “You cannot govern what you don’t own.” If your AI gateway routes prompts through an external cloud to perform the governance check, you haven't secured your data – you've just added another point of failure. 

Smartflow operates as a network-layer control plane within your own infrastructure. No data leaves the network, yet the performance exceeds cloud-hosted alternatives.

The “Show Me the p95” Challenge

The AI gateway is critical infrastructure. In any other infrastructure category – databases, load balancers, firewalls – vendors are expected to provide reproducible benchmarks. Currently, the AI gateway market is an outlier.

We are calling on all vendors to stop hiding behind “up to” marketing claims and disclose:

  1. p95 and p99 Latency – What does the system look like under load?
  2. Hardware Specs – Was this run on a supercomputer or a standard server?
  3. Similarity Thresholds: what was the hit rate at the similarity threshold used?

Transparency shouldn't be a feature; it should be the standard. We’ve shown our numbers. It’s time for the rest of the market to do the same, and enterprises to hold them accountable.

Interested in seeing how Smartflow performs on your own infrastructure? Contact our team and try it for yourself.

Craig Alberino
Craig Alberino
Craig Alberino is the Co-Founder and CEO of APERION, which builds the runtime governance layer for AI agents in regulated enterprises. Inline policy enforcement and identity-bound audit, deployable on premises.

Put this in the path of your own agents.

Policy enforced inline between your agents and every model and tool they reach, with a record bound to the human who owns it.

Request a Demo Read the docs