Smartflow Enterprise
Horizontal Scaling & Latency Analysis

Executive Summary

This whitepaper documents systematic load testing of Smartflow Enterprise, LangSmart's Rust-based AI gateway, comparing end-to-end latency against direct OpenAI API calls across a range of request rates (5–80 RPS). Tests were conducted on a production Kubernetes cluster with two application nodes, using the gpt-4o-mini model.

Two workload modes were tested: mixed (50% repeated prompts enabling cache hits, representing realistic enterprise usage) and cache-miss (100% unique prompts, isolating pure proxy overhead). Both modes were run at each RPS level for a statistically sufficient duration.

Test Environment & Infrastructure

Cluster Configuration

Architecture Overview

Test Methodology

Request Rate Control

The test harness (rps_comparison_test.py) fires requests at a precise target rate using an async token-bucket scheduler built on asyncio and aiohttp. Each RPS level ran for a controlled duration (20–25 seconds) to capture steady-state behaviour rather than burst characteristics. The test client and the Smartflow cluster are co-located in the same DigitalOcean region (NYC1) to eliminate inter-region latency noise.

Two Workload Modes

Comparison Target

For each RPS level, the identical workload was run against direct OpenAI (api.openai.com) immediately after the Smartflow run, using the same API key, model, and prompt set. This apples-to-apples comparison eliminates model response time variability between test runs.

Metrics Collected

• p50 (median latency) — typical request experience
• p95 latency — tail latency under load
• p99 latency — worst-case outliers
• Success rate — HTTP 200 responses as a percentage of total requests fired
• Cache hit rate — requests served by Smartflow cache (x-smartflow-cache-hit: true header)
• Actual RPS achieved — measured vs target (confirms scheduler accuracy)

Results: Mixed Workload (Cache-Enabled)

The mixed workload (50% repeated prompts) reflects the most common enterprise use pattern — employees across a company asking similar questions about HR policy, internal documentation, product features, or code standards. Cache hit rates of 50–55% were sustained throughout all RPS levels.

Latency at 5 RPS: Cache Cold Start

At 5 RPS, Smartflow and direct OpenAI are essentially equivalent (642ms vs 614ms, p50). This is expected — at this rate the cache is cold and only the first occurrence of the repeated anchor prompt is cached. Once warm, subsequent runs at 5 RPS would show the same acceleration seen at higher RPS. In production, the cache is pre-warmed from prior traffic so cold-start effects are rare.

Results: Cache-Miss Workload (Proxy Overhead Isolation)

The cache-miss test sends 100% unique prompts, guaranteeing every request traverses the full proxy pipeline and reaches OpenAI. This isolates the pure overhead that Smartflow adds to each request: ingress TLS termination, policy evaluation, guardrail check, Redis miss, upstream forwarding, and response streaming back through the proxy.

Linear Scaling 5–60 RPS

The most important finding in the cache-miss data is the extraordinary p50 stability from 5 to 60 RPS. Overhead ranges from 141ms to 194ms — only a 37% variation across a 12× increase in request rate. This is the hallmark of a well-architected async system: no queuing buildup, no worker thread starvation, no lock contention degrading throughput.

Saturation at 80 RPS (All-Miss)

At 80 RPS with 100% unique prompts, p50 jumps from ~730ms to 8,762ms — a clear saturation signal. With 2 proxy pods each handling 40 simultaneous all-miss requests that all wait on OpenAI round-trips, the internal work queue backs up. This is the worst-case scenario for the platform: maximum load, zero cache benefit.

Architecture & Scaling Analysis

What the Proxy Does Per Request

Every request through Smartflow traverses the following pipeline, contributing to the ~190ms cache-miss overhead:

• TLS termination at ingress — nginx-ingress handles TLS, adds ~1–5ms
• Policy evaluation — guardrail rules checked against the request in-memory, <1ms (Rust, no I/O)
• Redis cache lookup — Phase 1 exact match hash lookup, ~1–3ms cluster-internal RTT
• BERT semantic similarity check — cosine similarity against cached embeddings (on cache miss), ~5–15ms
• Upstream forwarding — HTTP/1.1 to api.openai.com, includes OpenAI's own queue time
• Compliance pre-check — runs in parallel with OpenAI call, does not add to latency on pass
• Cache write-back — fire-and-forget (non-blocking), does not add to response latency

Why Overhead Is ~190ms, Not Sub-10ms

The ~190ms overhead compared to direct OpenAI is not proxy processing time — it is dominated by the additional network hop through the Kubernetes ingress and the connection overhead of TLS renegotiation between the proxy and OpenAI (the proxy maintains a persistent connection pool, but connection establishment for new pool members adds latency on the first request). In production deployments where the Smartflow cluster is co-located with the application (same VPC or data centre), this overhead is typically 40–80ms.

Latency Tuning for Latency-Sensitive Workloads

The ~190ms cache-miss overhead measured in this test reflects a full inspection configuration: semantic guardrail evaluation, BERT embedding lookup, compliance pre-check running in parallel, VAS audit logging, and policy enforcement. For workloads where sub-100ms proxy overhead is required, Smartflow supports a reduced inspection profile that disables or defers non-critical pipeline stages:

• Disable BERT semantic cache lookup — skips the embedding similarity phase and goes straight to OpenAI on miss. Saves 5–20ms per request.
• Async compliance checking — move compliance from pre-check to post-response (log-and-allow). Eliminates compliance latency entirely from the hot path.
• In-memory-only policy evaluation — policy rules are already evaluated in Rust in-memory (<1ms), no tuning needed.
• Persistent upstream connection pooling — Smartflow maintains a persistent HTTP/1.1 connection pool to OpenAI, eliminating TCP handshake overhead on every request. Ensure pool size matches your RPS target.
• Co-locate with your application — running Smartflow in the same VPC or data centre as your application reduces the ~190ms measured overhead (which includes inter-region TLS overhead) to 40–80ms in a co-located topology.

Semantic Caching as a TPM and Rate Limit Management Strategy

Beyond raw latency, Smartflow's semantic cache delivers a critical operational benefit for high-volume deployments: it directly reduces the number of tokens sent to the LLM provider. Every cache hit is a request that never reaches OpenAI — no tokens consumed, no quota used, no position in OpenAI's request queue.

This matters significantly in environments approaching OpenAI's TPM (tokens per minute) or RPM (requests per minute) rate limits. At 50% cache hit rate, an application consuming 2M TPM effectively operates at 1M TPM from OpenAI's perspective — keeping it below tier thresholds, avoiding 429 rate limit responses, and eliminating the need to implement exponential backoff retry logic in application code.

• Rate limit headroom — cache hits don't count against your OpenAI RPM/TPM limits, effectively doubling or tripling your usable capacity per tier.
• Queue backpressure relief — OpenAI processes requests in queues per organisation tier. Reducing the volume of requests reaching OpenAI reduces queue depth and lowers the p95/p99 tail latency for the requests that do reach it.
• Cost reduction — every cache hit is zero LLM cost. At 55% hit rate on a 10M token/day workload, 5.5M tokens are served for free, with only the embedding comparison cost (sub-millisecond, in-memory) as overhead.
• Burst absorption — traffic spikes that would cause 429s from OpenAI are absorbed by the cache. Smartflow serves the spike from Redis while the OpenAI-bound volume stays flat.

Horizontal Scaling Model — No Technical Ceiling

Smartflow's proxy layer is fully stateless. Each pod independently handles requests and shares cache state through Redis. There is no inter-pod coordination, no leader election, no shared memory — adding a pod increases capacity linearly with zero architectural changes. The Kubernetes HPA can scale proxy pods from 2 to 64 based on CPU or request rate metrics automatically.

There is no technical ceiling on request rate. At 250–500 RPS, p50 latency on the proxy layer remains identical to what was measured at 40 RPS — the additional pods simply distribute the load. The only components that require separate scaling consideration beyond ~8 pods are Redis (use Redis Cluster or a managed Redis service) and the ingress load balancer (DigitalOcean, AWS ALB, or nginx-ingress all handle 500+ RPS without configuration changes).

Why Python-Based Proxies Cannot Match This Scaling Model

A common alternative architecture uses a Python-based AI gateway (LiteLLM, custom FastAPI proxy, etc.) deployed on Kubernetes or Docker Compose. While Python async (asyncio/aiohttp) enables concurrency, it has fundamental limitations that prevent it from achieving Smartflow's per-pod efficiency:

• Global Interpreter Lock (GIL) — Python's GIL prevents true thread-level parallelism. Even with asyncio, CPU-bound operations (JSON parsing, policy evaluation, embedding computation) block the event loop for all concurrent requests on that pod.
• Memory overhead per pod — a Python proxy process typically consumes 200–500MB RAM at idle. A Rust binary consumes 10–30MB. At 16 pods, the difference is 3–8GB of wasted RAM that could be used for application workloads.
• Per-pod throughput ceiling — a Python asyncio proxy sustains approximately 8–15 RPS of all-miss proxy load before p95 latency begins degrading. A Rust proxy pod (as measured) sustains ~30 RPS. You need 2–4× more Python pods to match equivalent Rust throughput.
• Synchronous library risk — any synchronous call (a blocking Redis client, a synchronous HTTP call, even a slow JSON serialisation) in a Python async context blocks the entire event loop for that process. Rust's type system and async model make this class of bug impossible.
• Docker Compose without Kubernetes — Docker Compose --scale can run multiple containers but provides no health-based autoscaling, no rolling updates, no cross-node distribution, and no load balancing beyond a basic round-robin. For production HA, Kubernetes (or Docker Swarm) is required regardless of language.

Key Findings

Recommendations

For Deployments Targeting <10 RPS

For Deployments Targeting 10–60 RPS

For Deployments Targeting 60–200 RPS

For High-Cache-Hit Workloads

Reproducing This Test

All test tooling is open and available in the Smartflow repository. To run the full comparison against your own deployment:

bash

# Install dependencies
pip install aiohttp

# Run mixed workload (5, 10, 20, 40 RPS)
SF_API_KEY="your-openai-key" \
SF_BASE="https://your-smartflow-host" \
python3 tests/rps_comparison_test.py \
  --rps 5 10 20 40 \
  --duration 30 \
  --mode mixed

# Run cache-miss isolation (proxy overhead only)
SF_API_KEY="your-openai-key" \
SF_BASE="https://your-smartflow-host" \
python3 tests/rps_comparison_test.py \
  --rps 5 10 20 40 60 \
  --duration 30 \
  --mode miss

# Smartflow-only (no direct OpenAI comparison)
SF_API_KEY="your-openai-key" \
SF_BASE="https://your-smartflow-host" \
python3 tests/rps_comparison_test.py \
  --rps 5 10 20 40 60 80 \
  --duration 60 \
  --mode mixed \
  --smartflow-only

Test Environment Requirements

• Python 3.9+ with aiohttp installed
• A valid OpenAI API key (or any API key your Smartflow proxy accepts)
• Network access to both your Smartflow host and api.openai.com
• Run the test client from a machine with low, stable latency to the Smartflow cluster for best comparability

---

About LangSmart — LangSmart builds Smartflow Enterprise, a Rust-based AI gateway for regulated organisations. Smartflow provides enterprise identity integration, a four-phase semantic cache, real-time policy enforcement, compliance tooling, and a management dashboard — all in a single binary deployable on Kubernetes or Docker Compose.

For questions about this whitepaper, enterprise licensing, or deployment support, contact support@langsmart.ai.