The brief

Vega is a high-throughput inference serving system designed for workloads where traffic can fluctuate dramatically, from roughly 10,000 requests to bursts approaching one million requests, while operating on fixed, privately hosted infrastructure.

The challenge was not simply handling high traffic. The system had to absorb sudden traffic spikes without waiting for new compute instances or pods to start, while operating under a workload where some sensor-data requests could be safely dropped during extreme overload.

Instead of relying on reactive autoscaling, we designed the system around efficient utilisation of the compute capacity already available.

The architecture

The core of Vega is a dynamic request-batching pipeline.

Incoming inference requests are placed into a shared queue and grouped into batches before being sent to the model. Rather than executing a separate inference operation for every request, multiple requests can share a model forward pass.

The resulting flow is:

Clients → Nginx → FastAPI → Shared Queue → Dynamic Batcher → Inference Workers → Model

This separates request ingestion from model execution and allows the inference layer to process bursts efficiently.

Dynamic batching

Vega dynamically groups incoming requests based on configurable batching constraints.

A batch can be dispatched when it reaches its maximum size or when its waiting window expires. This creates a balance between throughput and latency: requests get an opportunity to share compute, but they are not held indefinitely waiting for a full batch.

For workloads where input sizes vary, length-aware batching can also group requests with similar characteristics, reducing unnecessary padding and improving computational efficiency.

Shared inference queue

The distributed configuration uses Redis Streams as a shared request queue.

This allows multiple workers or application processes to participate in a common inference pipeline rather than maintaining isolated queues.

A shared queue helps prevent traffic fragmentation and gives inference workers a broader pool of pending work from which efficient batches can be constructed.

Load shedding and backpressure

A key requirement of the workload was that individual sensor-data requests could be dropped when the system reached its safe capacity.

Vega treats this as a deliberate reliability mechanism rather than an unexpected failure.

Queues are bounded, and when the system is under extreme pressure, additional work can be rejected instead of allowing an unbounded backlog to consume resources and cause the entire service to become unstable.

This creates explicit backpressure:

Normal load → Queue → Batch → Inference

Extreme load → Capacity reached → Controlled request drop

For this workload, losing individual requests was preferable to allowing overload to cascade into system-wide failure.

Why we did not use autoscaling

Reactive autoscaling was not a good fit for this particular workload.

Traffic could increase faster than new pods or compute resources could become available. The service also needed to operate on private infrastructure, making rapid elastic provisioning less practical.

Rather than scaling after the spike arrived, Vega focused on getting more useful work from the infrastructure that was already running.

Dynamic batching improves compute utilisation, shared queuing coordinates work, and controlled load shedding prevents overload from destabilising the service.

Built for bursty traffic

Average request volume is not enough to describe this workload.

The system could experience relatively modest traffic and then suddenly receive an extreme burst. Vega was designed to absorb these bursts through queueing and batching while maintaining predictable behaviour when capacity was exceeded.

This turns the inference service into a controlled buffer between unpredictable traffic and comparatively expensive model execution.

Private infrastructure

Vega was designed for deployment within private infrastructure rather than depending on cloud-based autoscaling.

This makes the architecture suitable for organisations that need to keep inference workloads inside their own environments or operate under fixed compute constraints.

The system focuses on improving utilisation of the resources available rather than assuming unlimited elastic capacity.

Observability and benchmarking

Performance was treated as an engineering problem that needed to be measured.

The project includes benchmarking and monitoring capabilities for evaluating throughput, latency, batching behaviour, and overload characteristics.

These measurements make it possible to tune batch sizes, waiting windows, queue capacity, and worker configuration around the actual workload and available hardware.

The engineering principle

Vega was built around a simple systems principle:

When you cannot scale the infrastructure fast enough, make every unit of existing compute do more useful work.

Dynamic batching improves compute utilisation. Shared queuing coordinates workers. Bounded queues provide backpressure. Load shedding protects the system during overload.

Together, these mechanisms create inference infrastructure designed to remain predictable when traffic becomes extreme.

Project highlights

  • Designed for traffic ranging from approximately 10K to 1M requests
  • Private-infrastructure deployment model
  • Dynamic inference batching
  • Redis Streams-based shared request coordination
  • Distributed inference workers
  • Bounded queues and controlled load shedding
  • Backpressure under extreme traffic
  • Length-aware request batching
  • Nginx request gateway and load balancing
  • Monitoring and benchmarking infrastructure
  • Python/FastAPI-based inference service