The brief
A difficult operating reality shaped Shodh from the start: questions arrive in unpredictable ways, while the available knowledge stays fixed and curated.
Simply adding more papers isn’t always possible in this environment. New sources may take too long to integrate, and expanding the corpus after a research need has already emerged may not solve the immediate problem.
A different approach was needed. Shodh absorbs incoming questions, groups them intelligently against indexed research, sends them through retrieval workers in efficient batches, and protects the system when demand exceeds safe capacity.
Why Shodh was built this way
No individual researcher can realistically read and organise research literature manually at the pace it keeps growing. An intelligent layer between the researcher and that growing body of knowledge is what Shodh provides.
Combining semantic retrieval with generative AI helps researchers move from:
Finding information → Understanding information → Connecting information
The goal is simple:
When the knowledge base cannot scale quickly enough, make every available unit of context do more useful work.
Workloads could range anywhere from straightforward concept checks to deep, multi-paper investigations. Answering questions wasn’t the hard part — keeping the entire system grounded while curiosity changed faster than the indexed corpus could respond was.
The service needed to:
- Accept exploratory questions without immediately overwhelming the retrieval layer.
- Use fixed research material as efficiently as possible.
- Keep question handling separate from document processing.
- Prevent context windows from growing without control.
- Handle overload without causing a full system failure.
- Support questions that could be safely deferred when capacity was exhausted.
For certain research workflows, letting every question remain queued until latency and memory usage became unmanageable was riskier than a controlled deferral.
The central idea
Between unpredictable researchers and comparatively expensive document retrieval, Shodh acts as a knowledge buffer.
Rather than sending every question directly to the full corpus, the system places incoming work into a shared index. Compatible queries then get grouped by a retrieval layer and dispatched to search workers together.
The architecture is:
Researchers → Interface → FastAPI → Shared index → Semantic retriever → Retrieval workers → Context → AI Response
Room to absorb short-term bursts comes from this separation, while the retrieval process gets help surfacing more relevant context per execution cycle.
Making one retrieval pass do more
Available context can go underused when questions are sent individually. Multiple incoming queries get combined into a single search operation through dynamic retrieval, with the retriever waiting until either enough compatible questions have arrived or a configured time window has elapsed.
A practical balance results: under light traffic, questions get answered quickly to avoid unnecessary waiting; under heavy traffic, questions get grouped to improve throughput and context utilisation. Because the retrieval decision happens at runtime, researchers never need to know how the system prefers to organise its index.
Grouping compatible questions
Not every question is equally efficient to process alongside another. Where topics or domains vary, topic-aware retrieval can group similar questions — reducing wasted work caused by excessive context switching and helping the workers process each batch more efficiently.
More than counting questions, then, is what the retriever is doing — it’s shaping the workload around the limits and behaviour of the research system.
A shared index for distributed workers
Incoming research work gets coordinated across multiple processes and workers through embedding-based indexing. Questions don’t fragment across isolated application instances because a shared index prevents it, giving workers a common pool of pending research and the retrieval layer a wider view of the available work.
Keeping workers busy and maintaining consistent behaviour as traffic moves between normal operation and burst conditions both get easier as a result.
Backpressure by design
Overload doesn’t get solved by an unbounded queue — it just gets delayed. If questions keep accumulating beyond what the retrieval workers can process, memory usage and waiting time can grow until the entire service becomes unstable.
Bounded queues and explicit backpressure prevent that outcome in Shodh.
Under normal conditions:
Incoming question → Index → Retrieve → Context → Response
When the system reaches its safe capacity:
Capacity reached → New work deferred or dropped → Existing service protected
This behaviour is intentional. Degrading in a controlled way, rather than letting overload cascade through every layer, is the design goal.
Load shedding for research workloads
Not every workload requires every question to be processed, and this one didn’t always have that requirement. For certain research scenarios, processing a delayed question may matter less than keeping the service available for new inquiries — so Shodh supports controlled load shedding when the system is under extreme pressure.
Questions aren’t discarded randomly, though. Capacity limits and overload rules govern the behaviour so the system can protect its core retrieval path. Graceful degradation is the priority: process as much useful work as the infrastructure can safely handle, rather than letting unlimited demand bring everything down.
Why autoscaling was not the answer
New capacity becoming available quickly enough to respond to demand is the assumption reactive autoscaling makes. That assumption didn’t fit this workload. Questions could rise sharply while the system still had to operate within fixed corpus limits and curated collections — and by the time additional resources were ready, the most severe part of the burst might already have passed.
Optimising the capacity already running is where Shodh focuses instead: dynamic retrieval improves work per search execution, shared indexing prevents traffic fragmentation, backpressure limits unsafe accumulation, and load shedding protects the system at its boundaries.
Designed for bursty curiosity
The real engineering challenge can hide behind average traffic numbers. A system performing well at a steady question rate may behave very differently when a sudden burst arrives, so question variability was treated as a first-class design concern in Shodh.
Short-term differences between question arrival and retrieval execution get absorbed by the index. The retriever uses those waiting questions to improve throughput, while capacity controls keep the buffer from becoming an uncontrolled backlog.
Private deployment constraints
Privately hosted infrastructure is what Shodh was designed for. Unlimited cloud capacity or instant provisioning isn’t assumed by the architecture — instead, it’s built around the resources available inside the client’s environment, focused on using those resources efficiently.
Measuring the system
Intuition alone can’t tune high-throughput research infrastructure. Benchmarking and monitoring workflows are built into Shodh for evaluating question throughput, index depth, batch size and batch formation, waiting time, retrieval latency, worker utilisation, overload behaviour, and load-shedding thresholds.
Queue limits, retrieval windows, worker counts, and dispatch behaviour can all get tuned against the actual workload and hardware thanks to these measurements.
Outcome
A research-serving layer capable of handling extreme question variability on fixed private infrastructure is what Shodh created. Scenarios ranging from straightforward concept checks to deep, multi-paper investigations were supported by making the most of existing context, rather than depending on infrastructure that might not arrive in time.
Project highlights
- Designed a high-throughput research system for bursty academic workloads.
- Supported question scenarios from simple concept checks to deep investigations.
- Built a Python and FastAPI question-ingestion layer.
- Used embedding-based indexing as a shared distributed question queue.
- Implemented dynamic batching with configurable size and waiting limits.
- Added topic-aware batching for varied research domains.
- Coordinated distributed retrieval workers.
- Added bounded queues and explicit backpressure.
- Implemented controlled load shedding during extreme overload.
- Used Nginx as the request gateway and load balancer.
- Designed for fixed, privately hosted infrastructure.
- Added benchmarking and monitoring for throughput, latency, batching, and overload behaviour.