How we built VakeelSaathi: legal RAG on 50,000+ Indian court judgments

The problem: legal research shouldn't take four hours

India's legal ecosystem runs on judgments. Every argument in court, every contract dispute, every advisory memo starts with a question: what did courts say about this before? Answering that question the traditional way means opening Indian Kanoon or Manupatra, running keyword searches, opening 20 tabs, reading through irrelevant hits, and finally landing on the two or three cases that matter. After 3 to 6 hours of research.

When we started building VakeelSaathi (Decipher's legal operations SaaS), the founding hypothesis was simple: a lawyer should be able to ask "what have courts held about arbitral awards being challenged under Section 34?" and get a cited answer in 30 seconds.

Building it was harder than the hypothesis suggested.

The architecture

VakeelSaathi's legal Q&A runs on a RAG pipeline. Retrieval-Augmented Generation is straightforward in theory. Fetch relevant documents, give them to an LLM as context, let the LLM answer using only what you gave it. In practice, at 50,000+ documents and Indian legal complexity, every step required careful engineering.

Model selection: why Claude beat GPT-4 on Indian legal reasoning

Before locking in Claude, we ran evaluations on 500 curated Indian legal questions across GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and Llama 3.1 70B (self-hosted). Grading criteria: factual accuracy, correct citation format, sensitivity to jurisdiction (SC vs High Court binding force), and refusal quality on out-of-scope queries.

Results:

Model	Accuracy	Notes
Claude 3.5 Sonnet	89%	Near-perfect citation format, best refusal behavior
GPT-4o	84%	Occasional citation fabrications, slightly weaker on Indian statute interpretation
Gemini 1.5 Pro	79%	Verbose answers, unreliable citation format
Llama 3.1 70B (self-hosted)	71%	Cheapest to run, best for high-volume simple queries

Production runs Claude for user-facing legal Q&A. Llama 3.1 handles internal batch processing (bulk case summarization, catchword extraction). This split cuts our monthly API bill by roughly 60% versus running Claude everywhere.

How we killed hallucinations

"Reduce hallucinations" is what every AI vendor claims. We measured ours. Here's what worked:

Layer 1: Retrieval quality

Most hallucinations trace back to bad retrieval. The model got weak context, so it filled the gap. Reranking with Cohere improved top-3 retrieval precision from ~62% to ~91% on our eval set. This single change removed the majority of hallucination sources.

Layer 2: Strict grounding prompts

The system prompt is 800+ words. Key rules:

• Every factual claim must cite a specific paragraph from the retrieved chunks
• If no chunk supports a claim, the model must say "I don't have direct authority on this"
• Never invent case names, citations, dates, or holdings
• Distinguish binding precedent (Supreme Court, applicable High Court) from persuasive precedent (other High Courts, tribunals)

Layer 3: Confidence thresholds

Below a certain retrieval-similarity threshold, the system doesn't attempt an answer. It surfaces the top matches it found and lets the user decide. In legal, "I don't know" is a valid and expected answer. Pretending otherwise is dangerous.

Layer 4: Nightly evals

500 curated questions run through the system every night. Any regression on accuracy or citation quality alerts our on-call. This is standard software practice applied to AI. A habit far too few teams building AI products actually maintain.

Latency work: getting to sub-500ms retrieval

Initial retrieval was around 1.8 seconds. Too slow for a conversational UI. Four optimizations got us to 480ms average:

1. Parallel dense + sparse retrieval. Instead of running them serially, we fire both simultaneously and merge when both return. Saved ~300ms.
2. Cached embedding model. For the query embedding step, we moved from an OpenAI API call to a locally-hosted sentence-transformer for the initial embedding, calling the API only for reranking. Saved ~250ms per query and reduced cost.
3. Pinecone region optimization. Moved index to ap-south-1 (Mumbai) since our users are in India. Network latency halved.
4. Reranker batch size tuning. Cohere's rerank endpoint has non-linear latency with batch size. Tuning from 50 to 30 candidates cut this step 40%.

Full user-facing latency (including LLM generation) is 2-4 seconds for a cited answer. We stream tokens so users see the answer forming immediately.

Multilingual: 8 Indian languages, one system

VakeelSaathi supports legal drafting in Hindi, Marathi, Tamil, Bengali, Telugu, Kannada, Malayalam, and Gujarati. English is the primary language for legal research (since Indian case law is in English); regional languages are used for drafting notices, agreements, and client-facing documents.

The setup:

• Retrieval always runs against the English case-law index
• Generation uses Claude 3.5 for English drafts
• Language-specific fine-tuned Llama 3.1 models handle output in each regional language, seeded with the English draft as context
• Post-generation validation checks legal terminology consistency against a bilingual glossary

The results, in numbers

What this cost. And what it costs to run

• Prototype phase (3 weeks): ~₹2.5 lakh. 5,000-judgment prototype, model comparison, first UI
• Production build (10 weeks): ~₹15 lakh. Full 50k index, ingestion pipeline, multilingual, WhatsApp bot, evals, security review, deployment
• Ongoing operations: ~₹80,000/month. Model API costs (Claude + Cohere + embeddings), Pinecone, monitoring, weekly re-indexing, quality evals

For comparison: hiring a single mid-level lawyer to do the same volume of research work costs ~₹1.2 lakh/month fully loaded, and they can process a small fraction of the queries per day.

What we'd do differently

Two things, in hindsight:

1. Start with hybrid retrieval from day one. We spent 3 weeks tuning pure dense retrieval before adding BM25. Should have run both from the beginning. The improvement was immediate and cost was trivial.
2. Build the eval harness before the product. We built evals in Week 6. Every prompt or model change before that was measured by "does it feel better?". A bad way to make decisions on AI systems. Eval-driven development is what lets you actually improve over time.

Can we build something similar for you?

Every RAG project we've shipped since VakeelSaathi uses the same fundamentals: chunking that preserves citation ability, hybrid retrieval + reranking, strict grounding prompts, evals from day one. The domain changes (contracts, product docs, medical protocols, insurance policies) but the pattern holds.

If you're evaluating a RAG chatbot for your business. Internal knowledge, customer support, document Q&A, or industry-specific research. Ask Deci or use our contact form. We'll show you what a working prototype on your data would look like.