The problem: legal research shouldn't take four hours
India's legal ecosystem runs on judgments. Every argument in court, every contract dispute, every advisory memo starts with a question: what did courts say about this before? Answering that question the traditional way means opening Indian Kanoon or Manupatra, running keyword searches, opening 20 tabs, reading through irrelevant hits, and finally landing on the two or three cases that matter — after 3 to 6 hours of research.
When we started building VakeelSaathi (Decipher's legal operations SaaS), the founding hypothesis was simple: a lawyer should be able to ask "what have courts held about arbitral awards being challenged under Section 34?" and get a cited answer in 30 seconds.
Building it was harder than the hypothesis suggested.
Ask ChatGPT the same question and it will confidently cite Union of India v. Some Case, (2018) 4 SCC 123 — a citation it just made up. This is the hallucination problem. In legal work, a fabricated citation isn't a bug; it's malpractice. We needed a system where the model could only quote from real, verifiable judgments.
The architecture
VakeelSaathi's legal Q&A runs on a RAG pipeline. Retrieval-Augmented Generation is straightforward in theory — fetch relevant documents, give them to an LLM as context, let the LLM answer using only what you gave it. In practice, at 50,000+ documents and Indian legal complexity, every step required careful engineering.
text-embedding-3-large for semantic embeddings, stored in Pinecone (managed) with metadata filters for court, year, and legal domain. Sparse BM25 index alongside for keyword-heavy queries (statute numbers, section references).Model selection: why Claude beat GPT-4 on Indian legal reasoning
Before locking in Claude, we ran evaluations on 500 curated Indian legal questions across GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and Llama 3.1 70B (self-hosted). Grading criteria: factual accuracy, correct citation format, sensitivity to jurisdiction (SC vs High Court binding force), and refusal quality on out-of-scope queries.
Results:
- Claude 3.5 Sonnet: 89% accuracy, near-perfect citation format, best refusal behavior
- GPT-4o: 84% accuracy, occasional citation fabrications, slightly weaker on Indian statute interpretation
- Gemini 1.5 Pro: 79% accuracy, verbose answers, unreliable citation format
- Llama 3.1 70B (self-hosted): 71% accuracy, cheapest to run, best for high-volume simple queries
Production runs Claude for user-facing legal Q&A. Llama 3.1 handles internal batch processing (bulk case summarization, catchword extraction). This split cuts our monthly API bill by roughly 60% versus running Claude everywhere.
How we killed hallucinations
"Reduce hallucinations" is what every AI vendor claims. We measured ours. Here's what worked:
Layer 1: Retrieval quality
Most hallucinations trace back to bad retrieval — the model got weak context, so it filled the gap. Reranking with Cohere improved top-3 retrieval precision from ~62% to ~91% on our eval set. This single change removed the majority of hallucination sources.
Layer 2: Strict grounding prompts
The system prompt is 800+ words. Key rules:
- Every factual claim must cite a specific paragraph from the retrieved chunks
- If no chunk supports a claim, the model must say "I don't have direct authority on this"
- Never invent case names, citations, dates, or holdings
- Distinguish binding precedent (Supreme Court, applicable High Court) from persuasive precedent (other High Courts, tribunals)
Layer 3: Confidence thresholds
Below a certain retrieval-similarity threshold, the system doesn't attempt an answer. It surfaces the top matches it found and lets the user decide. In legal, "I don't know" is a valid and expected answer — pretending otherwise is dangerous.
Layer 4: Nightly evals
500 curated questions run through the system every night. Any regression on accuracy or citation quality alerts our on-call. This is standard software practice applied to AI — a habit far too few teams building AI products actually maintain.
On our 500-question eval, hallucination rate dropped from an initial 14% (unstructured RAG) to under 1% (production system). The remaining ~1% is almost entirely "refusal when it should have answered" — the safe failure mode.
Latency work: getting to sub-500ms retrieval
Initial retrieval was around 1.8 seconds — too slow for a conversational UI. Four optimizations got us to 480ms average:
- Parallel dense + sparse retrieval — instead of running them serially, we fire both simultaneously and merge when both return. Saved ~300ms.
- Cached embedding model — for the query embedding step, we moved from an OpenAI API call to a locally-hosted sentence-transformer for the initial embedding, calling the API only for reranking. Saved ~250ms per query and reduced cost.
- Pinecone region optimization — moved index to
ap-south-1(Mumbai) since our users are in India. Network latency halved. - Reranker batch size tuning — Cohere's rerank endpoint has non-linear latency with batch size. Tuning from 50 to 30 candidates cut this step 40%.
Full user-facing latency (including LLM generation) is 2–4 seconds for a cited answer. We stream tokens so users see the answer forming immediately.
Multilingual: 8 Indian languages, one system
VakeelSaathi supports legal drafting in Hindi, Marathi, Tamil, Bengali, Telugu, Kannada, Malayalam, and Gujarati. English is the primary language for legal research (since Indian case law is in English); regional languages are used for drafting notices, agreements, and client-facing documents.
The setup:
- Retrieval always runs against the English case-law index
- Generation uses Claude 3.5 for English drafts
- Language-specific fine-tuned Llama 3.1 models handle output in each regional language, seeded with the English draft as context
- Post-generation validation checks legal terminology consistency against a bilingual glossary
The results, in numbers
What this cost — and what it costs to run
- Prototype phase (3 weeks): ~₹2.5 lakh — 5,000-judgment prototype, model comparison, first UI
- Production build (10 weeks): ~₹15 lakh — full 50k index, ingestion pipeline, multilingual, WhatsApp bot, evals, security review, deployment
- Ongoing operations: ~₹80,000/month — model API costs (Claude + Cohere + embeddings), Pinecone, monitoring, weekly re-indexing, quality evals
For comparison: hiring a single mid-level lawyer to do the same volume of research work costs ~₹1.2 lakh/month fully loaded, and they can process a small fraction of the queries per day.
What we'd do differently
Two things, in hindsight:
- Start with hybrid retrieval from day one. We spent 3 weeks tuning pure dense retrieval before adding BM25. Should have run both from the beginning — the improvement was immediate and cost was trivial.
- Build the eval harness before the product. We built evals in Week 6. Every prompt or model change before that was measured by "does it feel better?" — a bad way to make decisions on AI systems. Eval-driven development is what lets you actually improve over time.
Can we build something similar for you?
Every RAG project we've shipped since VakeelSaathi uses the same fundamentals: chunking that preserves citation ability, hybrid retrieval + reranking, strict grounding prompts, evals from day one. The domain changes (contracts, product docs, medical protocols, insurance policies) but the pattern holds.
If you're evaluating a RAG chatbot for your business — internal knowledge, customer support, document Q&A, or industry-specific research — book a call. We'll show you what a working prototype on your data would look like.