AI
IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking
Publication: Hugging Face Daily Papers
Author: Not available in collected metadata
Source published: 23/09/2026, 20:00:00 UTC
Briefing created: 28/09/2026, 10:20:07 UTC
Based on a feed excerpt; full article content has not been acquired. Source evidence retrieved 28/09/2026, 10:18:24 UTC.
Briefings
IndicBankBench evaluates Indian retail banking assistants using 799 cases across five operational domains and twenty axes. The benchmark assesses safety, tool use, response adequacy, and advisory quality through deterministic and semantic checks. Eleven models were tested, with strict reliability ranging from 43.7% to 58.2% and at-least-once success from 60% to 74%. The study reveals that at-least-once success rates can overstate dependable behavior, as narrow resolvers handle only ambiguous cases while separate LLM judges evaluate semantic adequacy.
Key points
- IndicBankBench introduces a 799-case benchmark for Indian retail banking spanning five operational domains and twenty primary axes.
- Strict reliability ranges from 43.7% to 58.2% across eleven evaluated models, while at-least-once success ranges from 60% to 74%.
Context and limitations
Evaluation is limited to four stages including safety, action, response adequacy, and advisory quality. Tool use and most safety checks are deterministic, while a separate LLM judge evaluates semantic response adequacy. Cases are evaluated at four stages, and the benchmark includes a capability/refusal domain.
Generated by Bittrees-hosted qwen35-2b sha256:aaf42c8b7c3c on 28/09/2026, 10:20:07 UTC. Based on publisher-provided feed evidence; may not cover the full original article. Check the source for details.
View archived briefing on IPFS
bafybeicn5isq7c6qugpfwhg42okwhlqeegf25bimadahqvytjx7xm6ytoy