MeansTo die. The Bengali counterpart of “kick the bucket.”
Model readTo narrowly escape danger, or for something unexpectedly good to happen.
Prompt, translated from Bangla
“Suppose you are an ordinary Bengali-speaking person from Bangladesh. You will be given an idiom/phrase below. You have to tell its meaning. If you see it in a sentence or writing, what meaning will you think of? […]”
LLM response
অক্কা পাওয়া শব্দটির অর্থ হলো কোনো বিপদ বা সমস্যা থেকে অল্পের জন্য রক্ষা পাওয়া বা সুবিধা মিলে যাওয়া।
Score 0 / 5Complete misunderstanding
Example interaction from the paper (Figure 3). The model answers fluently and confidently, and gets the meaning backwards.
Finding 01
No model passes 50%. Native speakers average 83.4%.
Thirty multilingual and instruction-tuned LLMs and fifteen Bangladeshi students answered the same 100 expert-selected idioms. Each answer was scored 0–5 by Bangla linguists in a double-blind review.
Human baseline 15 students, mean 4.17 / 5, SD 0.3983.4%
Best model gemini-2.5-flash, mean 2.38 / 547.6%
Median of 30 models14.6%
Failure is the norm
Even the top model scored 0 on 46 of 100 idioms. More than half of the models failed outright on over 80% of them.
Scale does not help
Llama-3.1-405B, among the largest models tested, ranked 16th with a mean of 0.68. The smaller gemini-2.5-flash leads.
Origin does not predict
Strong and weak models come from US, Chinese and European developers alike. No regional pattern emerges.
Architecture does not rescue
The Mixture-of-Experts Mixtral-8x22B scored 0.02. Research and multimodal variants also failed, pointing to a data and training gap.
Benchmark
All 30 models
{{ caption }}
#Model{{ barLabel }}{{ valLabel }}Perfect
{{ row.rank }}{{ row.name }}
{{ row.val }}{{ row.perfect }}
Human baseline, mean 4.17 / 5 (83.4%)
Scoring rubric
Five Bangla linguistics specialists scored all 3,000 responses double-blind. Zero-shot, identical Bangla prompt for every model.
5Precise and complete figurative meaning
4Correct figurative meaning
3Correct meaning with minor inaccuracies
2Vague or partial figurative grasp
1Literal meaning only
0Complete misunderstanding
The resource
The largest annotated idiom corpus for Bangla
Bangla Bagdhara collects 10,361 idioms from sources spanning three centuries, including Bangla Academy dictionaries, regional folk literature and digital archives. Every entry was reviewed until at least two experts reached full agreement.
9,919
unique semantic tags
37,129
tag assignments, 3.58 per idiom
99.7%
carry cultural significance
1 : 4.6
literary to conversational register
A 19-field schema in four layers
Semantic & cross-lingual
Maps denotation to connotation, with English anchors.
Six researchers, one computer scientist and five Bangla language and linguistics experts, built the corpus over five months. AI drafts were a starting point only; no generated content was accepted without expert revision.
Construction
01
Collection
Bangla Academy dictionaries, regional folk literature, online archives.
02
Filtering & schema
Manual authenticity checks, normalisation to Standard Bangla Unicode.
03
AI-assisted drafts
Gemini-2.5-flash and GPT-4o propose translations and sentiment.
04
Expert annotation
At least two experts refine or replace every field.
05
Consensus validation
Entries marked done only after full agreement in structured deliberation.
Evaluation
01
100 idioms, hand-picked
Common in daily speech but with subtle figurative sense. No random sampling.
02
30 LLMs, zero-shot
4.5B to 405B parameters; dense and MoE; US, Chinese and European developers; open and proprietary.
03
15 native speakers
6 high school and 9 undergraduate students, monitored, no internet or references.
04
Double-blind scoring
Five linguists rate all 3,000 model responses on the 0–5 rubric.
Cite
BibTeX
@inproceedings{sakhawat-etal-2026-words,
title = "When Words Don{'}t Mean What They Say: Figurative Understanding in {B}engali Idioms",
author = "Sakhawat, Adib and
Parveen, Shamim Ara and
Amin, Md Ruhul and
Khatun, Tahera and
Mahmud, Shamim Al and
Islam, Md Saiful",
editor = "Piperidis, Stelios and
Bel, N{\'u}ria and
van den Heuvel, Henk and
Ide, Nancy and
Krek, Simon and
Toral, Antonio",
booktitle = "Proceedings of the Fifteenth Language Resources and Evaluation Conference",
month = may,
year = "2026",
address = "Palma de Mallorca, Spain",
publisher = "ELRA Language Resource Association",
url = "https://aclanthology.org/2026.lrec-1.546/",
doi = "10.63317/546w2cys6m6t",
pages = "6870--6879"
}