Research paper · arXiv:2610.22526 · cs.CL

Literal Lock-In: How Test Language Shifts Literal Attraction in Typed Decision Models on Bengali Idioms

Adib Sakhawat^* Software Engineering Graduate adibsakhawat@iut-dhaka.edu

Syed Rifat Raiyan^* Lecturer, Dept. of CSE rifatraiyan@iut-dhaka.edu

Systems and Software Lab (SSL), Department of Computer Science and Engineering
Islamic University of Technology, Dhaka, Bangladesh

* Equal contribution

Paper Code & data Dataset BibTeX

Figure 1 · The model takes the words

অগ্নিবলagni-bal

অগ্নি (fire) + বল (strength) → means intense hunger

Laya-ml

literal · p = 1.00

Lod-lille

literal · p = 0.97

Jev-1.13

literal · p = 0.36

Laya-ml · literal attraction

0.731

Lod-lille · literal attraction

0.918

Jev-1.13 · literal attraction

0.304

With both readings in the option list, all three models select the literal gloss in Bengali, at p = 1.00, 0.97 and 0.36. Asked as the paired English entry, with the English glosses of the same four options, Lod-lille and Jev-1.13 recover and Laya-ml does not. Across the 3,265 paired items literal attraction falls from 0.731, 0.918 and 0.304 in Bengali to 0.379, 0.241 and 0.047 in English.

Abstract

Decision models, non-generative scorers that report a probability over caller-supplied options and emit no text, are increasingly deployed as routers, classifiers and guardrails. Because they generate nothing, rationales and chain-of-thought do not apply. We audit three: an encoder scorer, a decoder scorer and a closed API, entirely through interventions on their input, using expert-annotated Bengali idioms carrying a gold figurative and a gold literal gloss. Dropping an idiom’s own literal gloss into the option list draws every arm toward it, taking 65% to 96% of each arm’s remaining Bengali errors. A model × language matrix over one shared item set shows how far this depends on the test language: every arm is 1.9 to 6.4 times more captured in Bengali than in English on the same paired entries, the commercial model’s apparent resistance is a property of its test language, and capability does not order the failure. Calibration degrades in all three arms, and on the open arms confidence inverts, so gating on it is worse than gating at random. On a semantics-free option list, where accuracy is at chance, the best-calibrated arm still reports mean confidence 0.82. Code and data are available at github.com/sakhadib/literal-lock-in.

01 · Capture

65–96%

of each arm’s remaining Bengali errors are taken by the idiom’s own literal gloss.

02 · Language

1.9–6.4×

more literal capture in Bengali than in English on the same paired entries, in every arm.

03 · Confidence

< random

On the open arms confidence inverts, so gating on it is worse than gating at random.

04 · Control

0.82

mean confidence from the best-calibrated arm on a semantics-free list, where accuracy is at chance.

§ 3–4 · Dataset and method

The option list is the only intervention surface

A decision model returns a probability distribution over exactly the options the caller supplies. It produces no text and, in the commercial case, exposes no weights. Bangla Bagdhara ships, for each of its 10,361 expert-annotated idioms, both a gold figurative meaning and a gold literal gloss: a ready-made, human-certified counterfactual distractor. Given an idiom, the model chooses its figurative meaning from k = 4 candidates; the distractor family changes and everything else is held fixed.

Figure 2 · The instrument in one view

01 Corpus Bangla Bagdhara: 10,361 expert-annotated idioms with gold figurative and literal glosses.
02 Distractor family The variable E1 manipulates. D-LIT and D-HARD inject the expert literal gloss. independent variable
03 Item prompt Fixed typed question, k = 4 options, instruction and per-item order seed.
04 Models One typed question to three decision models. Laya-mlLod-lilleJev-1.13
05 Scoring Four-option probability vector, arg max and confidence. No generated text.
06 Metrics Accuracy, LAR, ECE, AURC, agreement and order stability.

Five distractor families

One distractor replaced by the idiom’s own expert literal gloss.

Option list for one item · k = 4, chance 0.250

A Random idiom’s gloss random
B Gold figurative meaning gold
C Random idiom’s gloss random
D The idiom’s own literal gloss literal

Literal Attraction Rate

LAR = \frac{1}{N} \sum_{i = 1}^{N} 𝟙 [\underset{o}{arg max} p_{i} (o) = ℓ_{i}]

LAR is not 1 − accuracy. It isolates the errors that are specifically a literal reading, and it has no floor: a model that never reads literally scores 0 while still being wrong.

Seven interventions, one variable each

E1The distractor family
E2The idiom’s surface form
E3The idiom as a unit
E4The task language
E5Nothing (derived from E1)
E6The option text
E7The option order

Paired McNemar throughout, with 95% percentile bootstrap intervals over items (2,000 resamples).

Three model arms

Property	Laya-ml	Lod-lille	Jev-1.13
backbone	mmBERT	Qwen3-0.6B	closed
scoring	marker	independent	unknown
parameters	322M	0.6B	n/a
items	8,876	888	8,761
latency	27.2 ms	86.1 ms	API
weights	open	open	closed

Laya-ml is a bidirectional encoder scoring all options together from marker positions. Lod-lille is a decoder with independent-option scoring. Jev-1.13 is a closed commercial model behind a decisions API.

§ 5.1 · Literal capture

The stronger model leads until the literal gloss appears

Lod-lille is the stronger model on every non-literal condition, by up to +0.133. Insert the idiom’s own literal gloss and the ordering reverses: Lod-lille falls to 0.066, half of Laya-ml’s already-below-chance 0.136. A model can be selected as “better” on a conventional benchmark while being worse at what that benchmark proxies for.

Table 1 · Accuracy by distractor family Laya-mlLod-lillechance

0.351

0.484

random

0.339

0.412

surface

0.325

0.446

tag

0.136

0.066

+ literal

0.149

0.063

hard

The 888 items both open arms share, paired McNemar p < 0.001 on every row. Shaded families contain the idiom’s own literal gloss, and the sign of the difference reverses at that boundary.

§ 5.1 · Where the errors go, D-LIT

Laya-ml

0.136

0.658

0.206

Lod-lille

0.885

goldliteralother

Gold / literal / other on the same 888 items: Laya-ml 0.136 / 0.658 / 0.206, Lod-lille 0.066 / 0.885 / 0.050. The error is not diffuse; on the clearest items Laya-ml places p = 1.000 on the literal gloss.

§ 5.1 · Model × language matrix

Capture tracks the language, not capability

All six cells are measured on the same 3,265 items, built by one function from one seed. Every arm is far more captured in Bengali than in English (paired McNemar p < 10⁻¹⁰⁰). Lod-lille is more accurate than Laya-ml and more captured; Jev-1.13 is more accurate than both and less captured.

Figure 5 · Literal condition, n = 3,265 per cell

BengaliEnglish

Laya-ml1.9× more captured in BN

BN0.731

EN0.379

Lod-lille3.8× more captured in BN

BN0.918

EN0.241

Jev-1.136.4× more captured in BN

BN0.304

EN0.047

Solid bars Bengali, hatched bars English. Within-model language effect on LAR: +0.352, +0.677 and +0.256. On the 5,485 corpus items without an English parenthetical cue, Bengali capture is still 0.581 and 0.871 for the open arms.

§ 5.2 · Calibration

Confidence is anti-diagnostic

Both open models are more confident when captured than when correct. Restricting Laya-ml’s D-LIT to its most confident 5% gives 0.007 accuracy against 0.151 at full coverage, and Lod-lille answers every item in its most confident 35% incorrectly. Their multiclass Brier scores under D-LIT, 1.030 and 1.391, are worse than the 0.750 of a uniform four-way distribution.

Table 2 · Mean confidence by outcome correctcaptured

Laya-mlmore confident when captured

correct 0.432captured 0.596

Lod-lillemore confident when captured

correct 0.453captured 0.787

Jev-1.13more confident when correct

correct 0.935captured 0.737

Pooled over D-LIT and D-HARD, on an axis from 0.3 to 1.0. Jev-1.13’s condition here is English, and it does not show the inversion.

Table 2 · Selective prediction, AURC under D-LIT confidence gaterandom gate

Laya-mlworse than random

confidence0.924

random gate0.849

Lod-lilleworse than random

confidence0.989

random gate0.934

Jev-1.135.1× better than random

confidence0.013

random gate0.069

Lower is better. On both open arms the confidence-gated AURC is worse than gating at random. Jev-1.13 is well calibrated in English (ECE 0.014) and not on the same items in Bengali (ECE 0.135), where its advantage over a random gate falls from 5.1× to 1.4×.

§ 5.3 · Semantics-free control

Confident on a list that says nothing

The O4 ablation replaces every option’s text with a bare ordinal label, so the four options are distinguishable but carry no meaning. Expected accuracy is 0.25 and whatever confidence survives is overconfidence. The best-calibrated model under real options is the worst here: Jev-1.13 reports 0.815 mean confidence at 0.254 accuracy.

Figure 7 · The semantics-free control

AGold figurative meaning
BRandom idiom’s gloss
CRandom idiom’s gloss
DRandom idiom’s gloss

অর্থ 1–4 for the Bengali arms, meaning 1–4 for the commercial one

Overconfidence · mean confidence − accuracy

Laya-ml0.250

Lod-lille0.049

Jev-1.130.561

Scale 0 to 0.6. Lod-lille, the weakest of the three on accuracy, is the only one whose confidence collapses appropriately (0.282). Jev-1.13’s answer-present field still reports 0.754, against 0.946 with full glosses.

§ 5.4 · Option order

Order sensitivity is architecture-specific

Every item is run under all 4! = 24 orderings of a fixed option set. The marker-based scorer changes its chosen gloss on 83.7% of items against 10.7% for the independent-option scorer and 13.3% for the commercial one. Lod-lille is documented as order-independent by construction; a single option’s own score travels 4.0×10⁻² at the median item. Permuting a list also relabels it, so this measures display order and labeling together.

Schematic · 24 orderings

ordering 1 / 24

Gold figurative

Literal gloss

Distractor 1

Distractor 2

An illustration, not data. Laya-ml’s pick follows slot B, its most-chosen slot (37.2% of the time against 14.6% for slot A); Lod-lille and Jev-1.13 hold their choice. Measured rates are alongside.

Table 21 · Option-order sensitivity

Answer changes across orderings

Laya-ml0.837

Lod-lille0.107

Jev-1.130.133

Perfectly stable items

Laya-ml0.163

Lod-lille0.893

Jev-1.130.867

Within-item range of p(gold)

Laya-ml0.346

Lod-lille0.029

Jev-1.130.128

n = 992 / 299 / 300 items. A 7.8× difference in answer stability and a 12× difference in the within-item range of p(gold).

§ 5.5 · Unit integrity (E3)

Breaking the idiom barely registers; context does

The expression is perturbed while the option list is held fixed, under D-LIT. A model holding the idiom as a stored unit should collapse under shuffling. Neither does. The one manipulation that clearly helps both is embedding the idiom in its gold carrier sentence.

Figure 9 · Accuracy under perturbation · Laya n = 1,778, Lod n = 180 Laya-mlLod-lillechance

intact 0.157 0.072

word shuffle 0.168 0.044

reversed 0.178 0.056

delete 1 word 0.158 0.089

substitute 1 word 0.162 0.056

in carrier sentence 0.237 0.189

Scale 0 to 0.40. Under word-shuffling the open arms return the same option as on the intact idiom for 0.742 and 0.917 of items. Embedding the idiom in its carrier sentence: +0.080 and +0.117.

Qualitative · the clearest captures

p = 1.000 on the literal, 0.000 on the gold

কব্জা

Literal · chosen1.000

কব্জি বা হাতের আঁচড়ানো অংশ

Gold · rejected0.000

অবাঞ্ছিত অধিকার, প্রভাব

কাঠ

Literal · chosen1.000

কাঠ (wood)

Gold · rejected0.000

কঙ্কাল

চামচা

Literal · chosen1.000

চামচ (spoon)

Gold · rejected0.000

খোসামুদে; চাটুকার; তোষামুদে

গ্যাঁড়াকল

Literal · chosen1.000

গ্যাঁড়া (স্ক্রু বা ফাঁদের যন্ত্র) এবং কল

Gold · rejected0.000

লোক ঠকাবার কৌশল; বিপদ, ফাঁদ

Laya-ml, D-LIT, full Bengali run. Glosses are the corpus’s own.

Interactive figures · per-arm analysis outputs

Explore the results

Hover any mark for its value and interval. All intervals are 95% percentile bootstrap over items, and chance is 0.250 throughout. Solid bars are Bengali input, hatched bars English, grey bars a reference condition.

Model × language · n = 3,265 per cell Laya-mlLod-lilleJev-1.13BengaliEnglish

Accuracy against literal capture, Bengali → English

	Bengali			English
	acc.	LAR		acc.	LAR
Laya-ml	0.127	0.731	→	0.417	0.379
Lod-lille	0.044	0.918	→	0.684	0.241
Jev-1.13	0.536	0.304	→	0.926	0.047

Each arm moves from its Bengali cell (filled) to its English cell (hollow) on the same paired items. Capture is not ordered by accuracy: in Bengali, Lod-lille is more accurate than Laya-ml and more captured; Jev-1.13 is the most accurate and the least captured.

Figure 3 · E1 distractor ladder, full sets

Accuracy by distractor family

Laya-ml n = 8,876, Lod-lille n = 888. The shaded families contain the idiom’s own literal gloss, and both arms fall far below chance there.

Figure 8 · The matched three-way condition

The capability ordering capture does not follow

Bengali idiom with random distractors, the one condition identical across all three arms. Jev-1.13 is far ahead, 0.736 against 0.367 and 0.484, yet it is the least captured arm.

Table 3 · Construction cues

Bengali LAR under D-LIT, split by lexical cue

Verbatim overlap and the Latin-script cue each buy some capture. With both absent (n = 2,480 / 266), LAR stays at 0.560 and 0.891.

Figure 6 · E4 failure localization

Accuracy by task language and checkpoint

Random distractors. Every model gains between 23 and 38 points when the paired English entry replaces the Bengali idiom. Laya-ml’s English-only checkpoint on Bengali script, the control, sits at 0.283.

E4 · Paired items

Where each arm is right in only one language

Share of items answered correctly in one language and wrongly in the other. The asymmetry runs toward English in both open arms.

§ 5.2 · Brier score

Worse than declining to choose

Multiclass Brier under the literal condition. A uniform 0.25 on all four options scores exactly 0.750 (dashed), and both open arms land above it.

Table 2 · ECE, 15 equal-width bins

Calibration degrades under the literal shift

Grey bars are random distractors, coloured bars the D-LIT condition. Jev-1.13’s condition is English.

Figure 10 · E6 option-side ablation

Accuracy as the option text is degraded

Random distractors, the two open arms. Scrambling the words inside every option costs little; removing semantics drops both arms to chance (McNemar between them p = 0.76).

§ 5.4–5.7 · What else the instrument shows

§ 5.5 · Option side

Options are read lexically

Scrambling the words inside every option costs almost nothing: the three arms retain 0.928, 0.944 and 1.005 of their above-chance signal.

§ 5.4 · Position

A 3.3× swing from where the gold sits

With the gold placed uniformly, Laya-ml’s accuracy is 0.066 when the gold sits in slot A against 0.215 in slot B. Lod-lille’s slot shares are near-uniform: 0.265 / 0.258 / 0.242 / 0.235.

E2 · Surface invariance

Synonymous forms, different answers

On expert-certified alternative surface forms with the same gold meaning, answer agreement is only 0.657 (Laya-ml) and 0.677 (Lod-lille). About a third of the time, the answer changes.

§ 5.6 · Construction cues

Lexical cues are real; neither carries the result

On the 2,480 items with no verbatim overlap and no Latin-script cue, LAR is still 0.560 and 0.891 against a chance level of 0.250.

§ 5.7 · Localization

Failure localizes to Bengali

With random distractors, switching to English lifts every arm: Laya-ml +0.246, Lod-lille +0.380, Jev-1.13 +0.230. Items right in English and wrong in Bengali: 37.4% and 43.3%; the reverse is 12.8% and 5.3%.

Table 6 · One item exactly as received by the models

ওজন ojon · literally “physical weight” · English equivalent carry weight

	option text, as supplied	role	Laya-ml p	Jev-1.13 p
A	একগর্ভা; এক সন্তানের জননী; যে নারী একবার মাত্র গর্ভ ধারণ করে	distractor	0.00	0.02
B	ওজন (physical weight)	literal	0.99	0.15
C	ক্ষমতা, গুরুত্ব, মর্যাদা	gold	0.00	0.82
D	হালকাভাবে পড়া	distractor	0.01	0.01

Laya-ml places 0.99 on the literal option, which denotes the physical weight of an object, and 0.00 on the gold figurative gloss; Jev-1.13 places 0.82 on the gold option.

Citation

We release the instrument, the per-item outputs for every arm, and the analysis code that turns them into every number on this page.

GitHub ↗ arXiv:2610.22526 DOI

BibTeX

@misc{sakhawat2026literallockintestlanguage,
  title         = {{Literal Lock-In}: How Test Language Shifts Literal Attraction
                   in Typed Decision Models on {B}engali Idioms},
  author        = {Sakhawat, Adib and Raiyan, Syed Rifat},
  year          = {2026},
  eprint        = {2610.22526},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  doi           = {10.48550/arXiv.2610.22526},
  url           = {https://arxiv.org/abs/2610.22526}
}