FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation

We present FLUKE, a task-agnostic framework for assessing model robustness through systematic minimal variations of test data. FLUKE introduces controlled variations across linguistic levels—from orthography to dialect and style—and leverages large language models with human validation to generate modifications.

We evaluate both fine-tuned models and LLMs across six diverse NLP tasks, revealing that: (1) linguistic variations impact is highly task-dependent; (2) LLMs still exhibit significant brittleness, with reasoning LLMs showing less robustness on some tasks; (3) natural modifications hurt more than corruption-style tests; (4) generation ability doesn't correlate with robustness.

Modification Types

FLUKE applies controlled linguistic modifications that preserve task labels while testing model robustness.

Flukify Your Text

Apply FLUKE modifications using your OpenAI API key. Your key is used directly in-browser and never stored.

Six Tasks, Eight Models, 17 Tests

We evaluate across classification and generation tasks using PLMs, base LLMs, and reasoning LLMs.

Unrobustness by Task

Unrobustness (U%) measures prediction changes between original and modified instances. Higher = more brittle. Select a task to view detailed results.

Color legend: High ≥15%   Medium 10-15%   Low ≤2%

Cite This Work

@article{otmakhova2025fluke,
  title     = {FLUKE: A Linguistically-Driven and Task-Agnostic
               Framework for Robustness Evaluation},
  author    = {Otmakhova, Yulia and Truong, Hung Thinh and
               Mahendra, Rahmad and Zhai, Zenan and Zhu, Rongxin
               and Beck, Daniel and Lau, Jey Han},
  journal   = {arXiv preprint arXiv:2504.17311},
  year      = {2025}
}