Robustness Evaluation of the FLAN-T5 Model Against Prompt Variations in Text Classification Tasks

DOI

10.2991/978-94-6239-828-3_22How to use a DOI?

Keywords

Large language model; Robustness; Prompt engineering; PromptBench; Flan-T5; Natural language processing

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities across various natural language processing tasks; however, their performance is highly influenced by the phrasing of the provided prompts. This study evaluates the robustness of the Flan-T5 model against prompt variations on three text classification datasets: SST-2 (sentiment analysis), MNLI (multi-class natural language inference), and RTE (binary recognizing textual entailment). The variations tested include rewording, style change, noise injection, and adversarial attack, with the original prompt serving as a baseline, using the PromptBench framework with ten trials per prompt at a temperature of 0.3. Results indicate that the best performance was achieved in the original category with a mean accuracy of 0.845 and a mean F1-score of 0.836 across datasets, while the worst performance was exhibited by the noise cate- gory with a mean accuracy of 0.595 and a mean F1-score of 0.676. On the SST-2 dataset, noise reduced the mean accuracy from 0.940 to 0.340, with 63.1% of outputs failing to map to a valid label, as character-level perturbations obscured the target labels within the prompt. Adversarial attacks demonstrated non-uniform impacts: instructions directing the model toward negative labels on SST-2 yielded a mean accuracy of only 0.321, whereas a contradiction-type adversarial attack on MNLI essentially preserved the original-prompt baseline (0.828 versus 0.824), while entailment-type and neutral-type adversarial attacks on the same dataset caused substantially larger drops (0.663 and 0.621, respectively). Re- wording and style change produced moderate, dataset-dependent degradation, with mean accuracies across the three datasets of 0.817 and 0.759, respectively. These findings suggest that Flan-T5 is sufficiently robust against mild linguistic variations but vulnerable to structural perturbations in the instruction, with the degree of vulnerability shaped by task characteristics and, on multi-class tasks, by which classes the model already recognizes least reliably.

Copyright

© 2026 The Author(s)

Open Access

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

TY  - CONF
AU  - Muhammad Arsya Zain Yashifa
AU  - Tirana Noor Fatyanosa
AU  - Putra Pandu Adikara
PY  - 2026
DA  - 2026/11/04
TI  - Robustness Evaluation of the FLAN-T5 Model Against Prompt Variations in Text Classification Tasks
BT  - Proceedings of the 2026 11th International Conference on Sustainable Information Engineering and Technology (SIET 2026)
PB  - Atlantis Press
SP  - 246
EP  - 260
SN  - 2589-4900
UR  - https://doi.org/10.2991/978-94-6239-828-3_22
DO  - 10.2991/978-94-6239-828-3_22
ID  - Yashifa2026
ER  -