Fine-Tuning DistilBERT: A Pre-Trained Transformer Model for Software Defect Prediction

DOI

10.2991/978-94-6239-823-8_2How to use a DOI?

Keywords

Software Defect Prediction; DistilBERT; Transformer Models; Transfer Learning; Fine-Tuning

Abstract

Software Defect Prediction (SDP) is an important software engineering task that aims to identify defective code early in the development lifecycle, thereby improving software quality and reducing maintenance costs. Traditional software defect prediction methods often rely on handcrafted features or shallow representations that may fail to capture the rich semantic and contextual information contained in source code. Recent advances in Natural Language Processing (NLP) have introduced transformer-based models capable of learning complex patterns and long-range dependencies directly from sequential data. However, many existing approaches employ large and computationally expensive models, limiting their practicality in resource-constrained environments. This study proposes a software defect prediction approach based on the supervised fine-tuning of DistilBERT, a lightweight variant of BERT that retains strong representation capabilities while significantly reducing computational requirements. The proposed model was evaluated on the Devign dataset from the CodeXGLUE benchmark using Accuracy, Precision, Recall, and F1-score metrics. Experimental results show that the fine-tuned DistilBERT model achieved an Accuracy of 71.43%, Precision of 69.47%, Recall of 65.44%, and an F1-score of 67.39%. In addition, the proposed approach required substantially less training time than several baseline and existing transformer-based methods evaluated on the same benchmark dataset. The findings demonstrate that lightweight transformer architectures can provide an effective balance between predictive performance and computational efficiency, making DistilBERT a practical solution for software defect prediction in resource-constrained software engineering environments.

Copyright

© 2026 The Author(s)

Open Access

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

TY  - CONF
AU  - Nasraldeen Alnor Adam Khleel
AU  - Károly Nehéz
PY  - 2026
DA  - 2026/11/04
TI  - Fine-Tuning DistilBERT: A Pre-Trained Transformer Model for Software Defect Prediction
BT  - Proceedings of the Innovation in Engineering Education 2026 Conference (IEE 2026)
PB  - Atlantis Press
SP  - 4
EP  - 18
SN  - 2667-128X
UR  - https://doi.org/10.2991/978-94-6239-823-8_2
DO  - 10.2991/978-94-6239-823-8_2
ID  - Khleel2026
ER  -