A Comparative Study of On-Device LLM Text Summarization: Evaluating 4-bit, 8-bit, and BFloat16 Quantization of Qwen2.5-0.5B-Instruct on Apple Silicon with the MLX Framework

DOI

10.2991/978-94-6239-828-3_25How to use a DOI?

Keywords

Text Summarization; Large Language Models; Quantization Apple Silicon; MLX Framework; On-Device Inference

Abstract

This paper evaluates on-device abstractive text summarization with a small Large Language Model (LLM) on consumer hardware, focusing on the impact of weight quantization on summary quality and computational efficiency. We conduct a paired comparison of Qwen2.5-0.5B-Instruct under three precision configurations (4-bit, 8-bit, and BFloat16) on the same 50 California bills from the BillSum ca_test split, executed on a 2024 MacBook Pro with an Apple M4 Pro system-on- chip, 24 GB unified memory, and Metal acceleration through the MLX framework. Quality is assessed with ROUGE, BLEU, and BERTScore against the human-written references, and efficiency through throughput (summaries per second) and peak MLX memory (GB). We find no statistically distinguishable difference in summary quality across the three configurations at this sample size: ROUGE-1 spans 0.350 to 0.355 and BERTScore spans 0.841 to 0.847 (absolute spread 0.006). System metrics, in contrast, are monotonic: 4-bit quantization is 52% faster (1.34 vs. 0.88 summaries/s) and uses 26% less peak memory (1.23 GB vs.1.66 GB) than the BFloat16 baseline. Together, these results indicate that 4-bit quantization is a Pareto improvement within the resolution of this evaluation and a reasonable default for privacy-preserving, offline summarization with this model and framework on Apple Silicon.

Copyright

© 2026 The Author(s)

Open Access

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

TY  - CONF
AU  - Nanda Rosyadi
AU  - Benben Bagus
AU  - S. Pradiptya Kahvi
AU  - Lailil Muflikhah
AU  - Tirana Noor Fatyanosa
PY  - 2026
DA  - 2026/11/04
TI  - A Comparative Study of On-Device LLM Text Summarization: Evaluating 4-bit, 8-bit, and BFloat16 Quantization of Qwen2.5-0.5B-Instruct on Apple Silicon with the MLX Framework
BT  - Proceedings of the 2026 11th International Conference on Sustainable Information Engineering and Technology (SIET 2026)
PB  - Atlantis Press
SP  - 281
EP  - 291
SN  - 2589-4900
UR  - https://doi.org/10.2991/978-94-6239-828-3_25
DO  - 10.2991/978-94-6239-828-3_25
ID  - Rosyadi2026
ER  -