Developing and Validating Diagnostic Instruments for Measuring AI Literacy and Problem-Solving in Higher Education

DOI

10.2991/978-2-38476-543-0_15How to use a DOI?

Keywords

AI literacy; Diagnostic Assessment; Problem Solving; Rasch Modeling; Validation

Abstract

This study aimed to validate diagnostic assessment instruments designed to measure university students’ problem-solving ability and artificial intelligence (AI) literacy. The instruments were developed as part of an integrated web-based e-diagnostic system intended to strengthen students’ higher-order thinking and AI literacy in higher education. A total of 550 students from Universitas PGRI Semarang participated in the validation process. Two main instruments were evaluated: (1) a problem-solving essay test scored using a rubric with a 0–4 scale, and (2) an AI literacy test and questionnaire consisting of five dimensions understanding, application, effect, interpretation, and AI-based problem solving. The inter-rater reliability of the problem-solving instrument was assessed using the Intraclass Correlation Coefficient (ICC), while the AI literacy instruments were analyzed using Rasch modeling. The ICC values were above 0.70, indicating strong agreement between raters. The Rasch analysis showed that all items met acceptable validity criteria, with infit and outfit mean square (MNSQ) values ranging from 0.80 to 1.28. These results confirm that both instruments are valid and reliable for assessing problem-solving ability and AI literacy in higher education contexts.

Copyright

© 2026 The Author(s)

Open Access

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

TY  - CONF
AU  - Ipah Budi Minarti
AU  - Azizul Ghofar Candra Wicaksono
AU  - Atip Nurwahyunani
AU  - Rivanna Citraning Rahmawati
AU  - Eko Retno Mulyaningrum
PY  - 2026
DA  - 2026/02/26
TI  - Developing and Validating Diagnostic Instruments for Measuring AI Literacy and Problem-Solving in Higher Education
BT  - Proceedings of the 8th International Conference on Education and Social Science Research (ICESRE 2025)
PB  - Atlantis Press
SP  - 149
EP  - 157
SN  - 2352-5398
UR  - https://doi.org/10.2991/978-2-38476-543-0_15
DO  - 10.2991/978-2-38476-543-0_15
ID  - Minarti2026
ER  -