Proceedings of the Recent Advances in Artificial Intelligence for Sustainable Development (RAISD 2025)

AI and Optimization: Transforming Data Engineering Applications

Authors
Anil Kumar Jonnalagadda1, *, Kailash Pati Dutta2, Piyush Ranjan3, Praveen Kumar Myakala4
1Independent Researcher, Frisco, USA
2Department of Computer Science Engineering & Information Technology, Jharkhand Rai University, Ranchi, Jharkhand, 834010, India
3Independent Researchers, Edison, USA
4Independent Researcher, Melissa, Austin, USA
*Corresponding author. Email: anil.j78@gmail.com
Corresponding Author
Anil Kumar Jonnalagadda
Available Online 17 July 2025.
DOI
10.2991/978-94-6463-787-8_52How to use a DOI?
Keywords
AI-Driven Optimization; Data Engineering; Machine Learning Algorithms; Data Pipelines; Evolutionary Optimization; Reinforcement Learning; Cloud-Based Systems; Real-Time Analytics; Resource Utilization; Big Data
Abstract

The exponential growth of data in modern systems presents significant challenges for data engineering, requiring innovative approaches to optimize data ingestion, transformation, storage, and quality assurance. This article explores how AI-driven optimization techniques are transforming data engineering by automating complex workflows, reducing inefficiencies, and enabling intelligent decision-making. We present a comprehensive framework that combines machine learning algorithms, evolutionary optimization, and neural networks to tackle critical challenges such as reducing data pipeline latency, minimizing resource costs, and maximizing data throughput. For example, reinforcement learning dynamically optimizes data pipeline configurations, while evolutionary algorithms enhance resource allocation and load balancing in cloud-based distributed systems. Case studies demonstrate the framework’s significant impact, including a 40% reduction in data processing time, a 30% improvement in resource utilization, a 20% increase in data processing throughput, and a 15% reduction in data storage costs for big data and real-time analytics workflows. By bridging theoretical advancements in AI/ML with the practical demands of data engineering, this research highlights the transformative potential of AI-powered optimization algorithms. It provides a scalable and adaptive foundation for managing massive datasets, empowering organizations to harness the value of their data in a rapidly evolving digital landscape.

Copyright
© 2025 The Author(s)
Open Access
Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

Volume Title
Proceedings of the Recent Advances in Artificial Intelligence for Sustainable Development (RAISD 2025)
Series
Advances in Intelligent Systems Research
Publication Date
17 July 2025
ISBN
978-94-6463-787-8
ISSN
1951-6851
DOI
10.2991/978-94-6463-787-8_52How to use a DOI?
Copyright
© 2025 The Author(s)
Open Access
Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Cite this article

TY  - CONF
AU  - Anil Kumar Jonnalagadda
AU  - Kailash Pati Dutta
AU  - Piyush Ranjan
AU  - Praveen Kumar Myakala
PY  - 2025
DA  - 2025/07/17
TI  - AI and Optimization: Transforming Data Engineering Applications
BT  - Proceedings of the Recent Advances in Artificial Intelligence for Sustainable Development (RAISD 2025)
PB  - Atlantis Press
SP  - 686
EP  - 702
SN  - 1951-6851
UR  - https://doi.org/10.2991/978-94-6463-787-8_52
DO  - 10.2991/978-94-6463-787-8_52
ID  - Jonnalagadda2025
ER  -