AI and Optimization: Transforming Data Engineering Applications

DOI

10.2991/978-94-6463-787-8_52How to use a DOI?

Keywords

AI-Driven Optimization; Data Engineering; Machine Learning Algorithms; Data Pipelines; Evolutionary Optimization; Reinforcement Learning; Cloud-Based Systems; Real-Time Analytics; Resource Utilization; Big Data

Abstract

The exponential growth of data in modern systems presents significant challenges for data engineering, requiring innovative approaches to optimize data ingestion, transformation, storage, and quality assurance. This article explores how AI-driven optimization techniques are transforming data engineering by automating complex workflows, reducing inefficiencies, and enabling intelligent decision-making. We present a comprehensive framework that combines machine learning algorithms, evolutionary optimization, and neural networks to tackle critical challenges such as reducing data pipeline latency, minimizing resource costs, and maximizing data throughput. For example, reinforcement learning dynamically optimizes data pipeline configurations, while evolutionary algorithms enhance resource allocation and load balancing in cloud-based distributed systems. Case studies demonstrate the framework’s significant impact, including a 40% reduction in data processing time, a 30% improvement in resource utilization, a 20% increase in data processing throughput, and a 15% reduction in data storage costs for big data and real-time analytics workflows. By bridging theoretical advancements in AI/ML with the practical demands of data engineering, this research highlights the transformative potential of AI-powered optimization algorithms. It provides a scalable and adaptive foundation for managing massive datasets, empowering organizations to harness the value of their data in a rapidly evolving digital landscape.

Copyright

© 2025 The Author(s)

Open Access

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

TY  - CONF
AU  - Anil Kumar Jonnalagadda
AU  - Kailash Pati Dutta
AU  - Piyush Ranjan
AU  - Praveen Kumar Myakala
PY  - 2025
DA  - 2025/07/17
TI  - AI and Optimization: Transforming Data Engineering Applications
BT  - Proceedings of the Recent Advances in Artificial Intelligence for Sustainable Development (RAISD 2025)
PB  - Atlantis Press
SP  - 686
EP  - 702
SN  - 1951-6851
UR  - https://doi.org/10.2991/978-94-6463-787-8_52
DO  - 10.2991/978-94-6463-787-8_52
ID  - Jonnalagadda2025
ER  -