AI and Optimization: Transforming Data Engineering Applications
- DOI
- 10.2991/978-94-6463-787-8_52How to use a DOI?
- Keywords
- AI-Driven Optimization; Data Engineering; Machine Learning Algorithms; Data Pipelines; Evolutionary Optimization; Reinforcement Learning; Cloud-Based Systems; Real-Time Analytics; Resource Utilization; Big Data
- Abstract
The exponential growth of data in modern systems presents significant challenges for data engineering, requiring innovative approaches to optimize data ingestion, transformation, storage, and quality assurance. This article explores how AI-driven optimization techniques are transforming data engineering by automating complex workflows, reducing inefficiencies, and enabling intelligent decision-making. We present a comprehensive framework that combines machine learning algorithms, evolutionary optimization, and neural networks to tackle critical challenges such as reducing data pipeline latency, minimizing resource costs, and maximizing data throughput. For example, reinforcement learning dynamically optimizes data pipeline configurations, while evolutionary algorithms enhance resource allocation and load balancing in cloud-based distributed systems. Case studies demonstrate the framework’s significant impact, including a 40% reduction in data processing time, a 30% improvement in resource utilization, a 20% increase in data processing throughput, and a 15% reduction in data storage costs for big data and real-time analytics workflows. By bridging theoretical advancements in AI/ML with the practical demands of data engineering, this research highlights the transformative potential of AI-powered optimization algorithms. It provides a scalable and adaptive foundation for managing massive datasets, empowering organizations to harness the value of their data in a rapidly evolving digital landscape.
- Copyright
- © 2025 The Author(s)
- Open Access
- Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
Cite this article
TY - CONF AU - Anil Kumar Jonnalagadda AU - Kailash Pati Dutta AU - Piyush Ranjan AU - Praveen Kumar Myakala PY - 2025 DA - 2025/07/17 TI - AI and Optimization: Transforming Data Engineering Applications BT - Proceedings of the Recent Advances in Artificial Intelligence for Sustainable Development (RAISD 2025) PB - Atlantis Press SP - 686 EP - 702 SN - 1951-6851 UR - https://doi.org/10.2991/978-94-6463-787-8_52 DO - 10.2991/978-94-6463-787-8_52 ID - Jonnalagadda2025 ER -