Generative AI for Enterprise Data Pipeline Automation

DOI

10.2991/978-94-6463-978-0_59How to use a DOI?

Keywords

Large Language Models; Data Pipeline Automation; Transformer Architecture; Intelligent Validation Systems; Metadata Management

Abstract

Enterprise data management has advanced to a level of complexity that traditional Extract, Transform, and Load (ETL) approaches can no longer cope with. Data pipelines are often manually documented, consuming extensive engineering time and frequently lagging months behind actual deployments, creating gaps in transparency and governance. Recent strides in large language models (LLMs) point toward a new paradigm in which data operations can be automated through contextual understanding and adaptive reasoning. This paper presents the Integrated Generative Pipeline Management (IGPM) Framework, which automates documentation, intelligent data validation, anomaly detection, and metadata enhancement using LLMs. Transformer-based architectures, event-driven messaging, and containerized microservices provide scalable, modular automation across enterprise pipelines. Key discussions focus on deployment strategies, continual learning processes, and governance frameworks addressing privacy, security, and ethical considerations such as differential privacy and constitutional AI. The paper concludes with directions for future research in multimodal integration and federated learning, enabling collaborative model improvement while maintaining information sovereignty and computational efficiency.

Copyright

© 2025 The Author(s)

Open Access

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

TY  - CONF
AU  - Yaman Tandon
PY  - 2025
DA  - 2025/12/31
TI  - Generative AI for Enterprise Data Pipeline Automation
BT  - Proceedings of the 1st Engineering Data Analytics and Management Conference (EAMCON 2025)
PB  - Atlantis Press
SP  - 697
EP  - 715
SN  - 2352-5401
UR  - https://doi.org/10.2991/978-94-6463-978-0_59
DO  - 10.2991/978-94-6463-978-0_59
ID  - Tandon2025
ER  -