Big Data Event Streaming with Apache Kafka for Improved Data Flows in IoT using Optimized Kafka-Based Data Streaming Workflow

DOI

10.2991/978-94-6239-654-8_24How to use a DOI?

Keywords

Apache Kafka; Data Streaming; Real-Time Analytics; Distributed Architecture; Message Queuing; Scalable Data Pipelines; Event Stream Throughput; Low-Latency Processing; Big Data Infrastructure

Abstract

The distributed architecture and message queuing features of Apache Kafka significantly improve the reliability and efficiency of batch and real-time data processing. This research aims to create a scalable and dependable data streaming setup by optimizing Kafka deployments, data splitting, and Kafka Connect integration. The study focuses on enhancing data processing, input, and distribution across applications and systems. The project seeks to achieve substantial improvements in data processing speed, real-time analytics, and scalable data pipelines by fine-tuning Kafka settings and using its features. The analysis of Kafka Event Stream Throughput Over Time and Latency Distribution Across Brokers demonstrates the system's performance and efficiency. The results show a latency distribution of 6-15 milliseconds and a throughput of 750-1340 events per hour. Additionally, the Consumer Lag Over Time analysis indicates consistent performance, with values ranging from 70 to 140. This study highlights the effectiveness of Apache Kafka in creating a reliable and efficient data streaming infrastructure, which advances big data processing. The findings provide valuable insights for businesses aiming to make the most of real-time analytics and improve their data pipelines.

Copyright

© 2026 The Author(s)

Open Access

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

TY  - CONF
AU  - N. Javed
AU  - R. Yogesh Rajumar
PY  - 2026
DA  - 2026/04/24
TI  - Big Data Event Streaming with Apache Kafka for Improved Data Flows in IoT using Optimized Kafka-Based Data Streaming Workflow
BT  - Proceedings of the Global Conference on Sustainable Energy Systems, Smart Electronics and Intelligent Computing (GCSESEIC 2025)
PB  - Atlantis Press
SP  - 277
EP  - 287
SN  - 2352-5401
UR  - https://doi.org/10.2991/978-94-6239-654-8_24
DO  - 10.2991/978-94-6239-654-8_24
ID  - Javed2026
ER  -