Consumers who perform similar tasks will form a group known as the Consumer Group. They can also read the data either at the topic or partition levels. These producers can write either on the topic or partition levels.Ĭonsumers read or consume the data from the topics using the Consumer APIs.
Producers write or publish the data in the topics using the Producing APIs. Other than this, there are Producers and Consumers in the system. Each record of these partitions is associated with a unique identifier termed Offset, which denotes the position of the record in that partition. The topics are further classified into Partitions. We cannot create two topics with the same name. A topic can be uniquely identified by its name. A topic is a group that contains the messages of a particular category. Kafka stores the messages in different Topics. Kafka architecture consists of several key components:įig.3 Kafka Architecture | Image by ibm-cloud-architecture Depending on the user's configuration, this time limit can be a day, a week, or a month.įig.2 Publisher-Subscriber Message System | Image by Author The consumer has to consume the message before a certain amount of time, as it gets deleted after that, even if it didn’t get read. In this, multiple senders and receivers can read or write data simultaneously. In the Publisher-Subscriber model, the sender is termed a publisher, and the receiver is termed a subscriber. There is no time dependency laid for the receiver to read the message.įig.1 Point-to-Point Message System | Image by Author Also, the data gets deleted once it gets read, and only a single receiver is allowed at a time. In point-to-point, the sender pushes the data into the queue, and the receiver pops from it like a standard queue system following FIFO(first in, first out) principle. Point-to-Point and Publisher-Subscriber are two widespread message-passing systems. It is currently open-source and used by many organisations like Netflix, Walmart and Linkedin.Ī Message Passing System makes several applications send or receive data from each other without worrying about data transmission and sharing. Due to these benefits, it can be suitable for making real-time data processing applications and streaming services. It can handle and store large volumes of data with minimal latency and high throughput. Kafka was created to overcome the problem faced by the distribution and scalability of traditional message-passing systems. It is developed by Apache Software Foundation and written in Java and Scala. Apache Kafka is a distributed message-passing system that works on a publisher-subscriber model.
0 kommentar(er)
