Sponsor: Do you build complex software systems? See how NServiceBus makes it easier to design, build, and manage software systems that use message queues to achieve loose coupling. Get started for free.
If you’re interested in playing around with Apache Kafka with .NET Core, this post contains everything you need to get started.
I’ve been interested in Kafka for awhile and finally sat down and got everything configured using Docker, then created a .NET console app that contained a Producer and a Consumer.
Here’s my complete process of what that involved.
YouTube
Check out my YouTube channel where I created a video that accompanies this blog post.
Docker
The first thing you need is to pull down the latest Docker images of both Zookeeper and Kafka.
Before we create any contains, first create a new network that both contains are going to use.
Now you can create both Zookeeper and Kafka containers. Kafka needs to communicate with Zookeeper. All the port mappings are the standard ports listed in the Zookeeper and Kafka docs.
Producer
Now that we have Zookeeper and Kafka containers running, I created an empty .net core console app. I’m going to create a hosted service for both Producer and Consumer.
For the producer in this demo, I’m using the Confluent.Kafka NuGet Package.
For this example, I’m going to iterate 100 times and produce a string of “Hello World {i}” that is going to be sent to a topic called “demo”.
In the constructor I’m creating a new producer and specifying that Kafka is running on localhost:9092, which is our container.
The other thing to note is that this “demo” topic I’ve created manually ahead of time, not within this code sample.
Kafka UI
If you need to create a topic, you can do so using the command line tools, or if you prefer something visual you can check out Conduktor.
Consumer
For the consumer, I decided to try the kafka-sharp NuGet Package after I saw this tweet.
The consumer follows a similar type of pattern where in the Constructor I’m creating the connection to localhost:9092. In the StartAsync is where the magic happens of subscribing to the demo topic. The MessageReceived lambda is what will handle all messages received, where I’m just logging the message value.
Full Sample
For completeness, here’s the generic host and both Producer and Consumer hosted services above.
You can find the entire sample on my GitHub.
If you’re using Kafka, which library are you using? Are you using a event/message dispatcher on top of Kafka or using it directly. Let me know in the comments or on Twitter.