How to Create Managed Kafka Cluster in the Cloud for Free
This post explains how to setup managed Kafka cluster in the cloud using confluent cloud platform.
In my previous blog posts (linked below) , I explained about setting up Kafka cluster in local development machine using Kafka binary and Docker
Kafka in the Cloud
confluent provides fully managed Kafka cluster in the cloud. You try the kafka services by registering a free account. You will receive $400 credit for 60 days to tryout services. You can use this option to test your application
Note : You can also use the promo code SPRING101 to get additional $101 of free Confluent Cloud usage. Offer ends on 31st Dec 2022.
After registering, Once you login into confluent , you can see the button to create cluster
Click on Create Cluster button. You will get following screen. Select Basic cluster option. You can also select other options but you will incur charges which will be deducted from your free credit amount. Next click on Begin configuration button
In next screen, select your preferred cloud provider and region. You can go with default options and click on Continue button.
In next screen you can see cluster name ( which you can change), usage charges and cluster configuration details.
Click on the Launch `Cluster` button to start the cluster.
If cluster is successfully launched , You will see below confirmation screen. Clicking on the cluster name ( in breadcrumb) on top left corner takes you to cluster dashbaord.
On the cluster dashboard screen, You can see option in the left side navigation menu. Click on the Topic option to create a topic.
Next click on Create Topic button
In next screen provide the Topic name and no. of partitions . If you want to create topic with default option, You can click on the Create with defaults button
If you want to customize the advanced options, You can click on the link and select different settings. Click on Save and Create button to create a Topic.
In the next screen you can see the confirmation of Topic creation.
Once Topic is created , we can connect to cluster and send messages/events to Topic using Confluent CLI.
To connect Kafka cluster in confluent cloud , first we need to create a API key and secret first.
Go to the cluster dashboard page. Click on the DataIntegration link on left side navigation.
Under Data Integration menu , click on the API Keys sub menu.
In the next screen, click on the Create Key button
In next screen select Global Access option , click on next button.
Next screen generates key and secret either copy them or click on Download and Continue button to download the api key file.
1. Install / Update the Confluent CLI
curl -L --http1.1 https://cnfl.io/cli | sh -s -- -b /usr/local/bin
Note
If you are using Windows OS, you need to install WSL (Windows Subsystem for Linux) to run sh and curl commands.
If are running these command as non root user, run commands with su
2. Log in to your Confluent Cloud organization using the Confluent CLI
confluent login --save
When prompted for your username and password, enter the same credentials that you used to log in to Confluent Cloud.The optional –save flag saves your login credentials to a local file for future use.
3. View available environments
confluent environment list
4. Select your environment
confluent environment use env-z29gy
5. List available clusters
confluent environment list
6.Select your cluster
confluent kafka cluster use lkc-pgkr12
7. Use an API key and secret in the CLI
An API key is required to produce or consume to your topic.We will add already produced to CLI with following command.
confluent api-key store --resource lkc-pgkr12 Key: <API_KEY> Secret: <API_SECRET>
Assign API key to use the resource
8.List all topics available in the cluster
confluent kafka topic list
9. If there are no topics, you can create one with following command
confluent kafka topic create sample-topic
10. Produce messages to your topic
confluent kafka topic produce sample-kafka-topic
Run that command, then produce a few messages to your cluster in Confluent Cloud using the console producer. You can do this by typing anything into the console and pressing Enter
11. Consume the messages you produced to your topic
confluent kafka topic consume -b sample-kafka-topi
Confluent cloud provides free Kafka cluster for up to 2 months. If you are looking for free Kafka cluster for life time.
you can look at the below blog post. CloudKarafka’s free Kafka cluster has limited resources but no time limit.
References