How to Create Managed Kafka Cluster in the Cloud for Free

This post explains how to setup managed Kafka cluster in the cloud using confluent cloud platform.

In my previous blog posts (linked below) , I explained about setting up Kafka cluster in local development machine using Kafka binary and Docker

Kafka in the Cloud

confluent provides fully managed Kafka cluster in the cloud. You try the kafka services by registering a free account. You will receive $400 credit for 60 days to tryout services. You can use this option to test your application

Note : You can also use the promo code SPRING101 to get additional $101 of free Confluent Cloud usage. Offer ends on 31st Dec 2022.

After registering, Once you login into confluent , you can see the button to create cluster

Click on Create Cluster button. You will get following screen. Select Basic cluster option. You can also select other options but you will incur charges which will be deducted from your free credit amount. Next click on Begin configuration button

In next screen, select your preferred cloud provider and region. You can go with default options and click on Continue button.

In next screen you can see cluster name ( which you can change), usage charges and cluster configuration details.

Click on the Launch `Cluster` button to start the cluster.

If cluster is successfully launched , You will see below confirmation screen. Clicking on the cluster name ( in breadcrumb) on top left corner takes you to cluster dashbaord.

On the cluster dashboard screen, You can see option in the left side navigation menu. Click on the Topic option to create a topic.

Next click on Create Topic button

In next screen provide the Topic name and no. of partitions . If you want to create topic with default option, You can click on the Create with defaults button

If you want to customize the advanced options, You can click on the link and select different settings. Click on Save and Create button to create a Topic.

In the next screen you can see the confirmation of Topic creation.

Once Topic is created , we can connect to cluster and send messages/events to Topic using Confluent CLI.

To connect Kafka cluster in confluent cloud , first we need to create a API key and secret first.

Go to the cluster dashboard page. Click on the DataIntegration link on left side navigation.

Under Data Integration menu , click on the API Keys sub menu.

In the next screen, click on the Create Key button

In next screen select Global Access option , click on next button.

Next screen generates key and secret either copy them or click on Download and Continue button to download the api key file.

1. Install / Update the Confluent CLI

curl -L --http1.1 https://cnfl.io/cli | sh -s -- -b /usr/local/bin

Note

If you are using Windows OS, you need to install WSL (Windows Subsystem for Linux) to run sh and curl commands.

If are running these command as non root user, run commands with su

2. Log in to your Confluent Cloud organization using the Confluent CLI

confluent login --save

When prompted for your username and password, enter the same credentials that you used to log in to Confluent Cloud.The optional –save flag saves your login credentials to a local file for future use.

3. View available environments

confluent environment list

4. Select your environment

confluent environment use env-z29gy

5. List available clusters

confluent environment list

6.Select your cluster

confluent kafka cluster use lkc-pgkr12

7. Use an API key and secret in the CLI

An API key is required to produce or consume to your topic.We will add already produced to CLI with following command.

confluent api-key store --resource lkc-pgkr12
Key: <API_KEY>
Secret: <API_SECRET>

Assign API key to use the resource

8.List all topics available in the cluster

confluent kafka topic list

9. If there are no topics, you can create one with following command

confluent kafka topic create sample-topic

10.  Produce messages to your topic

confluent kafka topic produce sample-kafka-topic

Run that command, then produce a few messages to your cluster in Confluent Cloud using the console producer. You can do this by typing anything into the console and pressing Enter

11. Consume the messages you produced to your topic

confluent kafka topic consume -b sample-kafka-topi

Confluent cloud provides free Kafka cluster for up to 2 months. If you are looking for free Kafka cluster for life time.

you can look at the below blog post. CloudKarafka’s free Kafka cluster has limited resources but no time limit.

References

https://kafka.apache.org/documentation/

https://docs.confluent.io/

Similar Posts