Kafka Deployment on AWS EC2

2 min readAug 9, 2019

Ec2 is an IaaS(Infrastructure as a Service) platform provided by AWS. any authorized user can create AWS provided EC2 instances as per required configuration. it provides the choice of OS, memory size with other customization.

Kafka is a PUB-SUB based stream processing solution that provides scalable stream processing.

Pre-Requisites:
AWS account with access to EC2 service.

Create EC2 instance

AWS provides very good documentation for creating an EC2 instance. Use Ubuntu Linux machine for instance creation. please flow this link https://aws.amazon.com/ec2/getting-started/ and allow access of ports

22 to allow ssh connection
9092 to allow outside access to Kafka server

Login to EC2 instance

Use an ssh client for it with command

$ssh -i “<your-pemfile>” <username>@hostaddress

more instruction to connect with instance — https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AccessingInstancesLinux.html

Awsome you have reached so far 🎉 🎉 🎉

Install Java

$sudo apt-get upate 
$sudo apt install openjdk-11-jre-headless

Download Kafka

Run given command to download

$wget https://www-us.apache.org/dist/kafka/2.3.0/kafka_2.12-2.3.0.tgz

Command output:

Great you have achieved a milestone 🎉 🎉 🎉

Unzip File:

$tar -xzf kafka_2.12-2.3.0.tgz$cd kafka_2.12-2.3.0

Start Server Kafka Server

To start Kafka we need two servers up and running

Zookeeper and Kafka

let's start Zookeeper first. run given command

$bin/zookeeper-server-start.sh config/zookeeper.properties

Open a new window and run command to run Kafka server

$bin/kafka-server-start.sh config/server.properties

Great now we all set for run Kafka clients 🎉 🎉

Create Topic:

$bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic test

List Topic:

$bin/kafka-topics.sh --list --bootstrap-server localhost:9092

Send Some messages to the topic

$bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
> Hello Topic Test [type this message and hit enter]
> How are you?

Start a consumer

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
> Hello Topic Test #[Same message your would receive]
> How are you?

All set, you did awesome job Kafka server is up and running 🎉 🎉

Run Kafka on public IP-Address

Let's stop both servers and clients and please note down your public IP address from the description tab.

Edit config/server.properties

advertised.host.name=<public_ip>
host.name=0.0.0.0

Edit config/producer.properties

metadata.broker.list=<public_ip>

Run both servers agian

$bin/zookeeper-server-start.sh config/zookeeper.properties
$bin/kafka-server-start.sh config/server.properties

Refernce:

Kafka Deployment on AWS EC2

Written by vinay mavi