Kafka Tutorial - Quick Start Demo

Spark Programming and Azure Databricks ILT Master Class by Prashant Kumar Pandey - Fill out the google form for Course inquiry.
-------------------------------------------------------------------
Data Engineering using is one of the highest-paid jobs of today.
It is going to remain in the top IT skills forever.

Are you in database development, data warehousing, ETL tools, data analysis, SQL, PL/QL development?
I have a well-crafted success path for you.
I will help you get prepared for the data engineer and solution architect role depending on your profile and experience.
We created a course that takes you deep into core data engineering technology and masters it.

If you are a working professional:
1. Aspiring to become a data engineer.
2. Change your career to data engineering.
3. Grow your data engineering career.
4. Get Databricks Spark Certification.
5. Crack the Spark Data Engineering interviews.

ScholarNest is offering a one-stop integrated Learning Path.
The course is open for registration.

The course delivers an example-driven approach and project-based learning.
You will be practicing the skills using MCQ, Coding Exercises, and Capstone Projects.
The course comes with the following integrated services.
1. Technical support and Doubt Clarification
2. Live Project Discussion
3. Resume Building
4. Interview Preparation
5. Mock Interviews

Course Duration: 6 Months
Course Prerequisite: Programming and SQL Knowledge
Target Audience: Working Professionals
Batch start: Registration Started
Fill out the below form for more details and course inquiries.

--------------------------------------------------------------------------
Best place to learn Data engineering, Bigdata, Apache Spark, Databricks, Apache Kafka, Confluent Cloud, AWS Cloud Computing, Azure Cloud, Google Cloud - Self-paced, Instructor-led, Certification courses, and practice tests.
========================================================

SPARK COURSES
-----------------------------

KAFKA COURSES
--------------------------------

AWS CLOUD
------------------------

PYTHON
------------------

========================================
We are also available on the Udemy Platform
Check out the below link for our Courses on Udemy

=======================================
You can also find us on Oreilly Learning

=========================================
Follow us on Social Media

========================================

[ScholarNest]

Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code.

[vizvan]

Thanks for the tutorial. Works like a charm. I typed up the commands being used for others to copy-paste.

bin/zookeeper-server-start.sh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic MyFirstTopic1 --partitions 2 --replication-factor 1
bin/kafka-topics.sh --zookeeper localhost:2181 --list
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic MyFirstTopic1
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic MyFirstTopic1

[杨正云]

I have a question about partitions and brokers. In your demo at 6:20 you created 2 partitions and Kafka had no choice but allow 2 partitions live in the same broker. What if later on there is a new borker joined the Kafka cluster? Will Kafka reassign one of the partitions to the new broker? How do I reassign partitions to new joined brokers manually?

[DeepakSharma_youtube]

Would NiFi be an alternative to Producer/Consumer without the need for APIs? What's the common practice that you see being adopted instead of API approach?

[DeepakSharma_youtube]

I watched until Video 10 in this series and gave up because it was getting too technical. But, I got a good idea about the whole infrastructure and process flow, thanks!

There are still a few basic things that I am not clear, and I think these should be explained up-front or early on:

- In your example, you started Producer and Consumer on the same Linux VM, but in the real world where exactly do you 'Start' the 'Producers'
- If I just want some CSV files from a few remote servers, I assume there will need to be a Producer running on each of those servers, but is there an example of how to get those files to a local storage?
- In a Hadoop environment, I assume the consumers would place the messages in a Data Lake and not in HDFS, some other process will do that later on, is that correct?

[monikajha]

I get this error for produce and consumer both..

[2019-07-08 15:02:47, 321] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 1249 :


I followed all your steps

[neerajnegi]

its a really good series on Kafka. I have a query ==> as per my understanding in kafka consumer side mechanism is pull based means consumer will pull data from Kafka but in console example the data goes directly to consumer once provider sends data to broker.

[xuelvming]

Really helpful . Good for starting, but after a few weeks diving, I hope you sir could give some deep-in video tutorials about kafak, especially, the kafka connect and kafka streaming .Thanks .

[csharma]

This is a great tutorial series. Thanks a lot. Keep up the good work Sir.

[prabhleenkaur]

Hey sir,

when I an trying to run command "bin/kafka-server-start.sh config/server.properties", it shows me an error message "usage: dirname path
Java HotSpot(TM) 64-Bit Server VM warning: Cannot open file /../logs/kafkaServer-gc.log due to Permission denied

Error: Could not find or load main class config.server.properties"

Can you help?

Thanks in advance!

[saravanankumar]

command line from your video:

bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic TEST --partitions 2 --replication-factor 3

command line from apache.kafka.org:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test


As per the above 2 commands both has been scripted to create a topic, but the fields are mismatched . will it still work if we mismatch the fields after the .sh ?
Please correct me if I'm wrong .

[tatavarthyvenkatesh]

How broker will connect here to zookeeper ?? Are we giving any zookeeper port details to broker properties before starting it ?

[soniyabali]

Hello Sir,
Thank you for valuable guidance..!
I have one doubt, In my usecase, Sensors are generating real time data and I want to load this data into Kafka Topic using TCP/IP, how to do it??

[tekadeaniruddha]

I am using Mac OSX Mojave. After I tried to create zookeeper the, there has been no message as `started`. Moving forward, when I am trying to create topic, getting error that `Replication factor: 2 larger than available brokers: 0`. I believe this means that zookeeper is not started. I am not able to understand why. I am using Kafka_2.11-2.0.0

[saurabhs]

getting an error while running this:
MAC_XYZ:kafka-2.3.0-src user1$ bin/zookeeper-server-start.sh config/zookeeper.properties
Classpath is empty. Please build the project first e.g. by running './gradlew jar -PscalaVersion=2.12.8'

[monfusinglisha]

what is the linux flavor/version you are using ? I am going to download kafka_2.11-2.0.0.tgz

[siddhantaggarwal]

I am getting an error like this while trying to initiate kafka server: kafka-run-class.sh: line 270: exec: java: not found
I couldn't find a solution to this online. Could you help? Thank you

[chin-said]

Hi,

I am using MACBook Air, so, can I directly install KAFKA Setup in MAC Terminal, following the steps that is demonstrated in your video?

regards,

[gobiviswa]

I am getting the following error while typing message in the producer console ..

"Warn error while fetching metadata with correlation in id (int number) :


please explain this...

[kamruzzamantanim]

Fantastic Tutorial. I like it. Hope, more video will come in future