Kafka Tutorial - Custom Partitioner

Spark Programming and Azure Databricks ILT Master Class by Prashant Kumar Pandey - Fill out the google form for Course inquiry.
-------------------------------------------------------------------
Data Engineering using is one of the highest-paid jobs of today.
It is going to remain in the top IT skills forever.

Are you in database development, data warehousing, ETL tools, data analysis, SQL, PL/QL development?
I have a well-crafted success path for you.
I will help you get prepared for the data engineer and solution architect role depending on your profile and experience.
We created a course that takes you deep into core data engineering technology and masters it.

If you are a working professional:
1. Aspiring to become a data engineer.
2. Change your career to data engineering.
3. Grow your data engineering career.
4. Get Databricks Spark Certification.
5. Crack the Spark Data Engineering interviews.

ScholarNest is offering a one-stop integrated Learning Path.
The course is open for registration.

The course delivers an example-driven approach and project-based learning.
You will be practicing the skills using MCQ, Coding Exercises, and Capstone Projects.
The course comes with the following integrated services.
1. Technical support and Doubt Clarification
2. Live Project Discussion
3. Resume Building
4. Interview Preparation
5. Mock Interviews

Course Duration: 6 Months
Course Prerequisite: Programming and SQL Knowledge
Target Audience: Working Professionals
Batch start: Registration Started
Fill out the below form for more details and course inquiries.

--------------------------------------------------------------------------
Best place to learn Data engineering, Bigdata, Apache Spark, Databricks, Apache Kafka, Confluent Cloud, AWS Cloud Computing, Azure Cloud, Google Cloud - Self-paced, Instructor-led, Certification courses, and practice tests.
========================================================

SPARK COURSES
-----------------------------

KAFKA COURSES
--------------------------------

AWS CLOUD
------------------------

PYTHON
------------------

========================================
We are also available on the Udemy Platform
Check out the below link for our Courses on Udemy

=======================================
You can also find us on Oreilly Learning

=========================================
Follow us on Social Media

========================================

[ScholarNest]

Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code.

[glpathy]

Excellent Sir, Thank you for giving detailed insights and showing how message key partition will get in to trouble. Very Practical.. Thank you sir.

[rickydebroy]

Just to add, will also give you the number of partitions for a topic. I am using in kafka_2.13-2.8.1, so this should be available in the upper versions as well. Thank you for this awesome series on Apache Kafka.

[musicmania-dz]

How we can print the Partition with Message in windows . same like you have printed at @11:44. Actually I am using window machine and can't able to find any command which give me in which partition what message reside . Through topic we can read but want to check through the partition .If possiable read all the partition and message to the topic .it's greatful to me

[shrinivasashetty]

Hi Sir, Am using Java IDE and i get below error messge


Exception in thread "main" Invalid value SensorPartitioner for configuration partitioner.class: Class SensorPartitioner could not be found.
SensorPartitioner has been define under same package

[杨正云]

Thanks for explaining such important detail!

[rameshhawks]

Where your calling the int partitioner???

[sahilgarg]

Sir, how can we create topic while reading some csv file ? Like there are 10-15 fields in CSV and we want to create topics on every unique value which is present in column 6. is this possible ?

[TheNayanava]

I think there is a little mistake in the explanation where you say that SSP3 and SSP8 have been allotted the same partition because the hash doesn't guarantee unique value. It could very well be because we do a % (numberOfParitions - sp), which in this case will range between 3-9.. so there is a probability that every 1/6th key gets assigned to the same partition.. Correct me if I am wrong please

[veeranjikomatineni]

In Kafka we are using and write the code Only in Java.... we don't need to write the code in Spark. if Spark is possible send some links for learning Spark Producer codes and etc...
thanks for giving very expensive Knowledge!!

[LivenLove]

If different keys can lead to same hash..isnt it possible for different values to give same hash??

[ragavkb]

Thanks for the explanation, Is there a benefit of implementing such custom partitioner to determine the partition to store instead of using different topic altogether for such cases? I might be missing something here, appreciate if someone could help me answer. Thanks again

[pramodsripada]

Sir, in the Custom partitioner that you have implemented, starts returning different partition when the number of partitions increase right? Because of the dependency of number of partitions we have moved away from Key based partitioning. Am I missing anything

[vivekkatakam]

when i run the program, i am getting below error, please help me..

Exception in thread "main" Invalid value SensorPartitioner for configuration partitioner.class: Class SensorPartitioner could not be found.

[venkateswarlukomirisetti]

Hi sir, I have some queries ...., can you please clear those?

In above program we are passing broker configuration through "bootstrap.servers" property. Is it mandatory, or any possibility to achieve same through Zee-Keeper?

How Producer interacting with Zoo-Keeper to have brokers information(cluster information) for a topic?

[roshanbisht]

Kafka Security tutorials will be helpful as well

[saurabhsaxena]

Thanks for this wonderful video.
I have a doubt here: In custom partitioner section, you are hashing the message value in step3 and hashing key in step 4.
So in step 3, do you mean message value is the actual message sent by producer? Are you hashing the actual message?

[ayyappa]

p = % sp;
I am getting the error at .toPositive.
i am unable to find toPositive method in sensorpartitioner.
Plzz Help me!!

[mkvjaipur]

it would be great if you can tell something about how you used sbt tool :)

[hadoopworld]

Sir, Can I get list of partition from consumer class.