Installing a JDBC driver for the Kafka Connect JDBC connector

Stuck trying to setup #ApacheKafka #KafkaConnect JDBC and hitting "No suitable driver found for … "? Use this _one simple trick_ to install a JDBC driver for use with Kafka Connect's JDBC connector with flawless results every time!

tl;dr:

* Put the JDBC driver in the *same folder as the Kafka Connect JDBC plugin*.
* Restart the Kafka Connect worker

Table of Contents - Local install
* 00:00 Introduction and background
* 01:19 Installing JDBC driver on local install
* 05:00 Determining the Kafka Connect JDBC plugin folder
* 07:16 How to find the Kafka Connect worker log
* 09:23 Viewing the data in Confluent Control Center
* 11:05 Installing JDBC driver for Kafka Connect running on Docker

Table of Contents - Docker
* 11:50 Installing the JDBC driver at runtime vs baking an image
* 14:29 Listing installed Kafka Connect plugins
* 15:00 Creating the connector and figuring out network addresses for the database
* 16:22 Using kafkacat to view the topics and data

The tutorial shows how to do it with both a 'bare metal' install of Apache Kafka or Confluent Platform, as well as on Docker.

--

☁️ Confluent Cloud ☁️

[daveklein]

Very clear and thorough, as always. Thanks!

[navarrodba]

Hi Robin, thank you. You explanation helped me a lot.

[TimoBarth-mb]

Great tutorial and very helpful! Thanks for this

[florinmarcus]

I don't think this is needed any longer.
running "confluent-hub install gets me both the driver and connector jar file automatically installed.

[vishnumurali]

Great and clear explanation video..


👏👏👏👏👏👏

[hamedmoradi]

hi Robin, how can I add db2 driver in kafka-connect docker container, it's so confusing, please explain and help me. thanks

[ddreddy]

Hello Robin, How are you getting kafkacat command? are you running on docker host or container? I tried with your github code, I am not getting confluent web base with port 9021, with code in getihub demo-scene, can I bring up all the services and write data from DB to kafka?

[scorpiooooo]

Hi Robin, thanks for your clear and organised contents. I'm trying to stream a table contains geospatial data from PostgreSQL (PostGIS enabled) to a Kafka topic. I have installed postgis-jdbc-2.5.0.jar through docker runtime. My JDBC source connector is up and running, but it does not write the geometry column to the Kafka topic. Is there anything wrong with my configuration? Thanks.

[aggankit]

Thanks Robin for all your wonderful video.

I have some doubt in jdbc source connector. I have created below table
CREATE TABLE Emp (
ID INT NOT NULL,
Firstname VARCHAR (20),
Lastname VARCHAR (20),
AGE INT NOT NULL,
MonthlySalary Int,
PRIMARY KEY (ID)
);

but in Query parameter I am using : select cast(CONCAT(Firstname, ' ', Lastname) as VARCHAR(20)) as Name, cast((MonthlySalary * 12) as INT) as AnnualSalary from Emp

Now, connector start properly but when I insert data in table it is not reflecting in topic. Mainly here I am manipulating query to use different parameter and want to sink that in target DB. Can I use define schema for this, is there a way?

[sarathbaiju]

Hi robin, i have a question regarding JDBC source connector offset reset. my requirement is whenever my connector restart it will load the entire data from the table. to achieve this what configuration property should i added in connector configuration. i searched alot in the internet most of the answer is rename the existing connector.. but i am not satisfied with this approach.
i need to change the source connector offset to 0 without renaming the connector, how we can do this? can you help me?

[BiswajitGhoshProfile]

Hi,

This video is great to configure KafkaConnect, thanks for sharing with us :),

Just one question: all time I'm getting full copy of a table after pool interval time, can I get the change only data, are there any configuration?

[ottodvalishvili]

Is /etc/confluent/docker/run mandatory? Should not it automatically start without that command?

[nikolaynedkov]

Hey great tutorial but i got the following error:
Could not initialize class oracle.jdbc.OracleDriver

Please help,

[JorgeGoncalvesdosSantos]

is it possible use kafka Connect on Docker connected to an external Oracle Database, outside network container? How can I achieve it?

[nnmartin]

For some reason, when querying :8083/connector-plugins the MySQL plugin does not show up, even though the confluent JDBC one does. Both the jar files are in the same volume for the docker container.
Anyone else run into this issue? I am not sure what more I can do...

[vyoivanov]

Hey thanks for the great vid! How can I restart Kafka Connect if I don't use confluent hub cli? It is in a K8S Pod and I seem to be unable to restart it...

[tariqmghari]

Hello can you please do a tutorial, where you can sink kafka topic to mongodb

[abhinavkumar-sefd]

@robin Where can I find Scala /java code for this ?

[oussemakouki]

the connector is created but when i run "list topics;" in ksql, there is no topic