Nifi - Ingest text file into flowfiles and put them into a mySQL table

Every business must deal with text files. Let’s ingest then into Apache Nifi in order to move the data where we want it.

Support the channel by Subscribing!
SUPPORT THE CHANNEL

FIND ME ON THE WEB

What is Apache NiFi?

Put simply NiFi was built to automate the flow of data between systems. While the term 'dataflow' is used in a variety of contexts, we use it here to mean the automated and managed flow of information between systems. This problem space has been around ever since enterprises had more than one system, where some of the systems created data and some of the systems consumed data.

Resources:

[StevenKoon]

If you found this video helpful for learning about Nifi and ingesting text files. Then consider subscribing to the channel and getting notified about more video’s.

Also feel free to leave a comment!

[Mrsantoshbr]

Tutorial helped me to fix a trivial bug in my processor configuration. Thanks Steven!!.

[greatjobbuddy]

So So So helpful. Thanks for making this.

[gauravpande]

It was an amazing tutorial. Thanks Steven !!

[alanmejia]

Thanks a lot Steven! You are making a great Job.

[pukar]

Does it only support row by row insert or update? How to implement bulk inserts though db specific load utilities?
Would you please help?

[annabeatriza]

I love this tutorial, it is so well explained and detailed :) I would love to see a video on how to read a fixed width file using Nifi, as I'm struggling with this assignment at work!

[radkoful]

Good work. But is possible to group multple sql operation in one single databse transaction ? Example: First delete all records from table, next insert multiple records ? Any bad insert rollback delete and previous inserts ?

[dungtrananh]

Dear sir, why do we need to convert flow file into json before making the insert sql statement?

[nunosilva]

Thanks for the tutorial...but I have a question...I have a executeSQL processor to get records from a table with a timestamp of the latest 10 minutes...this outputs do n flowfiles (1 row per flowfile)...I I run this processor every 5m (so I get a overlap of records, intended)....after that, I pickup those flowfiles and convert them from avro2json, then json2sql and send them to a putSQL processor to another database...I was expecting that the putSQL considered a flowfile as a transaction, in case of a flowfile data was already in the destination database, it discarded, moved to the next flowfile, extracted the data and try the INSERT INTO tablename...but if I have 3 flowfiles, with different data, only 1 flowfile data already in destination database, it fails all 3 flowfiles data insert...any hint/tip?

[MohamedMontaser]

does this csv files needs to be binded in a volume to the docker container?

[fedesardo]

Thanks for the tutorial, just starting with nifi. Could you please share a link to pull all the csv files? Regards from Argentina

[ΝικοςΚασαρακης]

Hi i saw that at the getfile after some time, if you have lets say 10 out, and it does not do anything (stoped or the queue on the connector is full), after some time it begins to reduce it until it gets back to 0. What does this mean and why this is happening? Also is this usefull for something? And how much is the time that reduces them?(for example every 10 sec? every 4-5 secs?) thank u

[gridplan]

Excellent tutorial. Thank you for creating/posting it. I like that I can inspect the queue at each step. For a given queue entry, is it possible to determine the line number in the CSV that produced it?

[samsal]

Hi Steven,
Thanks for the video. I have a question regarding the ConverntJsontoSQL & PutSQL, can you use them against MS SQL?

[hevertoncss]

thank you very much, great content! your video helped me a lot in learning the tool!

[luchardin]

Hi man, i'm learning to use nifi for my work, i'm having an issue with the put json to sql processor, i cant find the folder where i have my drivers of mysql, i'm trying to find them for downloading, but it seems more difficult than i thought, is there any other way to put the json into my database without declaring the driver?

[ΝικοςΚασαρακης]

What do the controller services really do? What would have happen if you didnt use csvreader ? I cant get the meaning of controller services

Thank u

[Maksimster]

First of all thanks for this video!
And I also had a question.

Why not use a processor "PutDatabaseRecord" for the same purposes? It seems to me that it looks much more logical and easier to understand.

[גלעדציסמדיה]

Greetings,
First, I wanted to thank you for the tutorial.
I've been trying to connect to orientdb database in the same way you connect to mysql and it fails. Do you know how to connect to orientdb, with emphasis on filling in the fields: Connection URL, Class Name and the path.
thank you very much!