7. Remove Duplicate Rows using Mapping Data Flows in Azure Data Factory

In this video, i discussed about Removing duplicate rows using Mapping Data Flows Or getting distinct rows using mapping data flow in azure data factory

Link for Azure Functions Play list:

Link for Azure Basics Play list:

Link for Azure Data factory Play list:

Link for Azure Data Factory Real time Scenarios

#Azure #ADF #AzureDataFactory

[anithasantosh]

Thank for the video . I was trying to use Groupby and rest of the columns as a stored procedure. Your video made my job easy.

[susmitapandit]

In Output file, Why EmpID is not in sorted format even though we used Sort function?

[rajkiranboggala]

Well explained!! Thank you. If I have only one csv file and I want to delete the duplicate rows, I guess I can do the same by self union’ing the file, I’m not sure if there’s any other simpler method

[mankev]

Good concise tutorial with clear explanations. Thank you.

[nareshpotla]

Thank you Maheer. If we have 2 same records with unique empid you use last($$)/first($$) to get either of one. If we have 3 records like
1, abc
2, xyz
3, pqr. if we use first($$) we will get 1, abc and last($$) will give 3, pqr.How to get the middle one (2, xyz)?

[luislacadena]

Excellent video, do you think that it is possible to eliminate the values keeping for example the one that has the higher department id/number? I've seen that you kept the first register by using first ($$), but im curious if you can remove duplicates in the RemoveDuplicateRows box based in other criteria. Is it possible to keep only the duplicates with higher department id?

[marcusrb]

Great video, it's clear. But, what happen with new records? Because If you use an Union table and use only upsert, check only duplicates rows isn't it? I tried same of yours, but new one are removed in the final step. I tried and I figure out an issue for INSERT, UPDATE and DELETE in three separate steps, how could I achieve it? Thanks

[lehlohonolomakoti]

Amazing video, super helpful, allowed me to remove duplicates from a restapi source and create a ref table inside my db

[maheshpalla]

How do you know which function to use since we are not sure about duplicate rows if we have millions of records in Source??

[PhaniChakravarthi]

Hi, thank you for the sessions. They are wonderful. Just have a query, can you make any video on identifying the DELTA change between two data sources and capture only the mismatched records with in ADF?

[EmmaSelma]

Hello Wafa,
Thank you so much for this tutorial, it's very helpful. New subscriber here.
Thinking of scenarios to use this, I have a question please : Is it correct to use this to get last data from ODS to DWH in the case of a full load (only insertion occuring in ODS and no truncate) just like row partition by ?
Thank you Upfront.

[gsunita]

Data in Output Consolidated CSV is not sorted on EmployeeID, we did use the Sort before the Sink, then why the data is not sorted ?

[rohitkumar-itqd]

How do I update the records in the same destination, the updated record and the new record without having any duplicates on ID. PLEASE SUGGEST.

[Anonymous-cjgy]

in the output file data is still not sorted, if you see it. same thing happen with me also. even after using sort - data is still unsorted.

[swapnilghorpadewce]

Hi, I am trying to bulk load multiple json files to cosmosDB. Each json file contains json array 5000 objects. total data size is around 120 GB.

have used "copy data" with "foreach" iterator It is throwing error for respective file but inserts some records from file.

I am not able to skip incompatible rows. also, not able to log skipped rows. have tried all available options. Can you please help?

[AkshayKumar-ouin]

thank you for the video, very good explanation

[kajalchopra]

How we can optimize the cluster start up time. Basically it is taking 4m 48 sec to start a cluster. So how i can reduce that?

[battulasuresh]

What if we wanna remove both columns
Point2 what if u wanna specifically want in middle of a row saying latest modified date column like that

[arifkhan-qetd]

Aggregates is not allowing me add $$ as an expression. Any suggestions pls.

[karthike]

Hi, I have to check all the colum duplicate and how to handle in aggregate activity, please help me