35. Compare source data with target data using data flows in Azure data factory or Azure Synapse

In this video, I discussed about comparing data between source and target using data flows in Azure data factory or Azure Synapse.

Link for Azure Synapse Analytics Playlist:

Link for Azure Databricks Play list:

Link for Azure Functions Play list:

Link for Azure Basics Play list:

Link for Azure Data factory Play list:

Link for Azure Data Factory Real time Scenarios

Link for Azure LogicApps playlist

#Azure #AzureDatafactory #DataFactory

[sambathnarayananparthasara]

I am using Synapse. As you mentioned, ADF is integrated into that. I am able to replicate your demos in Synapse Analytic. Thanks. Good job !

[annukumari]

How did I miss this video.. This is amazing 😊 Thanks

[sumanyarlagadda]

The shortcut to a clear-cut explanation is your knowledge sharing videos on ADF. Great to follow your channel or playlist. Thank you so much🙇 🎆

[ranjansrivastava]

Dear Maheer, Really very Informative Video. Could you please prepare one video for the Data Reconciliation in ADF the way you have created and from there we can able to send email to the customer or stakeholder like we have checked the data quality before sharing the report to the customers. Really appreciable for that.

[sonamkori]

Thank You sooo much for wonderful explanation.

[yedukondalu]

hi sir i hope i get a job by watching you your videos these are really looking good and good knowledge of information in it I really thank you

[NeumsFor]

When I have a small column set like this, I like to concatenate a delimiter to minimize collisions as a safe practice. Been burned before whwn not using it.

[akashsharma]

Hello sir. Firstly I wanna thank you for this amazing series on adf it's been helping a lot. Sir, will you be adding more real world scenarios to this playlist or its the end of this series.

[brahmareddypotlapalli]

Hi, how can we get updated records as part of incremental load ?

[spranavshanker]

Can you please create this scenario using pipeline and activity, without using mapping dataflow. I think this will not be best approach with large data and costlier.

[sunilpandey]

Hi Maheer, Thank you for good quality videos on Azure!
Could you please help me with the following:
1. I need to test two Azure SQL DB's -
- Need to compare if all the data rows are same, give output if there are any different rows or missing rows in the target DB table
2. All general ETL sceanrios - Schema, tables count, table data etc.
3. Transformation testing - if the table has any transoformation then test it on the target Azure DB table w.r.t to Source Table

Overall idea is to to proper ETL testing

Please let me know if that's possible? If yes, then please point me right direction by providing some links or reference to go through?

Regards,
Sunil

[kiranpatil]

WhenCan you start new course for azure databrics with pyspark

[guddu]

how to exculde column from columns function, lets say 1 have 5 columns and want to use just 4 column but don't want to select individual column rather i want to use column() funtion and exculde a columnfrom it

[JesI-lfjy]

Is data quality DQ in analytics? What is BG DQ

[ObservePeace]

which ETL tool can I use to compare data between on-prem SSMS and ADLS gen 2

[ayocs]

hello sir, is it appropriate to use Exists against two database source (SQL Server and Postgres)
i want to insert only the non existing rows from one database to another
and also to note that I've to do this on production data; and sink is postgres

[srinivasarao]

Hi Brother, it is just like SCD2 . shall we use same SCD2 logic here. Please let me know

[ksnaveenkumar]

Have you tried this approach with large volume of data? I am curious to see the performance with large datasets.

[ayushtiwari]

Sir, please make a video on Incremental Data Loading Using ADF and Change Tracking for multiple tables.

[katha]

how to replace a null records in a table with someother value using ADF?