12. How to perform Upsert for Incremental records using Azure Synapse Pipelines

In this video , we learnt how to perform upsert for incremental records using Azure Synapse Pipelines
#azuredatafactory
#azuresynapseanalytics
#adf

[paul-xrkj]

Thankyou very much for taking the time to share your knowledge and create these videos. I have learned a huge amount from them and they have been essential in getting a tricky synapse solution over the line for a customer.
You should consider advertising and selling your Professional Services on the channel as well - you clearly know all of this inside out

[DrumCodeLive]

Excellent Annu, very well presented, thanks for investing your time and effort into this.

[ShirazHussain-cgkq]

Hi, I was stuck for the last three days, but after watching your video, understanding the concept, and implementing it, everything ran successfully. I am really thankful to you. God will bless you

[jehok]

This topic is what I just needed. Thank you so much !!!

[Andydady-fpzm]

Superb!!!, i was looking same scenarios. This video help me a lot. Thanks. Keep posting..👍

[chetak]

Gone through all your videos, these are very well explained and detailed. Helped to answer scenario based interview questions. Thanks Mam for this content. Looking for more videos on this. 👏👏

[Xavwar]

great video, thanks, do you happen to know if the upsert supports a uniqueidentifier column as a key column?

[eqhome]

Well done. I have a question that you may have thought through. I have 800+ tables in one of my databases of a system disbursed geographically at various locations. The tables are organized around accounts, parties, events, etc. Of the 800+ tables, only several hundred have system-initiated create or update fields (timestampcreate and timestampchange.) My Azure repository requires all tables to be present for reporting purposes. To develop virtual timestampcreate and timestampchange (rather than copying several million records each night), I've linked the tables in views to the nearest tip of the spear (for example, accounts) that does contain a timestampchange field. Do you think this is a strategy that you would suggest? Long and short, when the tip changes, the rest of the tree also changes.

[ashredsa]

How to do upset in dataflow with 2 column as key ?

[sajinmp]

Hai, is there any way to update azure sql table rows to pg sql table using this concept ?

[moulshreesuhas]

Can we do deletes as well?
Also can we use source as on-premises SQL database and destination as Azure Synapse?

[ewoutlagendijk]

I had the exact same List1 error, {
"errorCode": "2200",
"message": "ErrorCode=UserErrorUnexpectedObjectTypeInPayload, 'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException, Message=The value of property 'keys' is in unexpected type 'List`1'., Source=Microsoft.DataTransfer.DataContracts, '",
"failureType": "UserError",
"target": "Src to PreStg",
"details": []
}
but when using @array the error stays the same. What could be the reason?
I also have the key columns in a table that a lookup looks through.

[mohammedumar]

I am getting the same error that states, azure data factorySql upsert key column '[ column_name]' does not exist in the table '[dbo].[InterimTable_c...]'
In my framework I am using the Temp DB feature, any assistance will be appreciated, thanks.

[jhellier]

I noticed Upsert still downloads all the rows so there is no time saving. Can you confirm?

[rathikavenkatesh]

I am trying to do insert and update incremental. Please share the complete steps . Thanks in advance

[deepjyotimitra]

very good content. thank you..
i have a scenario, where my source data wont have any updates, only new records will be there along with old data(without any update in records) for every run. in this case, if i do upsert based on key column then all existing records will be updated(although nothing to update) and new records will be inserted(which is fine).
how to handle this scenario? i just want to avoid update only new data should get insert and rest should be ignored..

[pamilad]

Assume source is csv file and sink is azure sql...And source data doesn't have any primary key column...in this case how to build upsert logic?

[baronharkonnen]

Hi Ma'am. Is there a way we can integrate incremental models in dbt to azure synapse?

[rathikavenkatesh]

Hi could you please share the previous incremental. I am not able to follow the intital steps