Power BI Incremental Refresh - Understanding Detect Data Changes

Patrick breaks down how detect data changes works with Power BI Incremental Refresh. He demystifies some common misunderstandings when using this setting in Power BI Premium.

********
LET'S CONNECT!
********

***Gear***

#PowerBI #IncrementalRefresh #PowerBIPremium

[trentonsmith]

For anyone reading in 2022, custom polling queries allow you use an M expression to detect data changes instead of a single datetime column selection that Power BI provides. You can use this for many things, including deletions if need be. No need for the soft deletion limitation, so if you are thinking that is a stopping point I would take a look at that feature.

[Baldur]

"I'm kind of big deal" when I spotted it I laughed. Patrick you've most swag from every business intelligence youtuber.

[andresgeb]

Hi Pathric! I've a question.. what if I have set incremental refresh for the last 1 month, Detect data changes on my "UpdatedDate" column (yes, Is a DATE, not datetime column unafortunatelly) AND I've schedule refresh every hour..
Then, one day records were loaded into the DB, the first Time the dataset Is refreshed loads the records into the dataset..
Then, the UpdatedDate column Is updated for the partition (todays date).
Later on, a user loads More records into the DB... same vale un the UpdatedDate column... Then question Is... As the UpdatedDate Will be the dame value... The refresh would not load the new data.. that Is correct?

[dangelo]

Epic, as always. Needed info, showing how to, straight to the point. Thank you for your videos!

[mabl]

What if you have no control over the data source and it does hard deletes.

Do you have to always get all the records in order to check if any record was deleted?

[avivzach]

Amazing feature! can it be used for updating data? lets say I have 5 years of data, records being updated randomly. can I use it to update just the records that were changed in the last day fir example? and refresh once a day?

[ayanmukherjee]

Hi I'm having an error ""OLE DB or ODBC error: [DataSource.Error] An error happened while reading data from the provider: 'An item with the same key has already been added.'."After I created a filter on a DateTime for an incremental refresh with RangeStart and RangeEnd. Can you help me with?

[sumanchakraborty]

My DB has 1 month rolling data . I trying keep 3 months data in power bi. So implemented incremental refresh . But once data gets deleted from DB also getting removed from the power bi. Can you help me on this ?

[rosecraigie]

Hi Patrick, I've just set up the incremental refresh with detect data changes and we are seeing a lot of queries come through with select top 1000, some with the partition parameters and some without. Is this due to something I have done wrong in the set up or is it just how incremental refresh works?

[philipmead]

i am about to get premium just so i can do this, however does this work when refreshing over 2500 sharepoint files? cause at the moment its only brining in around 1200 files into power bi. so if i got premium and setup incremental refresh would that fix the issue?

[tsepont]

This is gonna work great for CommCare/PowerBi - the refreshes take a while! Thanks chief!

[DineshKumar-hctj]

Selecting detect data changes on the last update time of the records is returning the error saying duplicate records found. There are no actual duplicates in the table, but seems like the refresh is trying to insert the record instead of updating the record. How can I get rid of this?

[roelturnbull]

Question: Would this technique also work with partitions in the future? The incremental refresh in PowerBI Desktop you can only refresh data in the last year/quarters/month/day. But I have data for future months. Can I add records for future months in the config_FIS_DOC table for the future so that the partitions are created as soon as the record in config_FIS_DOC is triggered for refresh?

[krishnansrinivasan]

I want to know if we can use power flow or power automate to detect changes to underlying tables and if so refresh the power bi dataset automatically

[buarvind]

Hi Patrick, I have a problem. How can I map two tables on multiple columns based on some conditions. Like I have table1 with columns A, B, C, D and table2 with columns A, B, C, E now I want to map table1 and table2 based on columns A, B, C in both the tables.

[marcford]

Sounds like Inc Refresh has to go and get (example) 5yrs of data the 1st time and can the settle into smaller refreshes afterward. My issue is, when I make a small change to the report in Desktop and publish it again, it seems like we start all over needing 5 years. Am I missing something?

[patsickboy]

I don't get email for download the cheat code file

[astriddini]

hello good explanation. but i have problem when using this. I often get duplicate IDs on certain tables. i use a dataset. and I have worskspace and everything is scheduled refresh. Is it because there are many refresh times at the same time causing someone to fail to delete the data that is there. For real data I use DataView in my database. have you also experienced something like me ..?
thank you..

[nataliyar]

was trying to ask this question on your website but could not find a way so here it is: I am trying to use a conditional formatting based on the if statement. If the statement is true take the info from this data source if false go to another data source. I want to make them colored based on which data source the date came from.

[pranavkumar]

Hi Patrick, is it possible to display value is x out of y in tooltip. For example, 10 out of 100 (10.00%) using DAX or some other