Add custom logging and auditing to ADF Data Flows

Learn how to add custom logging and auditing to your ADF Data Flows. #Microsoft #Azure #DataFactory #MappingDataFlows

[aqilahmed]

Thank you so much for posting this video. Its very useful and can be used in a lot of scenarios.

[Big_Khao]

Thanks for video. How can it append to logging file? What are recommended common folder names are there for sources and sinks - eg source, target, logging ... as either sub folders per pipeline folder name or main folders and then pipeline names within each main folder?

[torerock]

Sometimes I getting the "metrics" value in NULL in specific pipelines, any idea or solution for this problem ??

[NeumsFor]

When doing error row "continue on error", I know that we can output the data to a file. However, since that temp table is the JSON payload, can we actually still access that temp table and write it to an errorEventFact table; or, is that staging temp table already truncated, dropped, and cleaned up at the conclusion of the data flow activity? I think keeping it around until the conclusion of the entire pipeline could open up some interesting and creative possibilities.

Adding an option box in the settings to "keep staging table data until pipeline conclusion" when "continue on error" is selected may not be a bad idea.

Happy holidays, sir.

[kunalbhatia]

Getting Error :

The expression 'activity('Data flow1').output.Error' cannot be evaluated because property 'Error' doesn't exist, available properties are 'runStatus, effectiveIntegrationRuntime, billingReference, reportLineageToPurview'.

Can I get the solution .. ?

[aaa]

how do i estimate, the cost of the activities before i plan to design my ETL?