Use Asserts for Data Quality and Data Validation in ADF

preview_player
Показать описание
  1. Data Factory
  2. azure ETL
  3. azure data factory
  4. azure data integration
  5. azure synapse analytics
Рекомендации по теме
Use Asserts for 0:13:18

Use Asserts for Data Quality and Data Validation in ADF

89. Assert Transformation 0:16:45

89. Assert Transformation in Mapping Data Flows in Azure Data Factory

SQL Script Asserts 0:05:19

SQL Script Asserts and Drift in ADF

Comments are bad, 0:05:07

Comments are bad, do this instead!

Easily capture source 0:06:22

Easily capture source changes with ADF

Data Quality - 0:22:01

Data Quality - Validation Rules and Validation Rule Analysis 2021 Q2

ADF: Detect source 0:04:51

ADF: Detect source data changes in Data Flows

This saved me 0:01:47

This saved me HOURS of refactoring time

Thoughts About Unit 0:11:21

Thoughts About Unit Testing | Prime Reacts

Systematic data validation 1:07:12

Systematic data validation with the validate package

Cross Table Validations 0:02:52

Cross Table Validations

Apex Testing: Basics 0:08:21

Apex Testing: Basics and Asserts | Developer Quick Takes

ADF Data Lake 0:10:03

ADF Data Lake Clean-up with Data Flow Fuzzy Lookups

Use data flow 0:09:10

Use data flow output values inline in your pipeline

Stop Recommending Clean 0:27:05

Stop Recommending Clean Code

16. Validate file 0:13:16

16. Validate file schema before processing in Azure Data Factory

Asserting Your Way 0:37:54

Asserting Your Way to Faster Programs - Parsa Amini - CppCon 2021

Software Testing Theory 0:20:33

Software Testing Theory + A Few Less Obvious Testing Techniques

Azure Data Factory 0:06:07

Azure Data Factory Part 10 - ADF Data Flow Selects

Write effective Unit 0:46:10

Write effective Unit tests/ test Classes | DAY 6 PART 2

Google follows Samsung, 0:14:13

Google follows Samsung, asserts the right to steal your phone during a repair

Assertions in PyTest 0:05:29

Assertions in PyTest (PyTest - Part 6)

Pure Unit Testing 0:27:09

Pure Unit Testing in Apex

72. Different Author 0:10:06

72. Different Author Modes in Azure Data Factory

Комментарии
Автор

I love the directness of these videos. No flashy title cards, or "Hey YouTube! It's ya boy, Mark!" It reminds us that good documentation can be as easy as walking through a few examples on video. It doesn't have to be fancy or take a long time to produce.

I'm going to create asserts in my dataflows from now on. Thanks for the video!

andyparkerson
Автор

Awesome video, thanks for the explanation. wish all videos were like this

emiliogarzacantu
Автор

solid video! question, what's the point of the description? I don't see where it's ever useful. It must have a purpose though.

cburys
Автор

Love this demo and have adapted asserts where we can. Checking for lengths, data types ir null values seems to work well. But as soon as we introduce an assert checking a field for a set of values slows us down dramatically and 4 or 5 will cripple the data flow. Is there a better way to implement something along the lines of contains('a', 'b', 'c', item = myfield). Using 'or' seems to have similar issues for us. We've scaled the processors up for the dataflow and still find that some of our validation has to be implemented directly in a conditional split because the logic seems to overwhelm the assert transformation.

tybutton
Автор

What will be your approach of system testing in azure data factory.

nissarahmad