filmov
tv
How to test your Python ETL pipelines | Data pipeline | Pytest
Показать описание
In this tutorial we are going to cover how to test ETL pipelines. I have received a number of inquiries on the testing and especially testing the data pipelines we build using python. Testing is an important aspect of ETL pipelines. It ensures we are delivering accurate information to our stakeholders. We want to make sure our data is current, consistent and accurate.
Therefore, it is always a good idea to put test cases in place to catch data anomalies. A failing test can tell us that;
• An assumption about your source data is incorrect. For example, a column we expected never to be null contains nulls or a column we expected to contain unique values contains duplicates.
• Testing can catch the flaws in our transformation logic.
Errata in the tests: One of the viewers pointed that the null check was always returning true. It has been revised to to return false when nulls are present. test_null_check function is updated as follow:
def test_null_check(df):
assert df['ProductKey'].notnull().all()
#pytest #etl #python
Subscribe to our channel:
---------------------------------------------
Follow me on social media!
---------------------------------------------
Topics covered in this video:
0:00 - Introduction to ETL testing
0:56 - Benefit of testing
1:32 - Pytest testing library overview
2:26 - Pytest setup
3:05 - Import Data
3:36 - First test - column check
6:08 - Primary key column tests
7:22 - Pytest features
8:15 - Data Type check
9:36 - Expected Values check
Therefore, it is always a good idea to put test cases in place to catch data anomalies. A failing test can tell us that;
• An assumption about your source data is incorrect. For example, a column we expected never to be null contains nulls or a column we expected to contain unique values contains duplicates.
• Testing can catch the flaws in our transformation logic.
Errata in the tests: One of the viewers pointed that the null check was always returning true. It has been revised to to return false when nulls are present. test_null_check function is updated as follow:
def test_null_check(df):
assert df['ProductKey'].notnull().all()
#pytest #etl #python
Subscribe to our channel:
---------------------------------------------
Follow me on social media!
---------------------------------------------
Topics covered in this video:
0:00 - Introduction to ETL testing
0:56 - Benefit of testing
1:32 - Pytest testing library overview
2:26 - Pytest setup
3:05 - Import Data
3:36 - First test - column check
6:08 - Primary key column tests
7:22 - Pytest features
8:15 - Data Type check
9:36 - Expected Values check
0:39:13
Python Tutorial: Unit Testing Your Code with the unittest Module
0:00:47
How To Setup and Run Your First Test (Testing Python With Pytest)
0:08:49
Unit Tests in Python || Python Tutorial || Learn Python Programming
0:25:07
How To Write Unit Tests For Existing Python Code // Part 1 of 2
0:12:22
Unit Testing in Python using unittest framework - Basic Introduction and How to Write Tests
0:06:34
Machine Learning Tutorial Python - 7: Training and Testing Data
0:26:25
Unit Testing in Python
0:35:34
How To Write Unit Tests in Python • Pytest Tutorial
0:40:52
Programming a Hospital With Python! - Robospital [First Look] - Episode 1
0:04:05
Setting up unit tests in Python with VSCode
0:10:00
How Much FASTER Is Python 3.13 Without the GIL?
0:11:08
Writing automated tests for Python command-line apps and scripts
0:10:41
Python Unit Testing in 10 Minutes
1:59:40
Testing your Python Code with PyTest | Scipy 2019 Tutorial | John Leeman, Ryan May
0:10:04
Python unit testing - pytest introduction
0:15:10
Test-Driven Development In Python // The Power of Red-Green-Refactor
0:33:10
How Good Are You At Python? | Python Quiz/Assessment
0:25:33
Python TDD Workflow - Unit Testing Code Example for Beginners
0:01:43
How To Mock Patch A Function (Testing Python With Pytest)
0:16:02
Mastering Python - Everything You Need To Know To Become a Python Master
0:04:41
Python Tutorial: Write a simple unit test using pytest
0:20:07
Python for Data Analysis: Hypothesis Testing and T-Tests
0:07:32
Memoization: The TRUE Way To Optimize Your Code In Python
0:03:23
Mock Objects: Improve Your Testing in Python
Комментарии