How to Build Python Data Engineering Pipelines with Snowpark

Join Jeremiah Hansen, Principal Data Platform Architect at Snowflake, for a virtual hands-on lab that walks you step-by-step through the process of building Python data engineering pipelines on Snowflake using Snowpark Python. The video includes a brief overview of Snowpark and the benefits it offers by allowing users to work with Python, Scala, and Java natively in Snowflake.

00:00–Introduction
01:01–An Overview of Snowpark and Its Key Features
09:40–The Snowpark Python Quickstart Guide
10:33–An Overview of Topics to be Covered in the Hands-on Lab
14:11–Hands-on Lab Begins
15:01–Prerequisites for Participating in the Lab
17:16–Setting Up Snowflake
19:50–A Look at the Snowflake Extension
20:38–The Work Begins: Loading Data
26:41–Collecting Data from the Data Marketplace
30:44–Creating Views and Streams on Views
33:35–Creating a Python UDF
40:12–Building the First Pipeline
48:16–Creating a Second Stored Procedure to Populate Weather Data Table
51:38–Joining Data Together
53:53–The Deployment
55:35–Orchestrating the Stored Procedures
59:30–Running the Pipelines Incrementally
1:01:39–Viewing the Query History
1:06:25–Deploying Through a DevOps Pipeline using GitHub Actions
1:13:42—Reviewing What We’ve Done

To connect with Jeremiah Hansen of Snowflake:

❄ Join our YouTube community ❄

Learn how to build your application on Snowflake:

Join the Snowflake Community:

[sathwikemmadi]

I think this video is helpful but appreciate if you can send other videos for detailed explanation or online documentation.I am asking about the architecture it starts from 07:19

[joerozario]

If the raw files in csv format, is there anyway to infer the schema without explicitly defining the schema StructType.

[treyhannam]

Is there documentation on the toml file?

[bharathraj]

Do we need an external cloud storage account to implement this project