How to Use Snowpark Dataframes in Python

In this episode, we're going to introduce the Snowpark dataframe. In our previous episode, we showed how to get started with Python on Snowpark, so this episode is a follow up to show one of the main building blocks of Snowpark, the dataframe. Dataframes are extremely handy when handling data in Snowpark because they allow you to do operations while keeping data in Snowflake and not needing to import it locally to do processing. It also allows you to filter and select columns, save as pandas dataframes, or to new tables on Snowflake, all of which I demonstrate here. Let's go!

Related Videos:
How to Get Started with Snowpark for Python on Snowflake
How to Use Snowpark Dataframes in Python
You are watching this one now!
How to Run Python User Defined Functions in Snowpark
How to Create Python Stored Procedures in Snowpark

Join me on Patreon!

Want the code from this video?

Interested in transforming your career or finding your next gig?

Want my team to do a project for you? Let's get to it!

Follow me on social media:

Get Microsoft Office including Access:

Got a YouTube Channel? I use TubeBuddy, it is awesome. Give it a try:

#snowpark #snowflake #python

[cyrusb]

Thanks. This is a great video for those who want to start learning Snowpark.

[justinhalaby]

Hi Sean, thank you for your videos, they are great. I have one question regarding the use of toPandas. Does the transformations continue to use the snowflake engine and uses the snowflake magic or every operation will be executer like on a normal computer using pandas?

[subramanyams]

Thanks for the nice videos. Could you please make more videos on SnowPark, please.

[dextermorgan]

I've really been looking for beginner level of Snowpark videos :D ! Thank you a lot !
I still have hard time understanding how we can benefit from Snowpark- as far as data cleaning...! We could use Pandas after converting the data from Snowflake. Is it because Snowpark does all data wrangling in snowflake?

Anyways thanks for the great contents ! :)

[anantharamaniyer]

Hi Sean. I managed to answer my question by reading the docs 😅. I had another question and wondered whether you know the answer to it.
Is Snowpark python going to perform better than a CTE in sql ? Say I have a long chain of CTEs which are difficult to debug and read unless you dissect each CTE, is it worth transforming the same to Snowpark python statements ? I wondered if performance would take a hit although python is easier to document, maintain and debug.

[anantharamaniyer]

Thanks Sean, I used your video to query a table and apply string as well as time filters and they work as expected. How can we split a column (say 'name') based on a string limiter (say 'AAA') and get the first element of the split?
I tried

from snowflake.snowpark import functions as SF
df.select(SF.split_part("ut_sw_name", SF.lit("-AAA"), 0).as_("test")).show()

but get an error message saying - 'AttributeError: module has no attribute 'split_part''
Any thoughts?

[SudarshanSelvamani]

can anyone help me to the length of the snowpark dataframe