14 ways to filter your Spark DataFrame in Microsoft Fabric (Day 8 of 30)

Learn Apache Spark in Microsoft Fabric in the 30 days of September.

Spark is the engine behind both the Data Engineering AND the Data Science experience in Microsoft Fabric, so in September I'll be walking you through Apache Spark: what it is, why you should learn it, how to use it, and how it integrates into Microsoft Fabric.

No previous Spark knowledge is required, some basic Python would be useful!

#pyspark #microsoftfabric #apachespark

Here's the schedule:

Timeline
0:00 Intro
0:57 Reading data into DataFrame
2:10 Equal to, not equal to
4:13 Startswith, Endswith
4:42 Multiple conditions
6:05 If in list
7:30 String contains
8:07 SQL %LIKE% filtering
10:23 Other SQL-like filtering operations
11:23 Summary

--BROWSE MY OTHER FABRIC PLAYLISTS--

--LINKEDIN--

--ABOUT WILL--
Hi, I'm Will! I'm hugely passionate about data and using it to create a better world. I currently work as a Consultant, focusing on Data Strategy, Data Engineering and Business Intelligence (within the Microsoft/Azure/Fabric environment). I have previously worked as a Data Scientist. I started Learn Microsoft Fabric to share my learnings on how Microsoft Fabric works and help you build your career and build meaningful things in Fabric.

--SUBSCRIBE--
Not subscribed yet? You should! There are lots of new videos in the pipeline covering all aspects of Microsoft Fabric.

[adilmajeed]

How about doing the visualization using spark instead of charting through the toolbar.?

[ryan]

hey, a quick question: do you know if it is possible to export a table stored in your lakehouse, for example as a csv? I've done it using azure, writing a dataframe as a csv to the blob storage and through a logic app upload it to sharepoint. Is something like that possible using Fabric?