Microsoft Fabric: How to load data in Lakehouse using Spark; Python using the notebook

Microsoft Fabric: How to load data in Lakehouse using Spark. In this video I have used Python code on the notebook. Analyze using SQL, and Power BI Dataset. Load and Transform the data using Python into Microsoft Fabric Lakehouse
Microsoft has Microsoft Fabric, the next-generation Analytic Platform. Check out how to enable it on Power BI Service. How to start a Fabric (Preview) trial
The Microsoft Fabric Platform provides customers with a SaaS-ified, open, lake-centric(One Lake), full-featured data, analytics, and AI platform that meets all their data estate needs. Power BI, Synapse Data Warehouse, Data Factory, Spark, and Notebook all under one platform

00:00 Create Lakehouse
02:00 What are data and Spark setup
05:30 Load Data in Lakehouse using Notebook
10:30 Add an Additional column in DataFrame
12:30 Save files in Lakehouse in different formats
21:30 Test Data on Power Bi Service Report
22:30 Using Spark DataFrame

Microsoft Documentation link

GIT File link -

🔵 What is Microsoft Fabric
Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence. It offers a comprehensive suite of services, including data lake, data engineering, and data integration, all in one place.

With Fabric, you don't need to piece together different services from multiple vendors. Instead, you can enjoy a highly integrated, end-to-end, and easy-to-use product that is designed to simplify your analytics needs.

Don't forget to like, comment, and subscribe for more great content!
▶️Data:

---------
▶️Follow me on:
---------
▶️My Other Videos:

-~-~~-~~~-~~-~-
Please watch: "Microsoft Power BI Tutorial For Beginners✨ | Power BI Full Course 2023 | Learn Power BI"
-~-~~-~~~-~~-~-

[sanumpzha]

Tried and it went well.! thank you Amit

[joelluis]

Hi

Great video and I have 3 questions because I've started to learn about fabric this week and I would love to hear your comments to keep learning more about this

-Can I explore my table using pandas and all his methods? I thought that I had to use Spark lenguage to do that

- Why should I transform my table to a Delta Table if it's currently working fine from csv or it could be an SQL view from SQL serve ? I still don't understand the advantage of doing that..

-I saw many posts showing the transformation of flat files to delta files. But what if I have an sql view from SQL server .. We could also change that view to a delta table to improve or no make sense?

[samirsahin]

I love your fabric videos.

In this video we are pulling data from csv, transform it on notebook via pandas and creating table for sql so we can run sql queries on our data. Then visualize it on power bi. This is almost what I was looking for.

Next step how can we make sure refresh data in background?

If I create a scheduled refresh on PBI, I think it will only get data what is in table. It won’t check csv updateszz How can we create a refesh to make sure get most updated data from csv and python code refresh that gross column?

Second question related do datalakehouse. I’m on fabric trial. Can I access from this trial fabric workspace to my current premium workspace and manipulate data on my current pbi reports?

[faheemiftikhar]

Hi, really good video. I just had one question. I wonder what other ways are there to load data with similar approach, I mean in your case reading from your github repository. What other sources can I use to load the data. I mean like can i use jdbc or odbc

[hirenrami]

I think your videos are amazing! I have followed your instructions and managed to import data using an API and create a dataframe. However, I have encountered an issue where the column headers are not named as expected and have been automatically assigned. I am wondering if you have any suggestions on how to address this problem?

[wbezjds]

What if you want to execute that notebook from external editor like VS Code on your desktop? This method wouldn't work.

[Kulfi]

Great session.
Q:-To use python in this notebooks, what libraries are required Amit.

[parajf]

Now same table if I want to use in spark sql is it possible, how

[GenZSchool]

Amit Can weuse python in Microsoft fabric to read data from SFTP server?

[bulletkip]

is it possible to use python locally to send data frame or csv to lake house

[hirenrami]

One more question is I want to make my PowerBi dashboard interactive like I want to take input from user like API and use that API key in my pyspark and load the data accordingly can you suggest how this can be done ?

Thanks :)I have a question regarding my Power BI dashboard. I want to make it interactive by allowing users to input an API key, which will then be used in Pyspark to load the relevant data. Could you please advise on how I can achieve this?

Thank you very much!

[trone_tip]

Can you please create video on how to data pull from Excel file to fabric

[prachijain]

Hello Sir, I want to access SFTP Server data using Azure Notebook but for that I need to set Inbound Rule . As Notebook will have dynamic IP then how can we allow Notebook IP to access Port 22? Thank you in advance . Please advise

[faheemiftikhar]

Hi is there any way to integrate notebook with Github ?