Delta Lake for apache Spark | How does it work | How to use delta lake | Delta Lake for Spark ACID

Spark Programming and Azure Databricks ILT Master Class by Prashant Kumar Pandey - Fill out the google form for Course inquiry.
-------------------------------------------------------------------
Data Engineering using is one of the highest-paid jobs of today.
It is going to remain in the top IT skills forever.

Are you in database development, data warehousing, ETL tools, data analysis, SQL, PL/QL development?
I have a well-crafted success path for you.
I will help you get prepared for the data engineer and solution architect role depending on your profile and experience.
We created a course that takes you deep into core data engineering technology and masters it.

If you are a working professional:
1. Aspiring to become a data engineer.
2. Change your career to data engineering.
3. Grow your data engineering career.
4. Get Databricks Spark Certification.
5. Crack the Spark Data Engineering interviews.

ScholarNest is offering a one-stop integrated Learning Path.
The course is open for registration.

The course delivers an example-driven approach and project-based learning.
You will be practicing the skills using MCQ, Coding Exercises, and Capstone Projects.
The course comes with the following integrated services.
1. Technical support and Doubt Clarification
2. Live Project Discussion
3. Resume Building
4. Interview Preparation
5. Mock Interviews

Course Duration: 6 Months
Course Prerequisite: Programming and SQL Knowledge
Target Audience: Working Professionals
Batch start: Registration Started
Fill out the below form for more details and course inquiries.

--------------------------------------------------------------------------
Best place to learn Data engineering, Bigdata, Apache Spark, Databricks, Apache Kafka, Confluent Cloud, AWS Cloud Computing, Azure Cloud, Google Cloud - Self-paced, Instructor-led, Certification courses, and practice tests.
========================================================

SPARK COURSES
-----------------------------

KAFKA COURSES
--------------------------------

AWS CLOUD
------------------------

PYTHON
------------------

========================================
We are also available on the Udemy Platform
Check out the below link for our Courses on Udemy

=======================================
You can also find us on Oreilly Learning

=========================================
Follow us on Social Media

========================================

[ScholarNest]

Want to learn more Big Data Technology courses. You can get lifetime access to our courses on the Udemy platform. Visit the below link for Discounts and Coupon Code.

[niteshshet]

The way you explain things and topic is rediculously good sir Thank you

[shibashishroy]

🙏🙏 your ability to explain with real time demo is awesome.

[bharath]

The way you explain things is amazing. This helped me a lot. Keep up the great work!!

[NenuNaAdhya]

You are a legend, u explained such a complex concept in a damn simple way that too under 30 min...commendable !!

[hemanttoday]

Outstanding work. Most impressive part is the ability to explain.

[BrijeshSingh-tbhx]

sir, your teaching style is really unique and awesome.. you are a real guru.. I have watched your Kafka and Spark videos and learned a lot. you answer all those questions which come up to everyone's mind while learning like why and how questions..
my pranam to you..

[lukefeng]

Great video and very-well explained on Delta Lake, thank you

[BharCode]

You ARE A GEM! So nice and crystal clear like even a child can understand if they know the basics of s/w. Bought your course of Kafka Streams on Udemy! Your videos are just perfect me. Bcz I have special bond if they are made with my Indian Accent! Makes me feel home, like somebody from the family is teaching me! :D Thanks a ton for making such videos.

[debanjanbose]

Thanks for such an informative and simple video on Delta Lake. It clears my all the basics.

[mahammadshoyab]

Prasanth is bigboss for bigdata.
If you explain databricks scaling, it will be much useful.

[luckyvaliin]

Amazing video sir, the topic got crystal clear for me... Appreciate your knowledge sharing.

[JD-xdxp]

Excellent video, explained in a simplify manner, we need this kind of instructor who can teach in a layman terms, good job

[nitinware]

Simple easy to understand presentation, like it, keep them coming.

[vidhyalakshmiparthasarathy]

Hi Sir - This is such a great explanation 👍. Thank you for posting the videos. 🙏

[KoushikPaulliveandletlive]

Brief and informative as always.
I havent used databricks yet. Here are some of my asumption after watching this video.
1. Deltalake keeps multiple version of the data( like HBASE ) .
2. Deltalake takes care of the automicity for the user showing only the latest file if not specified otherwise.
3. Deltalake checks the schema before appending to prevent corruption of the table, this makes developers job easy, similar things can be achieved with manual effort like manually mentioning the schema instead of infering it.
4. In case of update it always overwrites the entire table or the entire partition(dataframes are immutable) .
Questions.
1. If it keeps multiple version is there a default limit of number of versions ?
2. As it keeps multiple versions so is it only for smaller tables ? for tables in terabytes wont it be a waste of space?
3. The log file maintains log file per table or per partition? as I understand having log file for each partition will give option to keep multiple version of only selected partitions hence saving space.
4. As Deltalake works with parquet and I believe like ORC, parquet also keeps the metadata ( min, max etc. ) with each part file, so while updating the table does it skip the part files where updates didnt happen ?

Update:
Deltalake is just amazing .
It minimizes pipelines with 100 steps to may be 20 steps or less. It also helps combine multiple pipelines into one.

Cant thank you enough Prashant for this wonderful demo.
and as he says " Keep learning keep growing" If you dont get time in your busy schedule leave your job for few months, when you join back in some other company you will definitely get a much better role. Your courage will be well paid.

[pradeeshma]

Cant thank you enough for this. EXCELLENT!!!

[tavneetsingh]

:) Wish we got more sessions like this often...Thanks Learning Journal

[ladakshay]

Awesome video, your explanation is superb, you already answer the questions coming into our mind :)

[sumantabasu]

Very clear explanation, thanks a lot.