StructType & StructField | Create DataFrame | Nested Schema|Spark Interview question| Spark tutorial

We are making a complete collection of 2019 Spark interview questions and Apache Spark tutorial.This video is an addition to the collection. In this video we talk about Apache Spark StructType and StructField.

StructType is a built in data type and useful for creating customzed schema,nested schema and eliminaring order dependencies in dataframe transformation.

Struct is the field in StructType with parameters
Name
Data Type
Nullable flag
Metadata.

In this video we talk about below points:

StructType

StructField

Add structfields to StructType
a) by passing seq of StructField and b) using various add methods, c) From existing StructType

Compare two schemas

Create dataframe with own customized schema

Create nested schema

Flatten nested schema.

About channel
---------------------
We are trying to make video series on technical interview questions on various skills for experienced and freshers. This is based on the industry experience. We are trying to compile all the

possible questions with answers and give you the core concepts of the topic.

We will present a series on tutorial on various skill sets as well. Our first tutorial will be on Apache Spark and we will present it soon.

Our aim is to make a difference in your learning.

Please subscribe to our channel and stay connected.

Thank you so much.

Our video links
________________

------------------------------------

------------------------------------

--------------------------------

----------------------------------------------------------------------

----------------------------------------------------------------------

---------------------------------------------

-------------------------------------------------------------

-----------------------------------------------

----------------------------------------------

---------------------------------------------

----------------------------------------------

-----------------

------------------------------------

[jamesang]

Hi thanks for the video.. How do I set nullable = false?

[ANUKARTHIM]

Thanks for sharing the knowledge with us bro. As you commented, will create a separate video for most of the content missed like order dependencies using struct type, tungsten & catalyst, parquet encoding, compression, and many.. kindly create and upload.
Your videos are more informative

[abhishekbarnwal]

the contents of your videos are really rice. can you please make a video on usage of CASE CLASS for defining scheme. Really appreciate your efforts for the hard work you are putting. Thanks !!

[vishalmishra]

have given schema in excel sheet and diffreent records in text file following different schemas .so how should i check that record is matching with paricular schema in spark using scala and separate into each file