BigData|Parquet File processing with sparkSQL by Suresh

preview_player
Показать описание
DURGASOFT is INDIA's No.1 Software Training Center offers
online training on various technologies like JAVA, .NET ,
ANDROID,HADOOP,TESTING TOOLS , ADF, INFORMATICA,TABLEAU,IPHONE,OBIEE,ANJULAR JS, SAP...
courses from Hyderabad & Bangalore -India with Real Time Experts.
so that our Supporting Team will arrange Demo Sessions.
Ph:Call +91-8885252627,+91-7207212428,+91-7207212427,+91-8096969696.
  1. Durga Software Solutions
  2. durga
  3. durga sir
  4. durgasoft
  5. interview
  6. questions
Рекомендации по теме
BigData|Parquet File processing 0:29:12

BigData|Parquet File processing with sparkSQL by Suresh

Apache Spark SQL 0:03:43

Apache Spark SQL Datasource Parquet

Advantages of PARQUET 0:00:44

Advantages of PARQUET FILE FORMAT in Apache Spark | Data Engineer Interview Questions #interview

Improving SparkSQL Performance 0:14:27

Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and Parquet Reader

BigData |Sample data 0:20:44

BigData |Sample data processing with Spark SQL Part -1by Suresh

Read parquet files 0:15:19

Read parquet files on ONTAP - Apache Spark workload SparkSQL

Data Engineering Spark 0:04:33

Data Engineering Spark SQL - Tables - DML & Partitioning - Creating Tables using Parquet

Spark Reading and 0:11:28

Spark Reading and Writing to Parquet Storage Format

Spark SQL - 0:04:17

Spark SQL - DML and Partitioning - Creating Tables using Parquet

Big Data on 0:16:15

Big Data on Spark Tutorial for Beginners [Part 23] Spark - How to Read Parquet File | Great Learning

Spark SQL Data 0:12:18

Spark SQL Data Source Format #batch #realtime #dataengineering #funlearning #bigdata #pvdata

SPARK SQL - 0:01:37

SPARK SQL - CONVERT PARQUET FILE TO AVRO SCHEMA

Apache Spark in 0:03:20

Apache Spark in 100 Seconds

The Parquet Format 0:40:46

The Parquet Format and Performance Optimization Opportunities Boudewijn Braams (Databricks)

PySpark Tutorial : 0:04:29

PySpark Tutorial : Understanding Parquet

Apache Parquet and 0:14:33

Apache Parquet and Apache Spark | Spark interview question | Bigdata file format

Apache Cassandra Lunch 0:19:20

Apache Cassandra Lunch #56: Using Spark SQL Parquet Tables in DSEFS / DSE Analytics

Spark SQL vs 0:00:14

Spark SQL vs Dataframes #apachespark #spark #dataengineering

SPARK SQL - 0:03:46

SPARK SQL - CONVERT JSON FILE TO PARQUET FILE

How to use 0:10:14

How to use DataFrames and SparkSQL in Spark Tutorial

Types of Data 0:07:33

Types of Data file formats in Big Data supported by Apache spark - PySpark Interview Question

Hive Optimization: Row-Columnar 0:08:55

Hive Optimization: Row-Columnar File Formats, Compression - Big Data Analysis: Hive, Spark SQL,

Compare Avro and 0:02:03

Compare Avro and Parquet file formats!!

The five levels 0:03:00

The five levels of Apache Spark - Data Engineering

Комментарии
Автор

Great!
I am beginner on Spark and i would like to compute some joins with 4 CSV files so that at the end i would have one dataframe. File1 is about 8GB, File2 5MB, File3 5MB, File4 15MB, File5 150MB.

- I loaded the files, saved them as Parquet, read them again and finaly i created views (createOrReplaceTempView) of these dataframes.

- I joined dataframe_File1 with dataframe_File2 as merge1.

- Now i would like to save merge1 as parquet again, read it an do others merges and so on...


probleme:
- lunch the spark Job and the job hangs.


Any Suggestions?
PS: I am using a standalone cluster (1 master = 1 slave).
I did some join optimization concepts and configurations(shuflle.partitions, etc..)

Best regards.

djibb.
visit shbcf.ru