Data Journey EP-02: Batch Ingestion 📦 - 5 ways to ingest files into Google Cloud

preview_player
Показать описание
Data Journey EP-02 Batch Ingestion - 5 ways to ingest files into Google Cloud

Intro: (00:00)
Data Ingestion Challenges: (00:39)
Data Ingestion Design: (01:32)
Ingestion Option-1 : Cloud Console: (05:42)
Ingestion Option-2 : GSUTIL: (06:17)
Ingestion Option-3 : Storage Transfer Service: (07:45)
Ingestion Option-4 : BigQuery Transfer Service: (08:55)
Ingestion Option-5 : Storage Transfer Appliance: (09:48)
Summary: (11:08)
What’s next?: (13:07)
  1. Arpit Agrawal (Elastiq.AI)
  2. Google
  3. Cloud
  4. Platform
  5. GCP
  6. Data
Рекомендации по теме
Data Journey EP-02: 0:13:28

Data Journey EP-02: Batch Ingestion 📦 - 5 ways to ingest files into Google Cloud

Adobe Experience Platform 0:05:31

Adobe Experience Platform - Batch ingestion

Batch framework(Data Ingestion 0:07:49

Batch framework(Data Ingestion Framework that process 1TB of data in a Month and run 200+ Jobs)

Data Journey EP-01: 0:07:25

Data Journey EP-01: Introduction 📈 - What to expect and Why Google Cloud?

What is Data 0:01:50

What is Data Ingestion in 2021 ?

02 Dataflow - 0:15:11

02 Dataflow - Cloud Storage to Big Query Batch Job

Hassle-Free Data Ingestion 0:38:19

Hassle-Free Data Ingestion into the Lakehouse

Tutorial 4-End To 0:26:33

Tutorial 4-End To End ML Project With Deployment- Data Ingestion Implementation Line By Line

Building stream processing 0:15:17

Building stream processing pipelines with Dataflow

Extract data from 0:13:06

Extract data from GCS and load to Bigquery using Dataflow and Python

Unify batch and 0:19:23

Unify batch and stream data for real-time intelligence at Google scale

Ingestion Job - 0:19:23

Ingestion Job - Ingest Data to GCS bucket from Compute Engine (Virtual Machine)

Data Ingestion 101 0:06:05

Data Ingestion 101 | Explained in 5 Minutes

Batch based data 0:03:57

Batch based data ingestion

Data Ingestion with 0:08:09

Data Ingestion with SQL using Google Cloud Dataflow - GCP Project | Overview of Architecture

Loading data into 0:05:31

Loading data into BigQuery

Azure Data Ingestion 0:10:51

Azure Data Ingestion Services

How to build 0:16:09

How to build a stream and batch processing Job On GCP Dataflow || DataFlow Tutorial

Google Cloud Platform 0:08:25

Google Cloud Platform | On-Premise to GCP Data Ingestion Pattern

Spark Structured Streaming 0:15:43

Spark Structured Streaming as a Batch Job? File based data ingestion benefits from pseudo streaming?

2. GCP Data 0:19:29

2. GCP Data Transfer services for Data Engineering

Dealing with massive 0:54:12

Dealing with massive data ingestion

Data Ingestion from 0:05:35

Data Ingestion from Oracle to Hadoop

Data ingestion on 0:10:11

Data ingestion on the cloud: fault-tolerance and performance

Комментарии
Автор

any mechanism to move files from GCP (GCS Storage) to another Cloud Provider like Azure ?

mulshiwaters
Автор

thank you very much very well explained

loufoua
Автор

awesome content..thanks for such videos

baldevsingh-bhj
Автор

Great explanation..really to the point.. expecting more videos please

vis
Автор

when will the next video on streaming? very useful for me.

MrJohnloose
Автор

Amazing explanation for beginners, thanks a lot for this informative video!

jiyasingh
Автор

Great content keep it up. But sort the echo please.

muzahmad
Автор

Brilliant content. Would love if more videos' on same series would come often.

karan
Автор

Hi sir, I want to ingest data, using an API from a 3 party vendor into GCP. The data comes once in a month. Can you suggest me a way to ingest data to GCP and which services should I use to ingest data and need a Scheduler or a trigger that run automatically after 30 days

jayantmathur
Автор

Very helpful video.

I just have a question. These techniques look like they are more for data upload, and not really ingestion. When we talk about "ingestion", we usually are referring to data pipelines, DataFlow, DataProc, Cloud SQL, BigQuery, etc. And as far as "streaming ingestion" is concerned, Pub/Sub is probably the first thing that comes to our mind.

I fully agree that there are no hard and fast rules and its perfectly ok to call copying data to Cloud Storage as 'ingestion'.

anandakumarsanthinathan