filmov
tv
Train Machine Learning Model with SparkML (...and Python) | Hands-on tutorial
Показать описание
To build and train a Machine Learning (#ML) model with Spark is not hard. With this tutorial we will build a simple Binary Classification ML model with Spark. We will use Logistic Regression built-in Spark algorithm, and then evaluate it by getting performance metrics from the model.
There are some different from we do it in Scikit-Learn. Spark provides a built-in SparkML engine with rich #SparkML API which you can leverage to build your unique Machine Learning model.
In this tutorial we are using SparkUI v.3.2.1 with pyspark-shell.
The critical points you should pay your attention to is:
- Datatypes (DTypes)
- String Indexer and One-Hot-Encoding for categorical features.
- Vector Assembler.
All these parts are explained and demonstrated in details in this tutorial. Also, you will learn what is SparkContext and SparkSession (differences between them). Therefore you will be able to check Data schema and handle data types in Spark DataFrame, selected features within your data. As required for ML modelling, you will also learn how to split your data into train and test sets.
Here you also learn how to setup ML stages with Spark and build a custom ML Pipeline to build your Machine Learning Model with Spark.
At the end, you will learn hot to get model performance metrics, such as Precision, Recall, or ROC curve values.
The tutorial is prepared with Jupyter Notebook, using Python programming language, so all the steps are executed with #pyspark .
The content of the video:
0:00 - Intro
0:32 - Start of Hands-on with Jupyter Notebook
0:46 - 1. Import main dependencies for Spark and Python
1:14 - Theory: Spark Session vs. Spark Context
3:10 - 1. Continuing importing dependencies
3:28 - 2. Load External CSV data to Spark (as Spark DataFrame)
5:40 - 3. Train and Test splits
6:39 - 4. Check Data Types
8:27 - 5. One-Hot-Encoding with Spark
10:07 - Theory: StringIndexer and One-Hot-Encoer
11:01 - 5. Continuing with StringIndexer hands-on
12:19 - 6. Vector Assembling
12:55 - Theory: Vector Assembling in Spark
13:53 - 6. Continuing with Vector Assembling
15:24 - 7. Make Spark ML Pipeline
18:31 - 8. Train ML Model with Spark
20:07 - 9. Get Model Performance Metrics
Spark API and SparkML API method used in the tutorial (incl. documentation):
Thank you for watching!
Please subscribe this channel - @DataScienceGarage to get more high-quality videos about #DataScience , #Python , #AI , #MachineLearning , #DeepLearning and much more!
There are some different from we do it in Scikit-Learn. Spark provides a built-in SparkML engine with rich #SparkML API which you can leverage to build your unique Machine Learning model.
In this tutorial we are using SparkUI v.3.2.1 with pyspark-shell.
The critical points you should pay your attention to is:
- Datatypes (DTypes)
- String Indexer and One-Hot-Encoding for categorical features.
- Vector Assembler.
All these parts are explained and demonstrated in details in this tutorial. Also, you will learn what is SparkContext and SparkSession (differences between them). Therefore you will be able to check Data schema and handle data types in Spark DataFrame, selected features within your data. As required for ML modelling, you will also learn how to split your data into train and test sets.
Here you also learn how to setup ML stages with Spark and build a custom ML Pipeline to build your Machine Learning Model with Spark.
At the end, you will learn hot to get model performance metrics, such as Precision, Recall, or ROC curve values.
The tutorial is prepared with Jupyter Notebook, using Python programming language, so all the steps are executed with #pyspark .
The content of the video:
0:00 - Intro
0:32 - Start of Hands-on with Jupyter Notebook
0:46 - 1. Import main dependencies for Spark and Python
1:14 - Theory: Spark Session vs. Spark Context
3:10 - 1. Continuing importing dependencies
3:28 - 2. Load External CSV data to Spark (as Spark DataFrame)
5:40 - 3. Train and Test splits
6:39 - 4. Check Data Types
8:27 - 5. One-Hot-Encoding with Spark
10:07 - Theory: StringIndexer and One-Hot-Encoer
11:01 - 5. Continuing with StringIndexer hands-on
12:19 - 6. Vector Assembling
12:55 - Theory: Vector Assembling in Spark
13:53 - 6. Continuing with Vector Assembling
15:24 - 7. Make Spark ML Pipeline
18:31 - 8. Train ML Model with Spark
20:07 - 9. Get Model Performance Metrics
Spark API and SparkML API method used in the tutorial (incl. documentation):
Thank you for watching!
Please subscribe this channel - @DataScienceGarage to get more high-quality videos about #DataScience , #Python , #AI , #MachineLearning , #DeepLearning and much more!
0:22:42
Python for AI #3: How to Train a Machine Learning Model with Python
0:30:57
Build your first machine learning model in Python
0:05:01
All Machine Learning Models Explained in 5 Minutes | Types of ML Models Basics
0:02:35
Machine Learning Explained in 100 Seconds
0:02:39
TensorFlow in 100 Seconds
0:06:34
Machine Learning Tutorial Python - 7: Training and Testing Data
0:02:20
Why do we split data into train test and validation sets?
0:00:45
Train an image classification model with AutoML #AI #AutoML
0:15:32
Build a Weather Data Prediction App | ML/AI Project Tutorial
0:07:52
Machine Learning | What Is Machine Learning? | Introduction To Machine Learning | 2024 | Simplilearn
0:22:33
Train, Test, & Validation Sets | How to Train Machine Learning Models (Properly!!!)
0:12:36
Building and training ML models with Vertex AI
0:24:26
How To Train Deep Learning Models In Google Colab- Must For Everyone
0:19:58
Machine Learning in Python: Building a Classification Model
0:18:40
But what is a neural network? | Deep learning chapter 1
0:00:32
HOW much 💵💰💵 did Stable Diffusion COST to Train?
0:17:27
Machine learning systems primer: How to train ML models
0:37:05
Build Your First Machine Learning Project [Full Beginner Walkthrough]
0:00:28
Learning Machine Learning has never been easier #shorts #machinelearning #statistics #datascience
0:49:43
Python Machine Learning Tutorial (Data Science)
0:02:43
PyTorch in 100 Seconds
0:05:50
What are Transformers (Machine Learning Model)?
0:02:41
AI: Training Data & Bias
0:13:57
How is data prepared for machine learning?
Комментарии