Implement and Train ViT From Scratch for Image Recognition - PyTorch

We're going to implement ViT (Vision Transformer) and train our implementation on the MNIST dataset to classify images! Video where I explain the ViT paper and GitHub below ↓

Want to support the channel? Hit that like button and subscribe!

ViT (Vision Transformer) - An Image Is Worth 16x16 Words (Paper Explained)

GitHub Link of the Code

Notebook

ViT (Vision Transformer) is introduced in the paper: "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale"

What should I implement next? Let me know in the comments!

00:00:00 Introduction
00:00:09 Paper Overview
00:02:41 Imports and Hyperparameter Definitions
00:11:09 Patch Embedding Implementation
00:19:36 ViT Implementation
00:29:00 Dataset Preparation
00:51:16 Train Loop
01:09:27 Prediction Loop
01:12:05 Classifying Our Own Images

[uygarkurtai]

In order to use this code for images with multiple channels: change self.cls_token = nn.Parameter(torch.randn(size=(1, in_channels, embed_dim)), requires_grad=True) to self.cls_token = nn.Parameter(torch.randn(size=(1, 1, embed_dim)), requires_grad=True).

Thanks @Yingjie-Li for pointing it out.

[learntestenglish]

Thank you so much, a video that difficult to find on the internet again 👏👏

[BOankur]

Thank you for the tutorial! Great work.

[FernandoPC]

Hey Uygar,
Thanks a lot for the tutorial, you're like my coding sensei!
I was wondering about something while coding the ViT. Why do you define hidden_dim if you're not using it later on? Or maybe you are using it and I just haven't noticed?
Appreciate your help!

[federikky]

Hello, very good explanation, i'm wondering how can i visualize the attention map of the transformer?

[abrarluvrabit]

hi, thank you so much for this video i really need that for understanding the training of ViT, can you please make a video for Multiscale Vision transformer MviT and MviTv2 for training them from scratch. i really appreciate all your efforts for ML DL and CV society.

[ssrinivasan]

I am a tech person and want to jumpstart into ML. I would really appreciate it if you could begin with hardware requirement (whether a GPU is required or not). Also, the sequence of various packages required to be installed. Thanks.

[MrMadmaggot]

Thats cool man, your coding skills and how smooth you are coding that is even scary, maybe AI is not for me xdddd.

Anyways my question is here: You are using only one layer, what If i want to use multiple layers? 22:44 after encoder_layer should I add another encoder_layer_2 with different parameters?

[arturovalle]

could you implement a DiT ? difussion transformer?

[Yingjie-Li]

Hi, I get some advice for this code. I deal with the images which in_channels = 3. But your work can not fit the situation that in_channels = 3. I do some fix based your code. self.position_embedding = nn.Parameter(torch.randn(size=(1, num_patches + in_channels, embed_dim)), requires_grad=True) After that, the code can work in the in_channels = 3 images. HOPE YOUR REPLY! -China-Beijing

[Movies_Daily_]

Can u tell me which version of python, torch, sckit learn, and other used

[PheaKhayMSumo]

Hi, I am a student and I was wondering if I could use your code as my basis for developing my thesis which is centered in sorting ripe and unripe strawberries?

[gitgat-wxvq]

import torch
import maxvit
# from .maxvit import MaxViT, max_vit_tiny_224, max_vit_small_224, max_vit_base_224, max_vit_large_224



# Tiny model
network: maxvit.MaxViT =
input = torch.rand(1, 3, 224, 224)
output = network(input)


my purpose is to do give an input as an image (1, 3, 224, 224) and generate output as its description for that. how should i do that, what should i add more to this code?

[muhammadatique]

Can you please change the theme in white ? Its hard to see in black theme

[РодионЧаускин]

Walker Edward Miller Steven Miller Dorothy

[staffankonstholm]

Shouldn't x be first in x = torch.cat([x, cls_token], dim=1) ?