Sentence Tokenization in Transformer Code from scratch!

ABOUT ME

RESOURCES

PLAYLISTS FROM MY CHANNEL

MATH COURSES (7 day free trial)

OTHER RELATED COURSES (7 day free trial)

TIMESTAMP
0:00 Dataset Source
2:39 Alpha Syllabery Explained
5:33 Reading & Processing Sentences
8:43 Pytorch Dataset & TextDataset
10:13 Batching Sentences
12:05 Character to Number Encoding
14:50 Masking
18:15 Creating a Class

[arturasdruteika]

Always nice to see your explanations of how ML stuff works. I was wondering if you are going to do a detailed explanation on how the vision transformers work and intuition behind it?

[MrGss]

Wonderful work and clarity....as a senior ai practitioner and fellow kannadiga do accept my appreciation....and keep going

[nasdaq_]

You are explaining better than many uni professors! Keep going!

I was just wondering, in the case of embedding a sequence of immagines, how can they be tokenized? Since they don't have a finite character rappresention like text, is that possible?

[GaneshBhat]

Thanks for the video. Can you please explain the create_mask function in a bit more detail on what is it actually doing?

[DSISSuhaasG]

Absolutely loved the video!! I wanted to follow along but the dataset seems to be unavailable at the moment. Can you suggest any alternatives?

[neetpride]

What type of embedding is best for making a chatbot that talks like me: character level, phoneme level, or word level? Also is the attention window just another way of describing the length of n-grams?

[ajaytaneja]

Hi Ajay, please can I ask, why have you not spoken about embedding vectors here? You talk about tokenizer, are embedding vectors not a part of your Transformer code? Thanks again for this marvellous series. Regards, Ajay

[aomo]

Can y make a video about using transformer for video classification ?

[cuckoo_is_singing]

hi
why did you choose different padding number for encoder and decoder?
can I choose zero for both padding numbers?

[amithajith]

Do we need to do word embedding and possitional embedding

[jusoijg]

Are the tokens which you matched with every character before the training randomly initialized?

[DLwithShreyas]

A question, I did this on a custom language (Marathi) but now I want to create a hugging face tokenizer out of this? What should I do ?

[sidbhattnoida]

I think the start token, padding token and end token should have some different name other that just empty string in the vocabulary as otherwise while initialising a language_to_index dictionary from vocabulary the last index having the value of empty string is overwriting all the previous indices with empty string.

[EducatedButton]

Can you please have a new video on how chatgpt works by accessing internet

[AyushRaj-ntot]

Can anyone plz tell me how to make this eng-hindi translator. where do i get the alpha-syllables list for hindi?

[gagangayari]

In 11:02 you mentioned we calculate one error value for 3 samples. But for 3 samples we will be getting 3 losses right. Do you sum up those 3 values or take average ?

Also is this technique called the gradient accumulation method or is it just batching ?

[punk]

"Hes a scientist? :D :D :D Who translated this??? :D

[Abdullahkbc]

hey i dont get what you mean in 6:29. why do you convert every single character rather than word? i think embeddings are for token/words rather than characters. could u pls make this clear?