Tokenization in Natural Language Processing

preview_player
Показать описание
This is a very, very short introduction into what tokenization in NLP is; with an example in Python.

SUBSCRIBE for more materials on NLP 🤖

NLTK library installation:


Icons used in the video are under CC BY 3.0, links to the authors:

music: FreeBackgroundMusic, Alston & Ozone – LSDesigns
Рекомендации по теме
Комментарии
Автор

Make sure to see this video on stems and lemmas!

skillbits
Автор

timestamps:

0:10 what tokenization is
0:30 why tokenization is important
0:44 how to apply tokenization in python
0:50 NLTK library
1:16 sentence tokenization
1:28 word tokenization

skillbits