Word and Sentence Tokenization Explained | NLP Concepts for Building AI Applications

preview_player
Показать описание
This video titled "Word and Sentence Tokenization Explained | NLP Concepts for Building AI Applications" explains Word and Sentence Tokenization in detail, with the help of a code example. Word and Sentence tokenization is a very important NLP concept for building text-based AI applications because the corresponding ML or DL model needs to understand the individual word or sentences before providing predictions.

------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------

******Links of Kindle & Machine Learning, Deep Learning & AI Books******

FOLLOW ME ON:

About this Channel:
The AI University is a channel which is on a mission to democratize the Artificial Intelligence, Big Data Hadoop and Cloud Computing education to the entire world. The aim of this channel is to impart the knowledge to the data science, data analysis, data engineering and cloud architecture aspirants as well as providing advanced knowledge to the ones who already possess some of this knowledge.

Please share, comment, like and subscribe if you liked this video. If you have any specific questions then you can comment on the comment section and I'll definitely try to get back to you.

#WordTokenization #SentenceTokenization #NaturalLanguageProcessing
Рекомендации по теме
Комментарии
Автор

When is it preferable
to remove punctuation? After doing tokenization or before doing tokenization.

TheAIUniversity
Автор

After tokenization, because if we remove the punctuation before, then tokenizer would fail to understand real context to tokenize the words as well as sentence

siddhantrai
Автор

Punctuation must be removed before tokenization

SpoonfulOfRespect