Understanding the Llama 3 Tokenizer | Llama for Developers

Aston Zhang, research scientist working on Llama at Meta discusses the new tokenizer in Meta Llama 3. He discusses the improvements made to the tokenizer in Meta's latest Llama 3 models. The new tokenizer uses Tiktoken instead of SentencePiece and has a larger vocabulary size of 128k, resulting in better performance on coding, reasoning, and more. The increased vocabulary size allows for more specific and nuanced encoding of inputs, while the higher compression ratio reduces the number of tokens required to represent an input. Additionally, the use of Group Query Attention helps balance out the increased memory and compute needs, resulting in a model that can process larger batches without increasing latency.

# Timestamps
00:00 Introduction
00:25 What's new in the Llama 3 tokenizer?
01:58 Vocabulary size and compression ratio
13:01 Performance, efficiency and improving costs
17:46 Recap and resources

# Additional Resources

#llama3 #llm #opensource
- - -

# Follow us on social media

Meta AI focuses on bringing the world together by advancing AI, powering meaningful and safe experiences, and conducting open research.

[loabrasumente]

TLDR

- from llama 2 to llama3 they switched from sentencepiece to tiktoken
- vocab size 32k -> 128k
- ~15% fewer tokens for english, ~50% fewer for "some other languages"

[parvesh-rana]

Aston please explain the attention mechanism, Actually I am stuck in the chapter "Attention and transformer" of your book d2l

[eziola]

Hey guys thanks for this great video. In your opinion, using, Python, what is the best approach to counting the tokens of my prompt to llama3.2?

[therobotocracy]

My instinct is tokenization is underestimate in importance. Usually the hard and boring fundamentals are where the magic happens. Across all fields.

[stephennfernandes]

could someone from the meta LLaMa 3 team please explain how to train my very own tiktoken tokenizer like you guys did for llama 3. there is no opensource steps to recreate this

[kaushilkundalia]

Is the Llama 3 paper out yet? He mentions it @ 24:02

[Sashvinth]

You develop Tamil language for Tamil users

[HamedSoheili-qr]

so this guy is payied to use open sourced tiktoken

[wryltxw]

i don't think this format works unless the intent is to discuss at a high level.

[inteist]

Classic example of a provably smart guy not being able to express his thoughts... 5 minutes of pain is all I managed to force myself to watch. A shame.