why llama-3-8B is 8 billion parameters instead of 7?

llama-3 has ditched it's tokenizer and has instead opted to use the same tokenizer as gpt-4 (tiktoken created by openai), it's even using the same first 100K token vocabulary.

In this video chris walks through why Meta has switched tokenizer and the implications on the model sizes, embeddings layer and multi-lingual tokenization.

he also runs his tokenizer benchmark and show's how it's more efficient in languages such as japanese

repos
------

[charbakcg]

Excellent demonstration Chris, thanks for sharing!

[goodtothinkwith]

Great stuff.. no nonsense presentation style, clear and technical, as it should be 😅.. question: is there a reason why it’s not better to have common English syllables in the vocabulary? I understand “lov” being there, but I can’t imagine that “el” is a very useful token as part of “Lovelace”.. intuitively, I would think that is should simply be tokenized as “love” and “lace”

[aaravsethi]

Im super excited to see the `llama.cpp`, `llama2.c`, etc. category be implemented for llama3!

[rluijk]

ok, that is all very concrete! Awesome. Thanks for this. This seems like a lot of quick wins that are easy to discover, or is that because hindsight by you explaining it so clearly? Anyway, its all a bit new to me. Perhaps, lets say Norway, would be wise to run this with their own tokeniser? Or is that to simplistic thinking?

[leeme]

What are you thought on including space in the tokenizer? I tried it once and the LLM was optimising to predict spaces as those easy wins for the LLM, but I like the way tiktoken has done to keep the space but not space as a token on it own....

[rogerc]

Why is there some pytorch? Does finetuned or merged versions need it?