LLAMA-3.1 405B: Open Source AI Is the Path Forward

Meta's open-source Llama 3.1 models, including the impressive 405B, have caught up to GPT-4 levels in just 16 months. Learn about their capabilities, performance compared to other leading models, and how they can be run locally. Plus, get insights into their new agentic system, licensing updates, and a mention of Mark Zuckerberg's open letter advocating for open-source AI.

LINKS:

💻 RAG Beyond Basics Course:

Let's Connect:

Signup for Newsletter, localgpt:

TIMESTAMPS
00:00 Introduction to Meta's Lama 3.1 Models
01:05 Technical Details and Improvements
03:37 Capabilities and Use Cases
04:26 Comparisons with Other Models
06:09 Running the Models Locally
08:19 Agentic System and Tools
11:01 VRAM Requirements and Practical Considerations
12:53 Mark Zuckerberg's Open Letter on Open Source AI
13:48 Conclusion and Final Thoughts

All Interesting Videos:

[shaonsikder]

How do you edit your video? It's really interesting and smooth to see zooming following your pc cursor.

[pauluk]

For about a year now, I came to the realisation that the future of A.I. is open, not because of morals or that closed models are bad, but because of privacy, security and other concerns, especially as closed A.I. models could become very dangerous in the future if 2 or 3 corporations or governments control them, it would give them a massive advantage over everyone else.

Because of all that, I feel open source is the only way to go to level the playing field, and even thought that carries its own risk, I feel the risk is much lower than A.I. being controlled by so few people and I like to think that A.I. being open, we can make the right decisions on it together, as the risk to us all is the same so we have a collective impulse on getting it right.

I welcome what Meta is doing but we should remember, it's not fully open source in what you think of as open source, but it's close enough to do the job and we need counterweights to what the online closed models are doing, because long term, I think most of us will want to run A.I. at a local level on our own hardware because of privacy and security reasons, this will especially be the case as A.I. becomes better and more useful in more areas and once A.I. starts having long term memory so it can change and adapt to the user, I highly doubt most of us would feel comfortable with data going back and forth to a centralised service, and if that isn't frightening enough, wait till we have robotics around the house, these being run by online A.I. services would be a massive privacy invasion that would make the likes of Google and Facebook look tame in comparison, open source and locally run is the only way forward I think over the long run.

[ThomasConover]

What kind of hardware do I need to run a 405B model

[unclecode]

Thanks for the delightful review! We now know LLMs weren't OpenAI's moat. I wonder if GPT-3 was the pivotal moment for AI and "us, " or if it is now, the release of 405B Meta open weights model?

[MeinDeutschkurs]

It did not cach up. Far behind! GPT4o/Claude 3.5 sonnet: 70.000 tokens in, prompt: Write a summary by chapter, and you get it. Llama 3.1? Just outputs jibberish. (Tested M2 Ultra, 192GB unified RAM, unquantized 8B + Q4, 70b Q4) - in comparison, the 3.0 gradient version was able to, but it didn’t stop, it then hallucinated further chapters. Cannot test 405b, not enough VRAM.

[oecypher]

would be nice if they port it to android and make it work as an offline stand alone app

[azaph]

HELLO HOW MUCH MONEY AROUND I NEET TO FINE TUNING LAMMA 3 70 B IN GCP
PLSE

[henkhbit]

Hopefully we get multi modal LLama soon. 👋Meta. Thanks for the update 👍

[NX_Blocks]

Thanks for the walkthrough, quite helpful.

It's just hard to think that mr. Zuckerberg has the best interest of people at heart, given the impact of his contribution to the decrease in wellbeing of young people.

[StraussBR]

Calm down guys it is just out
I am very happy with this I also think in the furuetae all want our own model rather than sending our data into a third party that is too much trust is unsustainable model

[pylanookesh]

Great video as always ❤. GPU requirements is what I was looking for these models and you correctly mentioned about it.

[Jeerleb]

i heard you need close to 800 gigs of ram to run the 405

[MeinDeutschkurs]

Yo cannot use groq, because it is restricted to 16000 tokens. 128000 tokens with 16000 useable window. OMG! Disappointing!

[xspydazx]

Em I don't think you guys understand !
The code base did not change ... Hence it's only a finetuned model ! Or different settings ( they are playing with you and creating brainwashed AI models with bad settings ! And giving them to the public, whilst they enjoy a totally different settings and training set ! ... Unless the code base changes the model is the same !!
The most important is the correct settings !
Hence 8b is actually incorrect !
The internal layer count and hidden sizes are incorrect as they are not dividable to binary values, hence training is a bitch and often unstable ! Remebr they used the common crawl first .. now they have used some structured synthetic dataset generated using guardrails and used this dataset to DPO the common crawl, hence after guardrailed the model is highly unturned ..once you get into the Hidden codebase and tokenizer you will find the sub prompts (alao adding abother layer of guaedrailing ) hence making the model tough to get your response no mater the prime pronpt ! .
Changing the prompt or query that was given is summon t to intercepting the question and creating a new one ! It's a prompt in the middle attack ! I'm surprised nobody has exposed the hidden prompts in the llama tokenizer and hugging face library ! As well as the unsloth library and better transformer library as well as in the pretrained model !

[PeteDunes]

The quality of the 405B model is horrendous, it failed so many tests I threw at it, while both ChatGPT and Copilot performed way better.

[linklovezelda]

Could you please stop the discord sounds in your future videos? Thanks :)