[ML News] Llama 3 changes the game

Meta's Llama 3 is out. New model, new license, new opportunities.

References:

Links:

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n

[tantzer]

“If you don’t know what I’m talking about — and I don’t know why you wouldn’t…” I don’t know it because you’re my main source for important developments in machine learning.

[YuraCCC]

I had my doubts about Zuck, but check him out now—championing open source AI like a boss! Maybe he should just grab the name 'Open AI'—that is, if nobody's snagged it yet

[olcaybuyan]

I am especially happy that Llama 3 supports multiple languages :-) Most open access or open source models are English only and no real alternative to OpenAI GPT.

[GuagoFruit]

The next revolution imo definitely needs to be getting things to run locally with any sort of fidelity.

[vladimirtchuiev]

Mixture of Depth is a promising direction in modularizing LLMs, you could basically use only part of the model for specific applications

[woolfel]

There a numerous papers about data quality and data selection going back to 2000. Good to see people realize quantity is not the "end all" of training LLM. Creating a good dataset has always been an art. Will the filters and pipeline for processing the data get open sourced?

[FIicybZraC]

6:23 "There is enough research to show that once you are capable at one language, you only need quite little data on another language to transfer that knowledge back and forth"

Does anyone give me related papers to this argument? I am interested in cross-lingual transfer in language models.

[propeacemindfortress]

As always the best curated ML news.
Love your expertise and humor :D
oh, and... more fish for Yann LeCat!

[thirdeye]

Udio is great in my opinion. You don't let the AI create whole songs, but segments (of around 33s). It usually creates 2 variants at the same time. You then can extend those segments (before or after); either by a midsegement, intro or outro. You can even insert your own lyrics and it works like a charm. If you are happy with the song, you then can "publish" it and even pick a text-to-image cover art. I love that stuff.

[YuraCCC]

Those t-shirt stripes are an example of reverse CAPTCHA - it spins humans right into dizziness and blackout, but AIs? They just keep watching and learning.

[mikayahlevi]

The anti-open source AI safety person impression at 13:48 is too accurate🤣

[marsandbars]

20:40 An alternative to this is using a documented SDXL Turbo workflow with ComfyUI locally, which can produce images of decent fidelity at even faster speeds than this demo, at least on my 3090.

[pietrorse]

if the training text would be plain ascii, and average token length 4 characters, the training dataset would have been ~ 55 terabytes plain ascii. wow!

[Voljinable]

I really like the way you frame meta/zuck making llama 3 open source. They choose the option that is best for the company, but whats best changes. For research and optimization an open source model is better. For profit a closed source one is better.
What they do just depends on what is best at the moment, but i like that its open source for llama 3 right now and hope it will stay that way!

[Nico]

Cool to have great open LLMs. Unfortunately, this is not the case for image generation models: all the recent advanced models like SDXL or Photoshop are not commercial free ones.

[Iaotle]

This channel is one of the rare ones that I genuinely watch, amidst hours and hours of clicbait recycled AI hype videos :)

[pawelkubik]

I'm not sure why people have reservations about Phi specifically. We don't know what data were used to train the other models and to what extend their performance rely on "fitting to the test dataset". Did OpenAI ever admit what role does the human-curated part of their training dataset play in the model's performance?

[sebastianp]

we are getting to model sizes where they might as well just be compressed lookup tables

[pablowentscobar]

Really enjoy these ML News vids. Great for keeping AI normies like me up to speed.

[propeacemindfortress]

I tried to use existing llm's to prepare a fine tuning dataset specific on theravada thought, philosophy and practice, turned out that all models I tried were incapable of capturing any nuances in the meaning of words and concepts but stuck diligently to "their own philosophical framework of interpretation" regardless of the different system prompt, regardless of feeding scriptures, papers or video transcripts, they couldn't even identify the proper questions, so please don't mind me on disagreeing that language alone, maybe even regardless of percentage distribution, doesn't cut it on any task that require cultural, philosophical or religious understanding... not even talking about the human component in it... translation ofc is a totally different thing, used phrases and stuff can be captured quite well... the underlying unspoken human component not so much.