What Do Neural Networks Really Learn? Exploring the Brain of an AI Model

Neural networks have become increasingly impressive in recent years, but there's a big catch: we don't really know what they are doing. We give them data and ways to get feedback, and somehow, they learn all kinds of tasks. It would be really useful, especially for safety purposes, to understand what they have learned and how they work after they've been trained. The ultimate goal is not only to understand in broad strokes what they're doing but to precisely reverse engineer the algorithms encoded in their parameters. This is the ambitious goal of mechanistic interpretability. As an introduction to this field, we show how researchers have been able to partly reverse-engineer how InceptionV1, a convolutional neural network, recognizes images.

▀▀▀▀▀▀▀▀▀SOURCES & READINGS▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀

This topic is truly a rabbit hole. If you want to learn more about this important research and even contribute to it, check out this list of sources about mechanistic interpretability and interpretability in general we've compiled for you:

On Interpreting InceptionV1:

More recent progress:

Mapping the Mind of a Large Language Model:

Extracting Concepts from GPT-4:

Language models can explain neurons in language models (cited in the video):

Neel Nanda on how to get started with Mechanistic Interpretability:

More work mentioned in the video:

▀▀▀▀▀▀▀▀▀PATREON, MEMBERSHIP, MERCH▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀

▀▀▀▀▀▀▀▀▀SOCIAL & DISCORD▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀

▀▀▀▀▀▀▀▀▀PATRONS & MEMBERS▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀

▀▀▀▀▀▀▀CREDITS▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀

[RationalAnimations]

This topic is truly a rabbit hole. If you want to learn more about this important research and even contribute to it, check out this list of sources about mechanistic interpretability and interpretability in general we've compiled for you:

On Interpreting InceptionV1:



More recent progress:

Mapping the Mind of a Large Language Model:

Extracting Concepts from GPT-4:

Language models can explain neurons in language models (cited in the video):

Neel Nanda on how to get started with Mechanistic Interpretability:


More work mentioned in the video:

[dezaim]

As a mass of neurons i can relate to being activated by curves.

[cheeseaddict]

You guys shouldn't overwork yourselves 😭
17 minutes of high quality animation and info
Seems like Kurzgesagt got competition 👀

[theweapi]

Polysemanticity makes me think of how we can see faces in things like cars and electrical sockets. The face detection neurons are doing multiple jobs, but there is not a risk of mixing them up because of how vastly different they are. This may also explain the uncanny valley, where we have other neurons whose job it is to ensure the distinction is clear.

[CloverTheBean]

I really appreciate your simplification without dumbing it down to just noise.
I've been wondering about how neural networks operate. Not that I'm a student or trying to apply it for any reason. I just love having nuggets of knowledge to share around with friends!

[jddes]

This is something I've been going on about for a long time. The real prize isn't in getting machines to do stuff for us, it will be using and adapting the shortcuts they are able to find. We just need to learn...what they've learned.

[Nikolas_Davis]

The delicious irony, of course, is that AI started out as a research field with the purpose to understand our _own biological intelligence_ by trying to reproduce it. Actually building a practical tool was a distant second, if even considered. But hardly anyone remembers that now, when AI systems are developed for marketable purposes.

So, now that AI (kinda) works, we're back to square one, trying to understand _how_ it works - which was the issue we had with our own wetware in the first place! Aaargh!! But all is not lost, because we can prod and peek inside our artificial neural networks much easier than we can inside our noggins. So, maybe there is net progress after all.

[DanksPlatter]

these videos are perfect edutainment and its crazy how much detail goes in even to background stuff like sound effects and music

[guyblack]

"there will also be lots of pictures of dogs"

well count me the fuck in LET'S GOOOO

[loopuleasa]

"To understand is to make a smaller system inside your head behave similarly to a larger system outside your head."

[Jan]

5:18 But that's exactly what leads to extreme misjudgments if the data isn't 100% balanced and you never manage to get anything to be 100% balanced. With dogs in particular, huskies were only recognized if the background was white, because almost all training data was with huskies was in the snow.

[gavinbowers]

The sound design and the animation were incredible in this one!

[chadowsrikatemo]

Ok, that explanation of image generation (the one that made a snout inside a snout) was one of the best ones i've found yet, Good Job!

[smitchered]

Thanks for educating the general public about AI and its dangers, Rational Animations! Your animations keep getting better and I still listen to your video's soundtracks from time to time... thanks for all this effort you're pouring into the channel!

[RitmosMC]

These videos are incredible, going into so much detail and yet managing to not lose the audience on the way- it’s amazing! The animation, the style, the music, everything! This channel is on par with Kurzgesagt and others in the educational animations genre, and the fact that it doesn’t have millions of subscribers is criminal. One of the top 10 channels on the entire platform for sure. Keep up the incredible work.

[VictorMartinez-zfdt]

My dog was in this video and I couldn’t be more happy when I saw her!

[foxxiefox]

Babe wake up rational animations posted

[KrasBadan]

16:11 It reminds me of a video by Welch Labs that I watched recently. It was about how Kepler didcovered his laws.

Basically he had a bunch of accurate data about the positions of Mars at some points in time and he wanted to figure out how it moves.

He noticed that the speed at which it orbits the sun isn't uniform, it is faster in one half of the orbit and slower in the other, and he tried to take it into account.

What he did was assume that the orbit is a circle, and inside that circle there is the sun, the center of the circle and some point called equant, and all these 3 points lye on the same line.

The idea of equant is as follows: imagine a ray that rotates uniformly around the equant, find the point at which it intersects orbit. In that model, this point of intersection is where the Mars should be at this moment.

He had 4 parameters: distance from the center of circle to sun, to equant, the speed or ray and the starting position of ray. These 4 parameters can describe wide range of possible motions.

By doing lots of trial and error, Kepler fine-tuned these 4 parameters, such that the maximum error was just 2 arcminutes, a hundred times more accurate than anyone else.

This process of having a system that can discribe almost anything and tuning it to describe what you want is similar to how neural networks recognise patterns.

But after more optimisation and taking more things into account, Kepler came to the conclusion that the orbit isn't a circle.

He then tried tuning a shape that is similar to egg, but it was worse than his old model, so he added more parameters. He assumed Mars orbits some orbit that itself orbits an orbit, and after noticing one thing about angles in his data, he found perfect parameters, that happen to perfectly describe ellipse.

The discovery of elliptical orbit of Mars was the main thing that allowed Newtown to develop his theory of gravity.

This thing is similar to how, given enough data, by lots of trial and error neural networks can generalize concepts and find underlying laws of nature.

[danielalorbi]

This what the most absolutely delightful animation of a Neural Net I've ever seen, by far. Kudos to the visual and animation artists

[BluishGreenPro]

These visualizations are incredibly helpful in understanding the topic; fantastic work!