A Walkthrough of Toy Models of Superposition w/ Jess Smith

This walkthrough mostly focuses on high-level ideas and themes, let me know if you want a part 2 that finishes going through the rest of the paper in detail!

(We had some audio and connection issues as we recorded this, sorry for any disruptions to the viewing experience!)

00:00 Toy Models of Superposition
03:53 Overview
10:20 Polysemanticity vs Superposition
24:05 Feature Importance
34:32 Large Dimensional Spaces
37:41 Interference vs Internal Reprsentation
42:02 Superposition in Toy Models vs. Real Transformers
50:46 Activation Functions and Interference
59:25 Internal Reprsentations of Features
01:22:54 Definitions of Features
01:42:47 Sparsity Diagram
01:44:16 Simulating Bigger Models
01:48:23 A Hierarchy of Feature Properties
01:51:42 Experimental Setup
01:59:18 Experimental Results
02:18:19 A Mathematical Analysis
02:25:22 Final Takeaways

[YoannPoupart]

Really cool walkthrough! Amazing insights. I would love to have a walkthrough of your Grokking paper and notebook.

[yucheng]

This video is so cool and valuable. Would be amazoning to have 1080p for the future ones.

[KevinKreger]

The limits of F32 are not a distraction. Thank you. I think high-freq/low-freq neurons are relative to textures.

[BuFuO]

@14:50 After I watched 3b1b latest video about superposition into transformers, I kinda feel those scalar projections onto feature vectors seems under-represented in your drawing.

[thecactus]

Why did you only cover like the first fifth of the paper?

[einsteinsapples]

Am I the only one who thinks there are no graphs in models? There are numbers, so explaining it with graphs/dimensions just makes it confusing for me.

[einsteinsapples]

Like you even say yourself how we don't have an intuition for high dimensional spaces, so why stick with the idea that we are dealing with a space. There is no physical space anywhere. It's just an idea, and for me it makes stuff more confusing then if you would just talk about the numbers in the vectors.