Decoding hidden states of Phi-3 with LogitLens (Interpretability Series)

In this series we discuss interpretability and explainability methods for deep neural nets focussing on Microsoft's Phi3. In this episode we zoom into LogitLens, a simple method that allows to decode intermediate representations.

[foreignconta]

Interesting.
What happens when token embeddings weight is used instead of lm_head weight I wonder. 🤔