Scalable MatMul-free Language Modeling (Paper Explained)

Matrix multiplications (MatMuls) are pervasive throughout modern machine learning architectures. However, they are also very resource intensive and require special accelerators (GPUs). This paper explores architectures that do away with MatMuls and use quantization and recurrence to keep performance up.

OUTLINE:
0:00 - Intro
2:30 - MatMul is everywhere
5:55 - Ternary accumulation as a substitute for matrix multiplication
16:35 - Replacing attention layers with recurrent layers
32:40 - Replacing dense layers with ternary channel mixing
38:30 - Language modelling results & scaling laws
45:00 - Other experimental results
48:20 - Conclusion

Abstract:
Matrix multiplication (MatMul) typically dominates the overall computational cost of large language models (LLMs). This cost only grows as LLMs scale to larger embedding dimensions and context lengths. In this work, we show that MatMul operations can be completely eliminated from LLMs while maintaining strong performance at billion-parameter scales. Our experiments show that our proposed MatMul-free models achieve performance on-par with state-of-the-art Transformers that require far more memory during inference at a scale up to at least 2.7B parameters. We investigate the scaling laws and find that the performance gap between our MatMul-free models and full precision Transformers narrows as the model size increases. We also provide a GPU-efficient implementation of this model which reduces memory usage by up to 61% over an unoptimized baseline during training. By utilizing an optimized kernel during inference, our model's memory consumption can be reduced by more than 10x compared to unoptimized models. To properly quantify the efficiency of our architecture, we build a custom hardware solution on an FPGA which exploits lightweight operations beyond what GPUs are capable of. We processed billion-parameter scale models at 13W beyond human readable throughput, moving LLMs closer to brain-like efficiency. This work not only shows how far LLMs can be stripped back while still performing effectively, but also points at the types of operations future accelerators should be optimized for in processing the next generation of lightweight LLMs. Our code implementation is available at this https URL.

Authors: Rui-Jie Zhu, Yu Zhang, Ethan Sifferman, Tyler Sheaves, Yiqiao Wang, Dustin Richmond, Peng Zhou, Jason K. Eshraghian

Links:

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n

[KostyaCholak]

Loved that references for BitNet are 10 and 11

[eoghanf]

Your point about estimating whether non-straight lines cross based on three datapoints is a very good one. HOWEVER, the reason for giving them the benefit of the doubt on the training dynamics side is that the *inference* time power efficiency gain (which you don't spend any time on!) is massive. From the abstract "We processed billion-parameter scale models at 13W beyond human readable throughput, moving LLMs closer to brain-like efficiency". That's pretty amazing.

[wolpumba]

*Summary*

*Problem:*

* *(**2:30**)* Matrix multiplications (MatMuls) are the core of modern machine learning, but they are resource-intensive and require specialized hardware like GPUs.

*Proposed Solution:*

* *(**0:00**)* This paper proposes eliminating MatMuls entirely from large language models (LLMs) while maintaining competitive performance.
* *(**16:35**)* The architecture replaces:
* *(**16:35**)* *Attention layers* with parallelizable recurrent layers inspired by GRUs.
* *(**5:55**)* *Dense layers* with "ternary accumulation, " using quantized weights limited to -1, 0, and 1. This replaces multiplication with simpler selection and addition operations.

*Key Findings:*

* *(**38:30**)* *Performance:* The MatMul-free models perform on par with state-of-the-art Transformers at scales up to 2.7 billion parameters.
* *(**38:30**)* *Scaling Laws:* The performance gap between MatMul-free models and traditional Transformers seems to decrease with increasing model size, suggesting a potential crossover point where MatMul-free models become more efficient. However, the video author expresses skepticism about this extrapolation.
* *(**45:00**)* *Hardware Efficiency:* The proposed architecture significantly reduces memory usage and latency. Implementing it on custom hardware like FPGAs, optimized for ternary operations, could lead to even greater efficiency gains.

*Author's Opinion (Yannic Kilcher):*

* *(**48:20**)* The research is exciting and promising for edge computing and energy-efficient AI.
* *(**48:20**)* He remains skeptical about:
* Whether MatMul-free models can truly surpass traditional Transformers in performance, especially for complex tasks.
* The validity of extrapolating scaling laws based on limited data points.
* The simplification trade-offs (like removing state-dependent hidden state updates) might limit the architecture's ultimate capabilities.

*Overall:*

The paper offers a compelling alternative to traditional MatMul-heavy LLMs, with potential for improved hardware efficiency. While challenges and open questions remain, it presents a promising direction for future research and development.

i used gemini 1.5 pro to summarize the transcript

[ttul]

The FPGA angle is what's interesting about this research. The paper proposes replacing all feed-forward operations in large language models with more computationally efficient operations, mostly by using ternary weights (i.e. -1, 0, and 1 are the only allowed values). Ternary weights are basically a simple logic gate with only three permitted operations:

a) Change the sign of the input (i.e. flip the sign bit and copy the rest)
b) Output zero
c) Copy the input to the output

If your goal is to make a neural network scream on hardware, having only three simple operations to choose from means you can use simple logic gates. The researchers tried this out in FPGAs and this is a promising area of research. From FPGA's it's not a big leap to ASICs, which nets the most power efficient computation theoretically possible. So if ternary gate networks can be made to scale, everyone should be excited.

Caveats:

1. The attention mechanism is replaced with a parallelizable form of recurrent neural network because applying ternary operations to attention does not train.

2. A linearized Gated Recurrent Unit (GRU) architecture allows for parallel computation; this is a neat trick.

3. The channel mixer (a feed-forward equivalent) uses dense layers with ternary accumulation operators.

Results show performance comparable to traditional Transformers, with better scaling properties at larger model sizes.
Yannick expresses some skepticism about the projected crossover point where this architecture would outperform traditional Transformers.

But I think the really interesting thing about this is the FPGA/ASIC aspect.

[KevinHorecka]

"stay hydrated" was a shockingly helpful reminder that I haven't drank any water today. Thanks!

[philiptren]

19:15 I think the model will learn to be more efficient with the extra accuracy. We can increase the length of the vector and the model will learn to use higher accuracy for the important values and a lower one for the ones where precision doesn’t matter as much, saving unnecessary precision. It’s like quantizing each and every weight of the model independently and exactly the right amount.

[Mordenor]

Thank you Mr Yannic for explaining MatMul free Language Modelling to your viewers!

[ronhightower]

Hopefully the research community gets these fundamental improvements figured out before Sam Altman spends a trillion dollars on data centers running Nvidia MatMul devices.

[unvergebeneid]

Anything that uses balanced ternary is already a superior method in my book :D

[adeeelh]

+100 to the rant at 25:32 about researchers relying on tricks instead of the main idea of the paper. It's my biggest pet peeve with deep learning papers.

[HansKonrad-lncg]

i have heard that after training you can basically throw away 90% of a network without changing the behaviour too much. that is because most of the weights are near zero which basically means a non-existent connection of the neurons. so if you omitt the calculation right away by taking it as exactly zero with the ternary values you save a lot of time that would have otherwise been spent with multiplying by zero for no reason.

[RPG_Guy-fxns]

if you have a layer of 64 neurons, the weights would be 16 bytes per neuron. You can use a look up table with 256 entries, instead of summing the binary digits. That way, most of the math is just turned into jumps into that table, finding 2 sums to subtract. its 16 boolean AND operations, to compare the previous layer output and this neuron's weights, 16 array lookups, adding them up as 2 totals, then subtracting the 2 bytes. That would be extremely fast compared to other neural networks, but I wonder if it can match the quality of other solutions.

[jmirodg]

It is only a first attempt I'm keen to see the following papers...

[pauldruhg]

Why stop with terniary? Go for powers of two and bit shifting. Speed and precision win-win.

[eruiluvatar]

I believe that you could still implement a fast "ternary multiplication" in a current GPU by using logic gates operating on multiple weights per register. Matmults are crazy fast on GPUs but by squeezing multiple weights together in a single register it might end up being faster.

[eoghanf]

I would really be interested in knowing more about the how the Straight-Through Estimator allows these things to train. That's the big mystery to me.

[adamrak]

Dot product in-memory architectures would be extremely fast and efficient for the inference. Less so for training.
So _if_ we change the architecture there are relatively simple ways we could add a few order of magnitudes to the inference performance.

[clray]

What I missed in the video and in the paper is an interpretation of replacing the weights with -1, 0, 1. And that would be: matrix multiplication xW is just calculation of n vector dot products - one dot product between x and each row of W. A dot product of two vectors is max when the vectors point in the same direction, min when the vectors point in the opposite direction, 0 if they are orthogonal. So it's basically deciding "let's glue all the KQV vectors, whose direction we compare with x, to the base axes (of the coordinate system), rather than allow them to point in any direction". I think that's what they call "privileged bases" in interpretability research. But given that you can only fit so many orthogonal vectors in n dimensions (and a lot more "almost" orthogonal vectors), it feels like it should impact the ability of the model to uniquely represent inputs.

[FryGuy]

As someone who has written CUDA code, this is relatively straightforward to do on GPUs. So your concern seems kind of unfounded that it will be basically the same performance as a full floating point multiplications

[WalterSamuels]

Look into VSA (hyperdimensional computing), and balanced ternary notation.