Were RNNs All We Needed? (Paper Explained)

This paper posits the interesting question: How much of the performance of Mamba, S4, and other state-space-like models is actually just attributable to some very core concepts - rather than their elaborate architectures. The authors construct minimal versions of GRUs and LSTMs and report competitive performance.

Abstract:
The scalability limitations of Transformers regarding sequence length have renewed interest in recurrent sequence models that are parallelizable during training. As a result, many novel recurrent architectures, such as S4, Mamba, and Aaren, have been proposed that achieve comparable performance. In this work, we revisit traditional recurrent neural networks (RNNs) from over a decade ago: LSTMs (1997) and GRUs (2014). While these models were slow due to requiring to backpropagate through time (BPTT), we show that by removing their hidden state dependencies from their input, forget, and update gates, LSTMs and GRUs no longer need to BPTT and can be efficiently trained in parallel. Building on this, we introduce minimal versions (minLSTMs and minGRUs) that (1) use significantly fewer parameters than their traditional counterparts and (2) are fully parallelizable during training (175x faster for a sequence of length 512). Lastly, we show that these stripped-down versions of decade-old RNNs match the empirical performance of recent sequence models.

Authors: Leo Feng, Frederick Tung, Mohamed Osama Ahmed, Yoshua Bengio, Hossein Hajimirsadegh

Links:

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n

[Bikameral]

It's great having you back !! Thank you and please don't leave us again

[fireinthehole]

Next paper: was NAND gates and registers all we needed?

[wolpumba]

*Were RNNs All We Needed? Revisiting the Power of Minimal Recurrent Networks*


* *0:00** Introduction:* The video explores a paper questioning the necessity of complex recurrent neural network (RNN) architectures like S4 and Mamba, suggesting that simpler RNNs might achieve comparable performance.
* *0:16** RNNs vs. Transformers:* RNNs handle sequences efficiently with constant memory requirements compared to Transformers' quadratic memory needs, but suffer from backpropagation through time (BPTT).
* *3:52** BPTT Limitations:* BPTT requires backpropagating gradients through all intermediate steps, limiting the length of sequences RNNs can effectively handle.
* *5:30** State Space Models:* Newer models like S4 and Mamba address BPTT by removing hidden state dependencies from input computations, allowing for parallel processing and training.
* *9:06** Minimal RNNs (minGRU, minLSTM):* The paper introduces minimal versions of GRUs and LSTMs that eliminate hidden state dependencies in gating mechanisms, further simplifying computation.
* *12:54** Parallel Scan:* These minimal RNNs can be trained efficiently using a parallel scan algorithm, similar to S4 and Mamba.
* *14:56** Trade-offs:* While simpler, minimal RNNs are less powerful than traditional RNNs in a single layer. However, this can be mitigated by using multiple layers.
* *19:55** Experimental Results:*
* *19:57** Selective Copying Task:* Minimal RNNs struggle with long-range dependencies in a single layer, but improve significantly with multiple layers.
* *21:02** Reinforcement Learning Benchmarks:* Minimal RNNs perform well, but the benchmarks are considered too simple to draw strong conclusions.
* *23:59** Language Modeling (Shakespeare):* Minimal RNNs perform comparably to Mamba on this small character-level dataset, where Transformers struggle due to the task's local nature.
* *26:45** Conclusion:* The paper's hypothesis that minimal RNNs can achieve comparable performance to complex state-space models is valid, but requires stronger experimental evidence. However, the potential for scalability and efficiency makes them promising candidates for future research.


I used gemini-1.5-pro-exp-0827 on rocketrecap dot com to summarize the transcript.
Cost (if I didn't use the free tier): $0.03
Input tokens: 21161
Output tokens: 467

[novantha]

Imagine how influential this paper could have been if it released in 2014, lol. It would have been revolutionary.

[Neomadra]

Excellent analysis of the benchmarks. Especially the analysis of character level tasks makes so much sense.

[HoriaCristescu]

Great explanation of the distinction between SSM and RNN at 5:30

[maccloud]

Use a dark theme, then you won't have to wear sunglasses.

[lizardy]

TLDR: It would have been more experimentally interesting to see results on an ensemble of minGRUs.
It is hard for me to say there is much takeaway here besides confirmation of the Mamba architecture's success. Perhaps they were a bit too excited with the release of the paper, that they decided to not focus on the stronger aspect of the paper, that being the minGRU and the concept of ensemble that Mamba also relies on.

[Mordenor]

Thank you Mr Yannic for discussing whether RNNs are all we needed.

[alirezaahmadi]

Hey man, I loved your video education, keep going with heavy booster, I love your work and just continue

[daniellu]

My first impression from reading the paper was that it seemed like a direct counterpoint to Transformers. From watching your video, I now see the paper as more building off Mamba and S4's successes so that RNNs might do even better at tasks that RNNs were already well-suited to perform. A strong validation benchmarking of minGRU/LSTM vs Transformer was not done in this paper and, even if a better benchmark was done, the Transformer may still have shown better performance. BUT the final takeaway could still be "to each their own: RNNs for RNN-suited tasks, Transformers for Transformer-suited tasks" (which sounds obvious and not-at-all exciting)

[PaganPegasus]

To me this paper highlights that RNNs actually aren't all we need and how powerful the transformer really is. A two layer transformer alone is capable of solving a bunch of tasks such as copying, sorting or other sorts of linear classification and reasoning thanks to the QK/OV circuits.

[GNARGNARHEAD]

was looking at doing something similar last week, but compressing the layers of a transformer into the weights for the RNN get around the training inefficiencies

[elpepemandioca]

In spite of not getting good results right now, I'd like more research to go this way, attempting to synthesize the plethora of models

[danielsautot]

Welcome back. Can you make a video of the architecture of the liquid foundation model?

[MohamedMagdy-uk]

24:00 correct me if I am wrong, what I see is Transformers is more generalized architecture that requires more training time, on the other side there is an inductive bias in Mamba, minLSTM, and minGRU that makes these architectures converges very quickly to that dataset

[the_primal_instinct]

Next paper: "Can multiplications be replaced with multiple additions?"

[easyBob]

I wonder if you could do a mixed system...have a set number of tokens for input and a hidden state based on past tokens. There's have to be a way to modify the hidden state with the new tokens and their importance. Then there'd have to be a way to let the hidden state influence the output. The model could then know what "we are currently talking about" and "what we've been talking about". [/thoughts]

[black-snow]

5th!
Finally able to leave a high-quality comment.

[xelaxander]

25:55 Constant gate decay might actually interesting for surrogate models of physical systems. Ignoring damage accumulation, a system response is independent of it‘s history.