[Paper Reading] Titans: Learning to Memorize at Test Time

Speaker:
Krishnan Ramaswamy
Gen AI Product Development & Principal Architect @ Cisco for, AI, ML, and Gen AI-enabled computer networking products & solutions.

Key Highlights of the session :
- Neural Memory Mechanism: Titan introduces a novel memory system inspired by human cognition, dividing memory into short-term (STM) for immediate context and long-term (LTM) for summarized historical information, enabling effective reasoning and scalability.

- Scalability: By abstracting and summarizing information, Titan scales context windows beyond 2 million tokens, outperforming traditional attention mechanisms constrained by quadratic computational costs.

-Memory Updating: Titan selectively updates memory by identifying "surprising" or new information, ensuring that critical updates are preserved without overwhelming memory resources.

-Efficiency: The architecture reduces computational demands by managing the size of attention matrices, maintaining efficiency even with large context windows.

-Enhanced Performance: Experimental results show 20-30% improvements in accuracy and reasoning tasks compared to traditional models, demonstrating Titan's ability to retain context over long sequences effectively.

- Applications: Titan is highly relevant for tasks requiring agentic reasoning, knowledge retention, and adaptive learning, making it suitable for real-world applications like large-scale automation and summarization.

- Foundation and Implementation: Titan builds on established transformer principles, integrating advancements from RNNs and LSTMs. Implemented in PyTorch and JAX, it is designed for practical experimentation, with potential code release anticipated.

Join over 2000 professionals
who have developed expertise in AI/ML
Become part of SupportVectors to learn about in-depth technical abilities and further your career.