Sliding Window Attention (Longformer) Explained

preview_player
Показать описание
In this video we talk about the sliding window attention, the diluted sliding window attention and the global+sliding window attention, as introduced in the Longformer paper. We take a look at the main disadvantage of the classical attention mechanism introduced in the Transformer paper (i.e. the quadratic time complexity) and how the sliding window attention proposes to solves this issue.

*References*
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

*Related Videos*
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

*Contents*
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
00:00 - Intro
00:26 - Original attention mechanism
00:50 - Sliding window attention
01:56 - Dilated sliding window attention
02:40 - Global + Sliding window attention
03:31 - Outro

*Follow Me*
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬

*Channel Support*
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
The best way to support the channel is to share the content. ;)

If you'd like to also support the channel financially, donating the price of a coffee is always warmly welcomed! (completely optional and voluntary)
► Bitcoin (BTC): 3C6Pkzyb5CjAUYrJxmpCaaNPVRgRVxxyTq
► Ethereum (ETH): 0x9Ac4eB94386C3e02b96599C05B7a8C71773c9281
► Cardano (ADA): addr1v95rfxlslfzkvd8sr3exkh7st4qmgj4ywf5zcaxgqgdyunsj5juw5
► Tether (USDT): 0xeC261d9b2EE4B6997a6a424067af165BAA4afE1a

#slidingwindowattention #longformer #attentionmechanism
  1. DataMListic
  2. deep learning
  3. machine learning
  4. artificial intelligence
  5. ml
  6. dl
Рекомендации по теме
Sliding Window Attention 0:03:51

Sliding Window Attention (Longformer) Explained

Longformer Model for 0:06:56

Longformer Model for dealing with Longer Documents | its Sliding Window Function | Data Science

Longformer: The Long-Document 0:26:36

Longformer: The Long-Document Transformer

LLM Jargons Explained: 0:15:22

LLM Jargons Explained: Part 3 - Sliding Window Attention

Mistral Architecture Explained 0:39:10

Mistral Architecture Explained From Scratch with Sliding Window Attention, KV Caching Explanation

Longformer | Lecture 0:08:05

Longformer | Lecture 57 (Part 1) | Applied Deep Learning (Supplementary)

Longformer for Automatic 0:07:30

Longformer for Automatic Summary Evaluation

Longformer: The Long-Document 0:29:07

Longformer: The Long-Document Transformer - Presented by Ahmed Baraka

Longformer Model in 0:00:54

Longformer Model in NLP - Basic Theory | Deep Learning | Machine Learning

Fastformer: Additive Attention 0:15:22

Fastformer: Additive Attention Can Be All You Need | Paper Explained

Neighborhood Attention Transformer 0:08:00

Neighborhood Attention Transformer (CVPR 2023)

Longformer Model in 0:00:50

Longformer Model in NLP - Basic Theory | Deep Learning | Machine Learning

Efficient Streaming Language 0:32:27

Efficient Streaming Language Models with Attention Sinks (Paper Explained)

[Yonsei NLP Study] 0:38:38

[Yonsei NLP Study] Sparse Attention - Reformer & Longformer

How much memory 0:09:19

How much memory does Longformer use?

Clinical Longformer - 0:05:07

Clinical Longformer - Deep Learning CMU

Lecture 4: Swin 0:11:55

Lecture 4: Swin Transformer from Scratch in PyTorch - Window Attention & Cyclic Shift

Big Bird: Transformers 0:34:30

Big Bird: Transformers for Longer Sequences (Paper Explained)

Deep dive - 0:40:54

Deep dive - Better Attention layers for Transformer models

Longformer end to 1:56:23

Longformer end to end with Kaggle NLP competition | Data Science | Machine Learning

Introducing Mistral 7B: 0:01:53

Introducing Mistral 7B: A Powerful Language Model with Grouped-query and Sliding Window Attention

Longformer presented by 0:05:36

Longformer presented by Fouzi

NEW: Unlimited Token 0:18:48

NEW: Unlimited Token Length for LLMs by Microsoft (LongNet explained)

ALiBi - Train 0:31:22

ALiBi - Train Short, Test Long: Attention with linear biases enables input length extrapolation

Комментарии
Автор

this was super helpful, thanks so much!

mutantrabbit
welcome to shbcf.ru