filmov
tv
REALM: Retrieval-Augmented Language Model Pre-Training (Research Paper Walkthrough)
Показать описание
#languagemodel #realm #nlproc
⏩ Abstract: Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents. We demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both explicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous methods by a significant margin (4-16% absolute accuracy), while also providing qualitative benefits such as interpretability and modularity.
Please feel free to share out the content and subscribe to my channel :)
⏩ OUTLINE:
0:00 - Background and Overview of REALM
04:07 - REALM's generative process
04:55 - Knowledge Retriever
07:02 - Knowledge-Augmented Encoder
09:07 - Understanding Pre-training and Fine-tuning Pictorially
10:44 - Training Challenges
12:03 - Maximum Inner Product Search (MIPS) working
14:07 - What does the retriever learn?
15:38 - Salient Span Masking
16:34 - Null Document
17:01 - Prohibiting trivial retrievals
17:48 - Initialization and Inverse Cloze Task
⏩ Paper Title: REALM: Retrieval-Augmented Language Model Pre-Training
⏩ Author: Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang
⏩ Organisation: Google Research
*********************************************
If you want to support me financially which totally optional and voluntary ❤️
*********************************************
*********************************************
Tools I use for making videos :)
#techviz #datascienceguy #naturllanguageprocessing #opendomain_qa #researchpaper #arxiv
About Me:
I am Prakhar Mishra and this channel is my passion project. I am currently pursuing my MS (by research) in Data Science. I have an industry work-ex of 3 years in the field of Data Science and Machine Learning with a particular focus on Natural Language Processing (NLP).
⏩ Abstract: Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts. To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents. We demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both explicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous methods by a significant margin (4-16% absolute accuracy), while also providing qualitative benefits such as interpretability and modularity.
Please feel free to share out the content and subscribe to my channel :)
⏩ OUTLINE:
0:00 - Background and Overview of REALM
04:07 - REALM's generative process
04:55 - Knowledge Retriever
07:02 - Knowledge-Augmented Encoder
09:07 - Understanding Pre-training and Fine-tuning Pictorially
10:44 - Training Challenges
12:03 - Maximum Inner Product Search (MIPS) working
14:07 - What does the retriever learn?
15:38 - Salient Span Masking
16:34 - Null Document
17:01 - Prohibiting trivial retrievals
17:48 - Initialization and Inverse Cloze Task
⏩ Paper Title: REALM: Retrieval-Augmented Language Model Pre-Training
⏩ Author: Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei Chang
⏩ Organisation: Google Research
*********************************************
If you want to support me financially which totally optional and voluntary ❤️
*********************************************
*********************************************
Tools I use for making videos :)
#techviz #datascienceguy #naturllanguageprocessing #opendomain_qa #researchpaper #arxiv
About Me:
I am Prakhar Mishra and this channel is my passion project. I am currently pursuing my MS (by research) in Data Science. I have an industry work-ex of 3 years in the field of Data Science and Machine Learning with a particular focus on Natural Language Processing (NLP).
1:00:41
REALM: Retrieval-Augmented Language Model Pre-Training (Paper Explained)
0:07:11
REALM: Retrieval-Augmented Language Model Pre-Training
0:19:41
REALM: Retrieval-Augmented Language Model Pre-Training (Research Paper Walkthrough)
0:18:06
REALM: Retrieval-Augmented Language Model Pre-Training | NLP Journal Club
0:32:18
Retrieval Augmented Language Model Pre-training (REALM)
0:23:37
REALM: Retrieval-Augmented Language Model Pre-Training (Reading Papers)
![[Paper Review] REALM:](https://i.ytimg.com/vi/gtf770SDkX4/hqdefault.jpg)
0:26:46
[Paper Review] REALM: Retrieval-Augmented Language Model Pre-Training
0:54:22
REALM: Retrieval-Augmented Language Model Pre-training | Qpen Question Answering SOTA #OpenQA
1:19:27
Stanford CS25: V3 I Retrieval Augmented Language Models
0:00:53
When Do You Use Fine-Tuning Vs. Retrieval Augmented Generation (RAG)? (Guest: Harpreet Sahota)
0:26:41
Retrieval Augmented Language Modeling
0:15:45
REVEAL: Retrieval Augmented Visual Language Pre Training with Multi Source Multimodal Knowledge Me
0:25:44
#100 Dr. PATRICK LEWIS - Retrieval Augmented Generation
![[T@W intro] Gautier](https://i.ytimg.com/vi/StC_wwzuE4A/hqdefault.jpg)
0:02:47
[T@W intro] Gautier Izacard — Retrieval Augmented Language Models
0:09:55
Retrieval Augmented Generation for Knowledge Intensive NLP Tasks
0:07:39
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks | NLP journal club
0:00:55
🤔 Retrieval Augmented Generation: Search Engines' Future?
0:24:04
Retrieval-Augmented Generation (RAG)
1:22:32
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, with Patrick Lewis, Facebook AI
0:01:33
Retrieval Augmented Generation (RAG) in 90 Seconds
0:00:53
Google Scientist - Retrieval augmented LLM - augmented reality
0:03:24
Unlocking REALM: The Next Evolution in Language Models
0:10:04
Understanding Retrieval Augmented Generation (RAG)
0:15:01
Atlas: Few shot Learning with Retrieval Augmented Language Models
Комментарии