How to Build Custom Q&A Transformer Models in Python

In this video, we will learn how to take a pre-trained transformer model and train it for question-and-answering. We will be using the HuggingFace transformers library with the PyTorch implementation of models in Python.

Transformers are one of the biggest developments in Natural Language Processing (NLP) and learning how to use them properly is basically a data science superpower - they're genuinely amazing I promise!

I hope you enjoy the video :)

🤖 70% Discount on the NLP With Transformers in Python course:

Medium article:

(Free link):

Code:

Photo in thumbnail by Lorenzo Herrera on Unsplash

[malikrumi]

How is this different from doing a semantic search, where the model searches for embeddings that match the question, wherever they may be, and thus no need to do this answer training? (~30:00). Thanks.

[leomiao]

This video is fantastic, I learned a lot!! Thank you so much!!! 😁😁

[ren-san]

What should I do with a bert large model? I'm new to programming and not sure what to do because the model I want to use doesn't have a maximum length

[acsport]

Sir i did not under stand from the whole topic on this squad test set ....is whether we trained our data set by own or by importing the para graph and it train the data by its own 5:58 please 🥺❤️

[aanchalgupta]

Hi James, nice explanation but how we can get the prediction in correct format.

[ax]

why didn't use huggingface trainer?

[cosmogyral]

I get an error when trying to train. Some weights of the model checkpoint at distilbert-base-uncased were not used when initializing ['vocab_projector.weight', 'vocab_layer_norm.bias', 'vocab_projector.bias', 'vocab_transform.bias', 'vocab_transform.weight', 'vocab_layer_norm.weight'] etc. How could I fix this?

[yinnungandylau]

Dear James, is there any possible way in todays technology that I can OCR a book and input it to the machine for MCR and make a Q and A system to ask the question related to context of the book?

[jantuitman]

I think the plausibele answers are not supposed to be used. They are adversarial answers on questions that are actually impossible to answer based on the context

[Teng_XD]

How can I run it in tensorflow? I don't know how to define loss in tf

[ducle]

Hi James! Thank you for this video. But I wonder how should we do if:
1. the dataset contains very long contexts or answers.
2. exist a question that has more than one answer and those belonging to different contexts?
Thanks a lot!

[osamabuzdar]

@James Briggs hey can you tell me what was your test accuracy!!!!

[viktorciroski]

Hey, I'm trying to pass new content and questions ie ["To day is the 10th of Feb"] ["What is the date?"] I'm getting a tensor output of the encoded text. My question is does anyone know how to decode/detokenize the torch model's output?

[LMAOgrass]

Hi James, thanks for your video! Was wondering how we would train a transformer model if we do not have context of the question. For example, we only have a dataset of question, and answer? Thanks!

[TiMbuilding]

Maybe need to calculate distance between start_pred and start_true to each of element? And if distance is upper that accuracy is lower?

[lsbcip]

Hi James! Thank you for this video.
How can we extract the predicted text with our predicted start and end indices?

[harryayce]

maybe out of context but, is it a custom theme for jupyter lab? I haven't seen that before

[garynico]

Can i replace the model distilBERT with XLM-R? Or we need a different configuration?

[tlpunisher]

outputs['start_logits'] is showing error can you explain why?

[bhavyakrishnabalasubramani]

could we have pdf file instead of context??