Coding Llama 2 from scratch in PyTorch - Part 3

Показать описание

In this video series, you will learn how to train and fine-tune Llama 2 model from scrach.

The goal is to code LLaMA 2 from scratch in PyTorch to create models with sizes 100M, 250M and 500M params. In this third video, you'll learn about KV cache, RoPE, and Hugginface Trainer in detail.

📋 KV cache:

🪢 RoPE:

🤗 Trainer:

Sebastian Raschka:

💻 To follow along you can use this colab notebook:

🎥 Coding Llama 2 from scratch video series

Рекомендации по теме

Комментарии

So in this series, you don't use any pre-trained weights? You build and train the model from scratch on a custom dataset?

sharjeel_mazhar

Now hwow to scale it, like how to run it in mltple GPUs? or in Multiple nodes having multiple GPUs?

tharunbhaskar

First time watching your video. Keep going bro 💪, its your friend Afzal

Bebetter

@user-vd7im8gc2w

Why do you need position ids?

You use it to map the input ids to their respective position in the sequence.

Example:

input_ids = [100, 20, 4, 50]
position_ids =

print(position_ids)
>> [0, 1, 2, 3]

princecanuma

Coding Llama 2 from scratch in PyTorch - Part 3

Coding LLaMA 2 from scratch in PyTorch - KV Cache, Grouped Query Attention, Rotary PE, RMSNorm

FINALLY! Open-Source 'LLaMA Code' Coding Assistant (Tutorial)

How to use the Llama 2 LLM in Python

Coding Llama 2 from scratch in PyTorch - Part 3

Coding LLaMA-2 from scratch in PyTorch - Part 1

LLaMA explained: KV-Cache, Rotary Positional Embedding, RMS Norm, Grouped Query Attention, SwiGLU

End To End LLM Project Using LLAMA 2- Open Source LLM Model From Meta

Train Llama 2 from Scratch in PyTorch Locally

Fine Tune LLaMA 2 In FIVE MINUTES! - 'Perform 10x Better For My Use Case'

How to Create Custom Datasets To Train Llama-2

Coding Llama 3 from scratch in PyTorch - Part 1

I used LLaMA 2 70B to rebuild GPT Banker...and its AMAZING (LLM RAG)

Build Anything with Llama 3 Agents, Here’s How

Getting to Know Llama 2: Everything You Need to Start Building

How to build a Llama 2 chatbot

'okay, but I want Llama 3 for my specific use case' - Here's how

Coding Llama-2 from scratch in PyTorch - Part 2

Build and Run a Medical Chatbot using Llama 2 on CPU Machine: All Open Source

The EASIEST way to finetune LLAMA-v2 on local machine!

Step-by-step guide on how to setup and run Llama-2 model locally

Is CODE LLAMA Really Better Than GPT4 For Coding?!

Steps By Step Tutorial To Fine Tune LLAMA 2 With Custom Dataset Using LoRA And QLoRA Techniques

Run Your Own LLM Locally: LLaMa, Mistral & More

Fine-tuning Llama 2 on Your Own Dataset | Train an LLM for Your Use Case with QLoRA on a Single GPU