Code Llama Paper Explained

In this video we dive deep into the research paper behind Code Llama, the new family of large language models for code by Meta AI, which were created by specializing Llama 2 model for code.

👍 Please like & subscribe if you enjoy this content.

The Code Llama family contains three type of models, foundations models called Code Llama, python specialization models called Code Llama - Python and instruction-following models called Code Llama - Instruct.
We review the Code Llama training pipeline to create each of these models.

We then explain thoroughly the interesting self-instruct method which is used in order to fine-tune the Code Llama - Instruct model.

We also explain how Code Llama is able to support its useful code infilling capability in addition to code completion.

Throughout the video we also review several tables and charts from the paper to understand how the models perform comparing to other models.

----------------------------------------------------------------------------------
----------------------------------------------------------------------------------
Chapters:
0:00 Introducing Code Llama
0:49 Code Llama Training Pipeline
2:13 Long Context Fine-tuning
4:32 Self-Instruct
6:05 Code Infilling
7:08 Results

[jayaraopratik]

Please keep these coming!!! Thank you for all these papers. Has become a weekly ritual of mine.

[ympeng]

❤ Very clear explanation as always, great job!

[kalilinux]

I want to know how they varry epochs length ? Like how do they train some portion of data with 4 epochs and other part with just 0.1 epochs