Who is this deep dive for?

A few days ago, Andrej Karpathy released a video titled “Deep dive into LLMs like ChatGPT.” It’s a goldmine of information, but it’s also 3 hours and 31 minutes long.
I watched the whole thing and took a bunch of notes, so I figured why not put together a TL;DR version for anyone who wants the essential takeaways without the large time commitment.

If any of this sounds like you, this post (and the original video) is worth checking out:

• You want to understand how LLMs actually work not just at the surface level.
• You want to understand confusing fine-tuning terms like chat_template and ChatML (especially if you’re using Axolotl).
• You want to get better at prompt engineering by understanding why some prompts work better than others.
• You’re trying to reduce hallucinations and want to know how to keep LLMs from making things up.
• You want to understand why DeepSeek-R1 is such a big deal right now.

I won’t be covering everything in the video, so if you have time, definitely watch the whole thing. But if you don’t, this post will give you the key takeaways.

Note: If you are looking for the excalidraw diagram that Andrej made for the video, you can download it here. He shared it through Google Drive and it invalidates the link after a certain time. That’s why I have decided to host it on my CDN as well.

Pretraining Data

Internet

LLMs start by crawling the internet to build a massive text dataset. The problem? Raw data is noisy and full of duplicate content, low-quality text, and irrelevant information. Before training, it needs heavy filtering.

• If you’re building an English only model, you’ll need a heuristic to filter out non-English text (e.g., only keeping text with a high probability of being English).
• One example dataset is FineWeb, which contains over 1.2 billion web pages.

Once cleaned, the data still needs to be compressed into something usable. Instead of feeding raw text into the model, it gets converted into tokens: a structured, numerical representation.

Tokenization

Tokenization is how models break text into smaller pieces (tokens) before processing it. Instead of storing raw words, the model converts them into IDs that represent repeating patterns.

• A popular technique for this is Byte Pair Encoding (BPE).
• There’s an optimal number of symbols (tokens) for compression. For example, GPT-4 uses 100,277 tokens. It is totally dependent on the discretion of the model creator.
• You can visualize how this works using tools like Tiktokenizer.

Neural Network I/O

Once the data is tokenized, it’s fed into the neural network. Here’s how that process works:

1. The model takes in a context window, a set number of tokens (e.g., 8,000 for some models, up to 128k for GPT-4).
2. It predicts the next token based on the patterns it has learned.
3. The weights in the model are adjusted using backpropagation to reduce errors.
4. Over time, the model learns to make better predictions.

A longer context window means the model can “remember” more from the input, but it also increases computational cost.

Neural Network Internals

Inside the model, billions of parameters interact with the input tokens to generate a probability distribution for the next token.

• This process is defined by complex mathematical equations optimized for efficiency.
• Model architectures are designed to balance speed, accuracy, and parallelization.
• You can see a production-grade LLM architecture example here.

Inference

LLMs don’t generate deterministic outputs, they are stochastic. This means the output varies slightly every time you run the model.

• The model doesn’t just repeat what it was trained on, it generates responses based on probabilities.
• In some cases, the response will match something in the training data exactly, but most of the time, it will generate something new that follows similar patterns.

This randomness is why LLMs can be creative, but also why they sometimes hallucinate incorrect information.

GPT-2

GPT-2, released by OpenAI in 2019, was an early example of a transformer-based LLM.
Here’s what it looked like:

• 1.6 billion parameters
• 1024-token context length
• Trained on ~100 billion tokens
• The original GPT-2 training cost was $40,000.

Since then, efficiency has improved dramatically. Andrej Karpathy managed to reproduce GPT-2 using llm.c for just $672. With optimized pipelines, training costs could drop even further to around $100.

Why is it so much cheaper now?

• Better pre-training data extraction techniques → Cleaner datasets mean models learn faster.
• Stronger hardware and optimized software → Less computation needed for the same results.

Open Source Base Models

Some companies train massive LLMs and release the base models for free. A base model is essentially the raw, unrefined LLM, it still needs tuning to be useful.

• Base models are trained on raw internet text, meaning they generate completions but don’t understand human intent.
• OpenAI open-sourced GPT-2.
• Meta open-sourced Llama 3.1 (405B parameters) which is a rather new LLM base model as compared to GPT-2.

In order to fully open-source a base model, two things are required:

• The code (e.g., a few hundred lines in Python that defines the steps the model will take to generate the prediction).
• The parameters (e.g., billions of tuned weights that define the model).

How Base Models Work

• They generate token-level internet-style text.
• Every run produces a slightly different output (stochastic behavior).
• They can regurgitate parts of their training data.
• The parameters are like a lossy zip file of internet knowledge.
• You can already use them for applications like:
  • Translation → Using in-context examples.
  • Basic assistants → Prompting them in a structured way.

Want to experiment with one? Try the Llama 3 (405B base model) here.

At its core, a base model is just an expensive autocomplete. It still needs fine-tuning.

---

Pre-Training to Post-Training

So far, we’ve looked at base models, which are just pre-trained text generators. But to make an actual assistant, you need post-training.

• Base models hallucinate a lot → They generate text, but it’s not always useful.
• Post-training fixes this by fine-tuning the model to respond better.
• The good news? Post-training is way cheaper than pre-training (e.g., months vs. hours).

---

Supervised Fine-Tuning (SFT)

Data Conversations

Once the base model is trained on internet data, the next step is post-training. This is where we replace the internet dataset with human/assistant conversations to make the model more conversational and useful.

• Pre-training takes months, but post-training is much faster. It can tak