LLM From Scratch · Author · Open Book

Free book on training LLMs end to end

A free, implementation-first book on training LLMs end to end. You build the tokenizer, the model, the pre-training loop, and the fine-tuning yourself, in PyTorch, and read the code behind each step.

Role: Author · Open Book
Stack: PyTorch · Python · Transformers
License: Open source
Status: In progress. Seven chapters written, more being added.

Direction

Goal.

Cover the full path from raw text to a deployed model, including quantization and serving, with every chapter backed by runnable code.

What it does

Features.

Character-level GPT

A minimal GPT trained from nothing, so attention and the training loop are fully visible.

BPE tokenizer

Byte-pair encoding written by hand, with the trade-offs you only see once you build it.

Embeddings and SimCSE

Contextual embeddings from a BERT-style encoder, then contrastive sentence embeddings.

Pre-training, SFT, and LoRA

Pre-train a small model, supervised fine-tune it, then adapt it cheaply with LoRA.