Repository
Deep-Learning
From-scratch deep learning: micrograd autograd → MLPs → BatchNorm → manual backprop → WaveNet → GPT, in notebooks.
What this is
A first-principles deep learning workbook. Every notebook here was typed out and run by hand while working through Andrej Karpathy's "Neural Networks: Zero to Hero" series. The goal was not to use deep learning, but to rebuild it: derive the gradients, write the training loops, watch the activations, and only then trust the framework.
Credit for the curriculum and the original material goes to Andrej Karpathy. The code in this repo is my own walk through it, plus side experiments in practise/.
The path
flowchart LR
A[micrograd<br/>scalar autograd] --> B[Bigram LM<br/>counting + NN]
B --> C[MLP LM<br/>embeddings]
C --> D[BatchNorm<br/>activations & grads]
D --> E[Backprop by hand<br/>no autograd]
E --> F[WaveNet<br/>hierarchical model]
F --> G[GPT<br/>self-attention]
Curriculum
| # | Topic | What gets built | Notebook |
|---|---|---|---|
| 1 | Backpropagation | micrograd: a tiny scalar autograd engine with graph visualisation (graphviz), then a small neural net trained on top of it |
micrograd |
| 2 | Language modelling | makemore part 1: character-level bigram model on names.txt, first by counting, then as a one-layer neural net |
makemore part 1 |
| 3 | MLP | makemore part 2: Bengio-style MLP with learned character embeddings, train/dev/test splits, learning-rate search | makemore part 2 |
| 4 | Activations, gradients, BatchNorm | makemore part 3: initialisation, saturated tanh, Kaiming init, Batch Normalization | makemore part 3 |
| 5 | Backprop ninja | makemore part 4: manual backward pass through cross-entropy, BatchNorm, tanh and the embedding layer, checked against PyTorch autograd | makemore part 4 |
| 6 | WaveNet | makemore part 5: hierarchical, WaveNet-style architecture built from custom layer modules | makemore part 5 |
| 7 | GPT | Character-level Transformer on Tiny Shakespeare: self-attention, multi-head attention, feed-forward blocks, residuals, LayerNorm, dropout | gpt_dev.ipynb · walkthrough · bigram.py |
The GPT script
bigram.py is the standalone training script that came out of the GPT lecture. Despite the filename, it trains the full decoder-only Transformer:
| Hyperparameter | Value |
|---|---|
| Layers / heads | 6 / 6 |
| Embedding dim | 384 |
| Context length | 256 characters |
| Batch size | 64 |
| Optimizer | AdamW, lr 3e-3 |
| Iterations | 5000 |
| Dropout | 0.2 |
It trains on input.txt (Tiny Shakespeare) with a 90/10 train/val split, uses CUDA if available, and writes a generated sample to output.txt.
Beyond the series
| Folder | What's inside |
|---|---|
fine tuning/ |
Early-stage: a notebook on the maths of 8-bit quantisation (scale / zero-point, int8 range) and a starter notebook for fine-tuning with Unsloth |
practise/ |
Practice re-implementations: several micrograd Value engines, makemore, WaveNet, a nanoGPT start, scikit-learn / PyTorch experiments on make_blobs and the diabetes dataset, and a tiny tensor-like MyList with view() |
reads/ |
The papers behind the lectures: Bengio et al. 2003, Batch Normalization, He et al. (Kaiming init), WaveNet, Attention Is All You Need |
Project structure
Deep-Learning/
├── micrograd/
│ ├── files/names.txt # makemore dataset
│ ├── intro to neural networks ... micrograd.ipynb
│ ├── The spelled-out intro to language modeling building makemore.ipynb
│ ├── Building makemore Part 2 MLP.ipynb
│ ├── Building makemore Part 3 Activations Gradients BatchNorm.ipynb
│ ├── makemore Part 4 becoming a backprop ninja.ipynb
│ ├── Building makemore part 5 Building a WaveNet.ipynb
│ └── Let's build GPT from scratch in code spelled out/
│ ├── gpt_dev.ipynb
│ ├── bigram.py # full GPT training script
│ └── input.txt # Tiny Shakespeare
├── fine tuning/
├── practise/
├── reads/ # reference papers (PDF)
└── requirements.txt
Quickstart
git clone https://github.com/nabeelalikhan0/Deep-Learning.git
cd Deep-Learning
python -m venv .venv
# Windows: .venv\Scripts\activate macOS/Linux: source .venv/bin/activate
pip install -r requirements.txt
pip install torch jupyter # PyTorch is used by the notebooks but not pinned in requirements.txt
Open the notebooks (run them from inside micrograd/ so files/names.txt resolves):
cd micrograd
jupyter notebook
Train the GPT:
cd "micrograd/Let's build GPT from scratch in code spelled out"
python bigram.py
A GPU is strongly recommended for bigram.py; on CPU it runs, but slowly.
The micrograd notebook renders computation graphs with graphviz, which also needs the Graphviz system binaries on your PATH.
Acknowledgements
- Andrej Karpathy - Neural Networks: Zero to Hero, micrograd, makemore, nanoGPT
- The authors of the papers in
reads/