https://jaykmody.com/blog/gpt-from-scratch/

In this post, we'll implement a GPT from scratch in just 60 lines of numpy. We'll then load the trained GPT-2 model weights released by OpenAI into our implementation and generate some text.

Note:

• This post assumes familiarity with Python, NumPy, and some basic experience with neural networks.
• This implementation is for educational purposes, so it's missing lots of features/improvements on purpose to keep it as simple as possible while remaining complete.
• All the code for this blog post can be found at github.com/jaymody/picoGPT.
• Hacker news thread
• Chinese translation
• Japanese translation

EDIT (Feb 9th, 2023): Added a "What's Next" section and updated the intro with some notes.

EDIT (Feb 28th, 2023): Added some additional sections to "What's Next".

Table of Contents

• Table of Contents
• What is a GPT?
  • Input / Output
    • Input
    • Output
  • Generating Text
    • Autoregressive
    • Sampling
  • Training
  • Prompting
• Setup
  • Encoder
  • Hyperparameters
  • Parameters
• Basic Layers
  • GELU
  • Softmax
  • Layer Normalization
  • Linear
• GPT Architecture
  • Embeddings
    • Token Embeddings
    • Positional Embeddings
    • Combined
  • Decoder Stack
  • Projection to Vocab
  • Decoder Block
  • Position-wise Feed Forward Network
  • Multi-Head Causal Self Attention
    • Attention
    • Self
    • Causal
    • Multi-Head
• Putting it All Together
• What Next?
  • GPU/TPU Support
  • Backpropagation
  • Batching
  • Inference Optimization
  • Training
  • Evaluation
  • Architecture Improvements
  • Stopping Generation
  • Fine-tuning
    • Classification Fine-tuning
    • Generative Fine-tuning
    • Instruction Fine-tuning
    • Parameter Efficient Fine-tuning