life circle
This is a repo that collects awsome projects focused on building GenAI from scratch.
Currently there are three subfields: LLM, MLLM, and Generation(image, video...).
TinyZero: reproduce deepseek-R1 from scratch
nanoGPT: build gpt2 from scratch
baby-llama2-chinese: build llama2-chinese from scratch
llama3-from-scratch: llama3 implemented from scratch
nanochat: build a chat model from scratch, including the entire pipeline of tokenizer, pretraining, and post-training.
nanoVLM: Train a VLM from scratch
nano-vllm: implement lightweight vLLM from scratch
mini-sglang: implement lightweight sglang from scratch
- What are Diffusion Models?: This is an excellent introductory tutorial for learning diffusion models.
- Introduction to Flow Matching and Diffusion Models: A MIT's tutorial on flow matching
- minitorch: Learn the core concepts behind deep learning systems by implementing a mini version of PyTorch.
- Hugginface cookbook: A awesome cookbook covering all you need about LLM
- DeepSeek-R1: A work that sparked the reasonning model boom
- DeepSeek-v3: A highly influential MoE model.
- Llama 3: A simple yet worthwhile open-source model.
- Scaling Law OpenAI version: Basic principle of LLM(OpenAI version)
- Scaling Law Google version: Basic principle of LLM(Google version)
- o1: Test time scaling: (It seems that there is no universally recognized paper.)
- gpt-o4: Multimodal Reasoning with Tools
- score based model: An attempt to unify diffusion models: A unified framework(SDE) of generative models
- LLaDA: A representative work about diffusion text model
- World Model from Feifei Li: World Model understanding from Feifei Li
We welcome contributions if you are interesting at building nano GenAI from scratch and want to maintain the awesome list together. Please read the CONTRIBUTING for more details.
- CONTRIBUTING.md
- Subdivided into multiple fields