GPU stuff

Practical recommendation (what I would do in your shoes)

• Spend about 1 to 2 weeks on Triton: you will already cover attention + optimizer + baseline GEMM with fast iteration.
• Then move to CuTe DSL specifically for GEMM + epilogues once you want CUTLASS-grade layout control (NVIDIA docs).
• If the kernel is "GEMM is the main event" -> CuTe DSL / CUTLASS.
• If the kernel is "softmax + streaming + attention structure" -> Triton.

Starting Sunday

• Watch CUDA 3 seconds vids from YouTube.
• Twitter guy Elliot.
• Follow GPU notes and inference.
• Check phys NN Twitter.

GPU mode competition

• GPU mode v2
• NVIDIA guidebook
• PPMP
• OpenAI Triton docs
• CUDA challenges

Core docs and languages

• Triton language
• CUDA programming guide (part 1)

Notes and links

• vikhyat GPU notes (gist)
• Inference optimization (Lilian Weng)
• Mixtral inference model.py
• KernelBench
• FlashMLA
• Modular puzzles
• fast.cu matmul example
• Precision notes (doc)
• FP16 (Turing Post)
• Add from scratch_project folder on project.
• Compile CUDA in the browser (HipScript)
• HipScript architecture blog
• LeetGPU
• Intro to CUDA on HIPs
• Intro from NVIDIA guy.
• Site by Googol lady.
• Ref 100d CUDA (GitHub)
• Modal GPU architecture
• Tristan Hume
• Bohem
• How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: a Worklog.
• GPU programming from scratch
• CUDA SGEMM optim (wangzyon GitHub repository).
• High performance attention mechanism
• AMD A2A
• Fast LLM inference (YALM)
• Programming Parallel Computers
• JAX scaling book: GPUs
• Inside tdkittens
• Cornell GPU architecture
• Thinderkittens
• Inside NVIDIA GPU
• CUDA handbook
• Modal GPU perf
• JAX high perf kernels (Blackwell matmul)
• GPU mindset
• Inside NVIDIA GPU (2)
• Kernel graphics (Triton)
• GPU related
• Start reading here: Fundamentals of GPU engineering
• Unsloth Docker images
• Efficient LLM training
• LLMQ C++ CUDA FSDP multigpu
• CUDA thinder.. yt
• Flash attention reverse engineering
• Flash attention from scratch
• Flash attention in Tilelang
• Linux Docker Kubernetes challenges
• H100 CUDA matmul kernel from scratch
• Stuart Sul thread
• Modular Magnum series (Blackwell matmul)
• ML engineering (stas00)
• Understand CPU (YouTube)
• Flash Infer
• Linux kernel IO deep dive

Some perf related must-reads

• How to Optimize a CUDA Matmul Kernel for cuBLAS-like Performance: siboehm.com/articles/22/CU...
• Outperforming cuBLAS on H100: a Worklog: cudaforfun.substack.com/p/outperformin...
• Defeating Nondeterminism in LLM Inference: thinkingmachines.ai/blog/defeating...
• Making Deep Learning go Brrrr From First Principles: horace.io/brrr_intro.html
• Transformer Inference Arithmetic: kipp.ly/transformer-in...
• Domain specific architectures for AI inference: fleetwood.dev/posts/domain-s..
• A postmortem of three recent issues: anthropic.com/engineering/a-...
• How To Scale Your Model: jax-ml.github.io/scaling-book/
• The Ultra-Scale Playbook: huggingface.co/spaces/nanotro...
• The Case for Co-Designing Model Architectures with Hardware: arxiv.org/abs/2401.14489

Some recent reads (this month)

• Inside NVIDIA GPUs: Anatomy of high performance matmul kernels: aleksagordic.com/blog/matmul
• Triton Flash Attention Kernel Walkthrough: The Forward Pass: nathanchen.me/public/Triton-...
• This guy substack: michalpitr.substack.com
• Deep Dive into Triton Internals (3 parts): kapilsharma.dev/posts/deep-div...
• HunyuanWorld-Mirror: Technical Report: 3d-models.hunyuan.tencent.com/world/worldMir...
• Understanding the CUDA Compiler and PTX with a Top-K Kernel: blog.alpindale.net/posts/top_k_cu...
• Geometry Meets Vision: Revisiting Pretrained Semantics in Distilled Fields: arxiv.org/abs/2510.03104

From scratch ideas

• Implement a smol-diloco.
• Mini-FSDP.
• Smol-vLLM (inference engine).
• RL envs (for puzzles like klotski or proving theorems).
• VLA models for robots.
• Dissecting NCCL, CUTLASS, TensorRT, SGLang, vLLM.
• Speed hacks: attention sinks, speculative decoding, quantization, KV-cache tuning, paged attention.
• Experimenting with MoEs: upcycling, expert parallelism, deep EP, router optmzn, DeepSeek-MoE style training.
• Mech interp: CoT probing in reasoning models, exploring how in-context learning emerges, induction heads, comparing distilled and undistilled reasoning models.
• This is a tiny list of things you could build. Do not get stuck going too deep into theory (especially around ML systems, inference, post-training).