Distributed Training Notes

DDP, FSDP, TP, PP: A Field Guide to Modern Parallelism Strategies

Every time I open Twitter/X/LinkedIn these days I see the same alphabet-soup: DDP, FSDP, TP, PP, DiLoCo.
It’s easy to think they’re just buzzwords slapped onto marketing…