Distributed & Parallel check infos GPU on device: nvidia-smi keyword: ddp, parallel, gradient synchronization, fsdp, all_reduce Table content Pytorch distributed overview Distributed data prallel