Pytorch Fsdp2
Skill Verified ActiveAdds PyTorch FSDP2 (fully_shard) to training scripts with correct init, sharding, mixed precision/offload config, and distributed checkpointing. Use when models exceed single-GPU memory or when you need DTensor-based sharding with DeviceMesh.
Enables users to correctly implement PyTorch FSDP2 in their training scripts to manage large models that exceed single-GPU memory or require advanced sharding strategies.
Features
- Adds PyTorch FSDP2 integration to training scripts.
- Correct initialization, sharding, and mixed precision configuration.
- Guidance on distributed checkpointing (DCP).
- Handles models exceeding single-GPU memory.
- Supports DTensor-based sharding with DeviceMesh.
Use Cases
- When models do not fit on a single GPU.
- For users needing DTensor-based per-parameter sharding.
- When composing DP with Tensor Parallelism using DeviceMesh.
- Integrating FSDP2 into existing PyTorch training loops.
Non-Goals
- Replacing standard DistributedDataParallel (DDP) when model fits on one GPU.
- Using older FSDP1 versions.
- Providing backwards-compatible checkpoints across PyTorch versions without careful management.
- Supporting PyTorch versions without the FSDP2 stack.
Installation
First, add the marketplace
/plugin marketplace add Orchestra-Research/AI-Research-SKILLs/plugin install AI-Research-SKILLs@ai-research-skillsQuality Score
VerifiedTrust Signals
Similar Extensions
Pytorch Lightning
99High-level PyTorch framework with Trainer class, automatic distributed training (DDP/FSDP/DeepSpeed), callbacks system, and minimal boilerplate. Scales from laptop to supercomputer with same code. Use when you want clean training loops with built-in best practices.
Huggingface Accelerate
99Simplest distributed training API. 4 lines to add distributed support to any PyTorch script. Unified API for DeepSpeed/FSDP/Megatron/DDP. Automatic device placement, mixed precision (FP16/BF16/FP8). Interactive config, single launch command. HuggingFace ecosystem standard.
TorchTitan Distributed LLM Pretraining
99Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.
HuggingFace Accelerate
97Simplest distributed training API. 4 lines to add distributed support to any PyTorch script. Unified API for DeepSpeed/FSDP/Megatron/DDP. Automatic device placement, mixed precision (FP16/BF16/FP8). Interactive config, single launch command. HuggingFace ecosystem standard.
PyTorch Lightning
100Deep learning framework (PyTorch Lightning). Organize PyTorch code into LightningModules, configure Trainers for multi-GPU/TPU, implement data pipelines, callbacks, logging (W&B, TensorBoard), distributed training (DDP, FSDP, DeepSpeed), for scalable neural network training.
Nnsight Remote Interpretability
99Provides guidance for interpreting and manipulating neural network internals using nnsight with optional NDIF remote execution. Use when needing to run interpretability experiments on massive models (70B+) without local GPU resources, or when working with any PyTorch architecture.