Torchforge
Skill ActiveProvides guidance for PyTorch-native agentic RL using torchforge, Meta's library separating infra from algorithms. Use when you want clean RL abstractions, easy algorithm experimentation, or scalable training with Monarch and TorchTitan.
To enable researchers and engineers to conduct PyTorch-native agentic RL experiments with clean abstractions, easy algorithm implementation, and scalable distributed training capabilities.
Features
- PyTorch-native RL abstractions
- Separation of RL algorithms from infrastructure
- Scalable training with Monarch and TorchTitan
- Easy algorithm experimentation (GRPO, SFT examples)
- High-throughput inference with vLLM
Use Cases
- When needing clean RL abstractions independent of infrastructure
- When experimenting with new RL algorithms in PyTorch
- For scalable training of RL models using distributed systems like Monarch
- When integrating with Meta's TorchTitan for model parallelism
Non-Goals
- Production-ready stability (considered experimental)
- Megatron-native training (use alternative skills)
- Replacing fully mature RL frameworks for production deployment
Workflow
- Define Configuration (YAML)
- Define Reward Function (Python)
- Launch Training (Python script)
- Monitor Progress (W&B, metrics)
Practices
- RL Algorithm Implementation
- Distributed Training
- Model Experimentation
- Scalable ML Systems
Prerequisites
- Python 3.12+
- 3+ GPUs recommended for GRPO training
- PyTorch >= 2.9.0 (nightly)
- Monarch, TorchTitan, vLLM
Practical Utility
- warning:Production readinessThe SKILL.md explicitly states that torchforge is experimental and APIs may change, suggesting it's not fully production-ready for all use cases, although it covers the stated research purpose.
Installation
First, add the marketplace
/plugin marketplace add Orchestra-Research/AI-Research-SKILLs/plugin install AI-Research-SKILLs@ai-research-skillsQuality Score
Trust Signals
Similar Extensions
Pytorch Lightning
99High-level PyTorch framework with Trainer class, automatic distributed training (DDP/FSDP/DeepSpeed), callbacks system, and minimal boilerplate. Scales from laptop to supercomputer with same code. Use when you want clean training loops with built-in best practices.
Huggingface Accelerate
99Simplest distributed training API. 4 lines to add distributed support to any PyTorch script. Unified API for DeepSpeed/FSDP/Megatron/DDP. Automatic device placement, mixed precision (FP16/BF16/FP8). Interactive config, single launch command. HuggingFace ecosystem standard.
Ray Train
99Distributed training orchestration across clusters. Scales PyTorch/TensorFlow/HuggingFace from laptop to 1000s of nodes. Built-in hyperparameter tuning with Ray Tune, fault tolerance, elastic scaling. Use when training massive models across multiple machines or running distributed hyperparameter sweeps.
Verl Rl Training
99Provides guidance for training LLMs with reinforcement learning using verl (Volcano Engine RL). Use when implementing RLHF, GRPO, PPO, or other RL algorithms for LLM post-training at scale with flexible infrastructure backends.
Openrlhf Training
99High-performance RLHF framework with Ray+vLLM acceleration. Use for PPO, GRPO, RLOO, DPO training of large models (7B-70B+). Built on Ray, vLLM, ZeRO-3. 2× faster than DeepSpeedChat with distributed architecture and GPU resource sharing.
TorchTitan Distributed LLM Pretraining
99Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP). Use when pretraining Llama 3.1, DeepSeek V3, or custom models at scale from 8 to 512+ GPUs with Float8, torch.compile, and distributed checkpointing.