Run Ab Test Models
Skill Verified ActiveDesign and execute A/B tests for ML models in production using traffic splitting, statistical significance testing, and canary/shadow deployment strategies. Measure performance differences and make data-driven decisions about model rollout. Use when validating a new model version before full rollout, comparing candidate models trained with different algorithms, measuring business metric impact of model changes, or when regulatory requirements mandate gradual rollout.
To enable data-driven decisions about ML model rollouts by designing and executing controlled A/B tests in production environments.
Features
- A/B test design with statistical significance
- Traffic splitting and user assignment
- Canary and shadow deployment strategies
- Performance metric collection and analysis
- Guardrail monitoring for safety thresholds
- Automated rollout decision support
Use Cases
- Validating new model versions before full rollout
- Comparing candidate models trained with different algorithms
- Measuring business metric impact of model changes
- Meeting regulatory requirements for gradual rollout
Non-Goals
- The actual deployment of ML models to production infrastructure
- The training of ML models
- Real-time model serving infrastructure management
Documentation
- info:Configuration & parameter referenceThe SKILL.md outlines the required and optional inputs for the A/B test experiment but does not provide a detailed reference for all parameters or their defaults within the main document. The referenced examples.md likely contain this detail.
Execution
- info:ValidationThe Python examples demonstrate basic data handling and analysis, but explicit schema validation libraries like Zod or Pydantic are not shown for all inputs/outputs within the main SKILL.md.
Installation
/plugin install agent-almanac@pjt222-agent-almanacQuality Score
VerifiedTrust Signals
Similar Extensions
Measure Experiment Design
100Designs an A/B test or experiment with clear hypothesis, variants, success metrics, sample size, and duration. Use when planning experiments to validate product changes or test hypotheses.
Arize Experiment
100Creates, runs, and analyzes Arize experiments for evaluating and comparing model performance. Covers experiment CRUD, exporting runs, comparing results, and evaluation workflows using the ax CLI. Use when the user mentions create experiment, run experiment, compare models, model performance, evaluate AI, experiment results, benchmark, A/B test models, or measure accuracy.
CE Optimize
100Run metric-driven iterative optimization loops -- define a measurable goal, run parallel experiments, measure each against hard gates or LLM-as-judge scores, keep improvements, and converge on the best solution. Use when optimizing clustering quality, search relevance, build performance, prompt quality, or any measurable outcome that benefits from systematic experimentation.
OraClaw Bandit
99A/B testing and feature optimization for AI agents. Pick the best option automatically using Multi-Armed Bandits and Contextual Bandits (LinUCB). No data warehouse needed — works from request
Experiment Designer
99Use when planning product experiments, writing testable hypotheses, estimating sample size, prioritizing tests, or interpreting A/B outcomes with practical statistical rigor.
Ab Test Setup
98When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "conversion experiment," "statistical significance," or "test this." For tracking implementation, see analytics-tracking.