NeMo Evaluator SDK
技能 已验证 活跃Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.
To provide a scalable and reproducible platform for evaluating LLMs against a wide range of benchmarks, supporting enterprise needs for benchmarking on various computing infrastructures.
功能
- Evaluate LLMs across 100+ benchmarks
- Supports 18+ evaluation harnesses (MMLU, HumanEval, VLM, safety)
- Multi-backend execution (Docker, Slurm, Cloud)
- Reproducible containerized evaluation
- Enterprise-grade platform with result export (MLflow, W&B)
使用场景
- Running scalable LLM evaluations on local Docker instances
- Benchmarking LLMs on Slurm HPC clusters
- Comparing multiple models on standard academic and industry benchmarks
- Ensuring reproducible LLM evaluations through containerization
非目标
- Training or fine-tuning LLMs
- Providing raw model APIs
- General-purpose code generation or analysis beyond benchmark tasks
工作流
- Configure evaluation parameters (execution backend, model endpoint, tasks)
- Select benchmarks and optionally override parameters per task
- Launch evaluation via CLI or Python API
- Monitor job status and retrieve results
- Export results for comparison and analysis
实践
- Benchmarking
- LLM Evaluation
- Reproducible Computing
- Distributed Systems
先决条件
- Docker installed and running (for local execution)
- SSH access to Slurm cluster (for Slurm execution)
- NGC API Key (for container pulls and NVIDIA services)
- HF_TOKEN (for some benchmarks)
安装
npx skills add davila7/claude-code-templates通过 npx 运行 Vercel skills CLI(skills.sh)— 需要本地安装 Node.js,以及至少一个兼容 skills 的智能体(Claude Code、Cursor、Codex 等)。前提是仓库遵循 agentskills.io 格式。
质量评分
已验证类似扩展
Nemo Evaluator Sdk
98Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.
Azure Container Registry SDK for Python
100Azure Container Registry SDK for Python. Use for managing container images, artifacts, and repositories. Triggers: "azure-containerregistry", "ContainerRegistryClient", "container images", "docker registry", "ACR".
Context Compression
100This skill should be used when the user asks to "compress context", "summarize conversation history", "implement compaction", "reduce token usage", or mentions context compression, structured summarization, tokens-per-task optimization, or long-running agent sessions exceeding context limits.
Evaluating Llms Harness
99Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
Lm Evaluation Harness
98Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
BigCode Evaluation Harness
98Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.