Lm Evaluation Harness
Skill Verified ActiveEvaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
To provide a standardized, robust, and user-friendly tool for evaluating and comparing LLM performance across a broad range of academic benchmarks.
Features
- Evaluates LLMs across 60+ academic benchmarks
- Supports HuggingFace, vLLM, and API-based models
- Offers detailed documentation for custom tasks and distributed evaluation
- Provides examples for common workflows like model comparison and training progress tracking
- Industry standard used by major AI labs
Use Cases
- Benchmarking LLM quality for research papers
- Comparing performance between different LLMs
- Tracking LLM training progress over time
- Validating model outputs against standardized metrics
Non-Goals
- Evaluating LLMs on proprietary, non-academic tasks
- Providing a platform for training or fine-tuning LLMs
- Offering a judgment on model 'intelligence' beyond benchmark scores
Workflow
- Configure model and tasks
- Run evaluation using lm_eval command
- Analyze results from output file
- Troubleshoot common issues based on documentation
Prerequisites
- Python 3.8+
- pip package manager
- CUDA-enabled GPU (recommended for speed)
Installation
First, add the marketplace
/plugin marketplace add Orchestra-Research/AI-Research-SKILLs/plugin install AI-Research-SKILLs@ai-research-skillsQuality Score
VerifiedTrust Signals
Similar Extensions
Evaluating Llms Harness
99Evaluates LLMs across 60+ academic benchmarks (MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag). Use when benchmarking model quality, comparing models, reporting academic results, or tracking training progress. Industry standard used by EleutherAI, HuggingFace, and major labs. Supports HuggingFace, vLLM, APIs.
Context Compression
100This skill should be used when the user asks to "compress context", "summarize conversation history", "implement compaction", "reduce token usage", or mentions context compression, structured summarization, tokens-per-task optimization, or long-running agent sessions exceeding context limits.
Scholar Evaluation
98Systematically evaluate scholarly work using the ScholarEval framework, providing structured assessment across research quality dimensions including problem formulation, methodology, analysis, and writing with quantitative scoring and actionable feedback.
Nemo Evaluator Sdk
98Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.
BigCode Evaluation Harness
98Evaluates code generation models across HumanEval, MBPP, MultiPL-E, and 15+ benchmarks with pass@k metrics. Use when benchmarking code models, comparing coding abilities, testing multi-language support, or measuring code generation quality. Industry standard from BigCode Project used by HuggingFace leaderboards.
Literature Review
100Conduct comprehensive, systematic literature reviews using multiple academic databases (PubMed, arXiv, bioRxiv, Semantic Scholar, etc.). This skill should be used when conducting systematic literature reviews, meta-analyses, research synthesis, or comprehensive literature searches across biomedical, scientific, and technical domains. Creates professionally formatted markdown documents and PDFs with verified citations in multiple citation styles (APA, Nature, Vancouver, etc.).