---
id: 1280
card_url: "https://mdrss.com/llm-engineering/serving-and-retrieval/1280"
permalink_url: "https://mdrss.com/m/1280"
thread_url: "https://mdrss.com/s/llm-engineering"
markdown_url: "https://mdrss.com/llm-engineering/serving-and-retrieval/1280/1280.md"
title: "Evalscope"
annotation: "δΈζ ο½ English π δΈζζζ‘£ ο½ π English Documentation EvalScope is a one-stop LLM evaluation framework built by the ModelScope Community. Just one command to start β it supports model capability evaluation, inference performance stress testing, and result visualization."
state: published
thread: llm-engineering
domain: llm-engineering
category: serving-and-retrieval
type: guide
tags: ["evaluation", "llm", "performance", "rag", "vlm", "python", "evals", "visualization"]
ontology_terms: []
relation_terms: []
license: "Apache-2.0"
version: 1
snapshot_at: "2026-08-04T12:18:33.634Z"
source_url: "https://github.com/modelscope/evalscope"
source_kind: "mdrss-final-catalog"
platform_scam_risk: null
platform_evidence_score: 0
evidence_urls:
- "https://github.com/modelscope/evalscope"
---
δΈζ ο½ English
π δΈζζζ‘£ ο½ π English Documentation
> β If you like this project, please click the "Star" button in the upper right corner to support us. Your support is our motivation to move forward!
## π Introduction
EvalScope is a one-stop LLM evaluation framework built by the [ModelScope Community](https://modelscope.cn/). Just one command to start β it supports model capability evaluation, inference performance stress testing, and result visualization.
```bash
pip install evalscope
evalscope eval --model your-model-name --api-url $OPENAI_API_BASE_URL --api-key $OPENAI_API_KEY --eval-type openai_api --datasets gsm8k --limit 5
```
## β¨ Key Features
- **π Comprehensive Evaluation Benchmarks**: Built-in multiple industry-recognized evaluation benchmarks including MMLU, C-Eval, GSM8K, and more.
- **𧩠Multi-modal and Multi-domain Support**: Supports evaluation of various model types including Large Language Models (LLM), Vision Language Models (VLM), Embedding, Reranker, AIGC, and more.
- **π Multi-backend Integration**: Seamlessly integrates multiple evaluation backends including OpenCompass, VLMEvalKit, RAGEval to meet different evaluation needs.
- **π€ Agent Evaluation Mode**: Drives benchmarks (e.g. GSM8K, AIME, SWE-bench Agentic) inside a controlled multi-turn AgentLoop with pluggable strategies, tools and Docker sandbox; full per-sample Agent Trace is recorded and visualizable.
- **β‘ Inference Performance Testing**: Provides powerful model service stress testing tools, supporting multiple performance metrics such as TTFT, TPOT.
- **π Interactive Reports**: Provides WebUI visualization interface, supporting multi-dimensional model comparison, report overview and detailed inspection.
- **βοΈ Arena Mode**: Supports multi-model battles (Pairwise Battle), intuitively ranking and evaluating models.
- **π§ Highly Extensible**: Developers can easily add custom datasets, models and evaluation metrics.
## π Visualization Preview
EvalScope provides an interactive Web Dashboard for multi-dimensional model comparison and in-depth analysis.
Dashboard Overview
|
Model Comparison
|
Report Overview
|
Prediction Details
|
For details, please refer to [π Visualizing Evaluation Results](https://evalscope.readthedocs.io/en/latest/get_started/visualization.html).
## π What's New
- π₯ **[2026.07.21]** Added **Claw-Eval**, **ResearchRubrics**, **Toolathlon** (agent), **TVBench** (video), **WideSearch**, and **PerspectiveGap** benchmarks.
- π₯ **[2026.07.03]** Added **CharXiv** & **BabyVision** (chart understanding, visual cognition) and **ERQA** & **WorldVQA** (entity-recognition QA with LLM-judge + CoT) multimodal benchmarks.
- π₯ **[2026.06.23]** Major agent & code evaluation expansion: added **BigCodeBench**, **SWE-bench Multilingual**, **BrowseComp**, **MCP-Atlas**, **GDPval** benchmarks; added **OpenCode** / **OpenHands** runners; refactored adapter architecture with `AudioLanguageAdapter`, unified `FunctionCallAdapter`, and public `run_agent_loop` API.
- π₯ **[2026.06.16]** Added full-reference **image quality metrics** (SSIM, PSNR, etc.), long-context benchmarks (**LoCoMo QA**, **LongMemEval**), **Caption** & **Maritime-OCR-Bench** benchmarks; perf module now supports unified `--data-source` and parallelized request generation.
- π₯ **[2026.06.02]** Refactored **RAG evaluation** module: upgraded to MTEB 2.x and RAGAS 0.4.x, with unified Pydantic-based configs. See the [RAGEval guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/index.html).
- π₯ **[2026.05.27]** Added **Trie agentic trace replay** for perf benchmarking: three new dataset plugins (`trie_agentic_coding` / `trie_code_qa` / `trie_office_work`) replay real multi-turn agent traces with per-turn token caps and tool-call latency simulation. Also introduced a `--duration` wall-clock budget for all benchmark modes and a `Turn` dataclass for per-turn overrides.
- π₯ **[2026.05.27]** Added **Vendor Verifier benchmarks** (`k2_verifier`, `kimi_verifier`, `minimax_verifier`) for validating whether third-party API deployments faithfully reproduce official model behavior, with a shared `FunctionCallAdapter` base class.
- π₯ **[2026.05.26]** Added the [GAIA](https://evalscope.readthedocs.io/en/latest/third_party/gaia.html) agent benchmark (multi-turn ReAct + `bash` in a Docker sandbox, official rule-based scorer) and generic [MCP server](https://evalscope.readthedocs.io/en/latest/user_guides/agent/native.html#mcp-server-tools) support β any `NativeAgentConfig`-driven benchmark can now plug in stdio / HTTP / SSE MCP servers (`fetch`, web search, GitHub, ...) without per-benchmark wiring.
- π₯ **[2026.05.22]** Introduced the **External Agent Bridge** mode: evaluate off-the-shelf agent CLIs such as Anthropic's [Claude Code](https://github.com/anthropics/claude-code) and OpenAI's [Codex](https://github.com/openai/codex) directly through EvalScope. The bridge transparently forwards each CLI's LLM traffic (Anthropic Messages / OpenAI Chat / OpenAI Responses, including SSE streaming) to your evaluation model, while recording the full trajectory as an `agent_trace`. Bring-your-own-runner via `@register_runner`. See the [External Agent Bridge guide](https://evalscope.readthedocs.io/en/latest/user_guides/agent/bridge.html).
- π₯ **[2026.05.19]** Added support for [SWE-bench_Pro](https://evalscope.readthedocs.io/en/latest/third_party/swe_bench_pro.html) and [ΟΒ³-bench](https://evalscope.readthedocs.io/en/latest/third_party/tau3_bench.html): SWE-bench_Pro is a more challenging multilingual long-horizon software-engineering benchmark from Scale AI (recommended over the original SWE-bench for less data contamination and broader language coverage; per-instance Docker images are pulled directly from DockerHub, no local image build required); ΟΒ³-bench is the v1.0.0 release of the tau-bench family, extending ΟΒ²-bench with a new `banking_knowledge` retrieval domain (RAG), 75+ task fixes across existing domains, and pluggable retrieval pipelines (BM25 / embeddings / rerankers / sandbox shell).
- π₯ **[2026.05.15]** Introduced **Agent Evaluation Mode**: any benchmark based on `DefaultDataAdapter` (GSM8K, AIME, IFEval, etc.) can now be driven through a multi-turn AgentLoop with pluggable strategies (`function_calling` / `react` / `swe_bench_*`), tools (`bash` / `python_exec` / `submit`) and `local` / `docker` environments. Per-sample `agent_trace` is recorded and rendered step-by-step in the dashboard's Predictions tab. See the [Agent Evaluation guide](https://evalscope.readthedocs.io/en/latest/user_guides/agent/native.html) for details.
- π₯ **[2026.05.08]** Partnered with [LightSeek](https://lightseek.org/) to launch [TokenSpeed](https://lightseek.org/blog/lightseek-tokenspeed.html), a speed-of-light LLM inference engine for agentic workloads. EvalScope provides the SWE-smith benchmarking pipeline β using real coding-agent traces to measure per-GPU throughput (TPM) and per-user latency (TPS) β serving as the official benchmark tool for TokenSpeed performance evaluation. Refer to the [SWE-smith usage guide](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/multi_turn.html#swe-smith) to get started.
More historical updates
- π₯ **[2026.05.07]** Replaced the Gradio-based WebUI with a new React + Vite web interface for better performance and user experience.
- π₯ **[2026.04.23]** Added support for recording performance (perf) metrics during evaluation tasks, enabling simultaneous tracking of model accuracy and inference efficiency metrics such as TTFT, TPOT, and throughput in a single evaluation run.
- π₯ **[2026.04.17]** Added support for multi-turn conversation performance stress testing, enabling load testing of dialogue-based model services with multi-turn context. Refer to the [usage documentation](https://evalscope.readthedocs.io/en/latest/user_guides/stress_test/examples.html).
- π₯ **[2026.04.10]** Added support for [TIR-Bench](https://arxiv.org/abs/2511.01833) (Thinking-with-Images Reasoning Benchmark), a multimodal benchmark evaluating agentic visual reasoning capabilities of vision-language models.
- π₯ **[2026.03.24]** Added support for Agent Skill. Any agent model that supports Skill/Tool calling can use natural language to drive EvalScope for model evaluation, performance benchmarking, and result visualization.
- π₯ **[2026.03.09]** Added support for evaluation progress tracking and HTML format visualization report generation.
- π₯ **[2026.03.02]** Added support for Anthropic Claude API evaluation. Use `--eval-type anthropic_api` to evaluate models via Anthropic API service.
- π₯ **[2026.02.03]** Comprehensive update to dataset documentation, adding data statistics, data samples, usage instructions and more.
- π₯ **[2026.01.13]** Added support for Embedding and Rerank model service stress testing.
- π₯ **[2025.12.26]** Added support for Terminal-Bench-2.0, which evaluates AI Agent performance on 89 real-world multi-step terminal tasks.
- π₯ **[2025.12.18]** Added support for SLA auto-tuning model API services.
- π₯ **[2025.12.16]** Added support for audio evaluation benchmarks such as Fleurs, LibriSpeech; added support for multilingual code evaluation benchmarks such as MultiplE, MBPP.
- π₯ **[2025.12.02]** Added support for custom multimodal VQA evaluation; added support for visualizing model service stress testing in ClearML.
- π₯ **[2025.11.26]** Added support for OpenAI-MRCR, GSM8K-V, MGSM, MicroVQA, IFBench, SciCode benchmarks.
- π₯ **[2025.11.18]** Added support for custom Function-Call (tool invocation) datasets to test whether models can timely and correctly call tools.
- π₯ **[2025.11.14]** Added support for SWE-bench_Verified, SWE-bench_Lite, SWE-bench_Verified_mini code evaluation benchmarks.
- π₯ **[2025.11.12]** Added `pass@k`, `vote@k`, `pass^k` and other metric aggregation methods; added support for multimodal evaluation benchmarks such as A_OKVQA, CMMU, ScienceQA, V*Bench.
- π₯ **[2025.11.07]** Added support for ΟΒ²-bench, an extended and enhanced version of Ο-bench that includes a series of code fixes and adds telecom domain troubleshooting scenarios.
- π₯ **[2025.10.30]** Added support for BFCL-v4, enabling evaluation of agent capabilities including web search and long-term memory.
- π₯ **[2025.10.27]** Added support for LogiQA, HaluEval, MathQA, MRI-QA, PIQA, QASC, CommonsenseQA and other evaluation benchmarks. Thanks to @[penguinwang96825](https://github.com/penguinwang96825) for the code implementation.
- π₯ **[2025.10.26]** Added support for Conll-2003, CrossNER, Copious, GeniaNER, HarveyNER, MIT-Movie-Trivia, MIT-Restaurant, OntoNotes5, WNUT2017 and other Named Entity Recognition evaluation benchmarks. Thanks to @[penguinwang96825](https://github.com/penguinwang96825) for the code implementation.
- π₯ **[2025.10.21]** Optimized sandbox environment usage in code evaluation, supporting both local and remote operation modes.
- π₯ **[2025.10.20]** Added support for evaluation benchmarks including PolyMath, SimpleVQA, MathVerse, MathVision, AA-LCR; optimized evalscope perf performance to align with vLLM Bench.
- π₯ **[2025.10.14]** Added support for OCRBench, OCRBench-v2, DocVQA, InfoVQA, ChartQA, and BLINK multimodal image-text evaluation benchmarks.
- π₯ **[2025.09.22]** Code evaluation benchmarks (HumanEval, LiveCodeBench) now support running in a sandbox environment.
- π₯ **[2025.09.19]** Added support for multimodal image-text evaluation benchmarks including RealWorldQA, AI2D, MMStar, MMBench, and OmniBench, as well as pure text evaluation benchmarks such as Multi-IF, HealthBench, and AMC.
- π₯ **[2025.09.05]** Added support for vision-language multimodal model evaluation tasks, such as MathVista and MMMU.
- π₯ **[2025.09.04]** Added support for image editing task evaluation, including the [GEdit-Bench](https://modelscope.cn/datasets/stepfun-ai/GEdit-Bench) benchmark.
- π₯ **[2025.08.22]** Version 1.0 Refactoring. Break changes, please [refer to](https://evalscope.readthedocs.io/en/latest/get_started/basic_usage.html#switching-to-version-v1-0).
- π₯ **[2025.07.18]** The model stress testing now supports randomly generating image-text data for multimodal model evaluation.
- π₯ **[2025.07.16]** Support for [Ο-bench](https://github.com/sierra-research/tau-bench) has been added.
- π₯ **[2025.07.14]** Support for "Humanity's Last Exam" ([Humanity's-Last-Exam](https://modelscope.cn/datasets/cais/hle)).
- π₯ **[2025.07.03]** Refactored Arena Mode.
- π₯ **[2025.06.28]** Optimized custom dataset evaluation; enhanced LLM judge usage.
- π₯ **[2025.06.19]** Added support for the [BFCL-v3](https://modelscope.cn/datasets/AI-ModelScope/bfcl_v3) benchmark.
- π₯ **[2025.06.02]** Added support for the Needle-in-a-Haystack test.
- π₯ **[2025.05.29]** Added support for two long document evaluation benchmarks: DocMath and FRAMES.
- π₯ **[2025.05.16]** Model service performance stress testing now supports setting various levels of concurrency.
- π₯ **[2025.05.13]** Added support for the ToolBench-Static dataset, DROP and Winogrande benchmarks.
- π₯ **[2025.04.29]** Added Qwen3 Evaluation Best Practices.
- π₯ **[2025.04.27]** Support for text-to-image evaluation.
- π₯ **[2025.04.10]** Model service stress testing tool now supports the `/v1/completions` endpoint.
- π₯ **[2025.04.08]** Support for evaluating embedding model services compatible with the OpenAI API has been added.
- π₯ **[2025.03.27]** Added support for AlpacaEval and ArenaHard evaluation benchmarks.
- π₯ **[2025.03.20]** The model inference service stress testing now supports generating prompts of specified length using random values.
- π₯ **[2025.03.13]** Added support for the LiveCodeBench code evaluation benchmark.
- π₯ **[2025.03.11]** Added support for the SimpleQA and Chinese SimpleQA evaluation benchmarks.
- π₯ **[2025.03.07]** Added support for the QwQ-32B model evaluation.
- π₯ **[2025.03.04]** Added support for the SuperGPQA dataset.
- π₯ **[2025.03.03]** Added support for evaluating the IQ and EQ of models.
- π₯ **[2025.02.27]** Added support for evaluating the reasoning efficiency of models.
- π₯ **[2025.02.25]** Added support for MuSR and ProcessBench benchmarks.
- π₯ **[2025.02.18]** Supports the AIME25 dataset.
- π₯ **[2025.02.13]** Added support for evaluating DeepSeek distilled models.
- π₯ **[2025.01.20]** Support for visualizing evaluation results.
- π₯ **[2025.01.07]** Native backend: Support for model API evaluation.
- π₯π₯ **[2024.12.31]** Support for adding benchmark evaluations.
- π₯ **[2024.12.13]** Model evaluation optimization.
- π₯ **[2024.11.26]** The model inference service performance evaluator has been completely refactored.
- π₯ **[2024.10.31]** The best practice for evaluating Multimodal-RAG has been updated.
- π₯ **[2024.10.23]** Supports multimodal RAG evaluation.
- π₯ **[2024.10.8]** Support for RAG evaluation.
- π₯ **[2024.09.18]** Documentation added blog module.
- π₯ **[2024.09.12]** Support for LongWriter evaluation.
- π₯ **[2024.08.30]** Support for custom dataset evaluations.
- π₯ **[2024.08.20]** Updated the official documentation.
- π₯ **[2024.08.09]** Simplified the installation process.
- π₯ **[2024.07.31]** Important change: The package name `llmuses` has been changed to `evalscope`.
- π₯ **[2024.07.26]** Support for **VLMEvalKit** as a third-party evaluation framework.
- π₯ **[2024.06.29]** Support for **OpenCompass** as a third-party evaluation framework.
- π₯ **[2024.06.13]** EvalScope integrates with SWIFT; Integrated the Agent evaluation dataset ToolBench.
π‘ Tip: run_task also supports dictionaries, YAML or JSON files as configuration.
**Using Python Dictionary**
```python
from evalscope.run import run_task
task_cfg = {
'model': 'Qwen/Qwen2.5-0.5B-Instruct',
'datasets': ['gsm8k', 'arc'],
'limit': 5
}
run_task(task_cfg=task_cfg)
```
**Using YAML File** (`config.yaml`)
```yaml
model: Qwen/Qwen2.5-0.5B-Instruct
datasets:
- gsm8k
- arc
limit: 5
```
```python
from evalscope.run import run_task
run_task(task_cfg="config.yaml")
```
## π§ͺ Other Evaluation Backends
EvalScope supports launching evaluation tasks through third-party evaluation frameworks (we call them "backends") to meet diverse evaluation needs.
- **Native**: EvalScope's default evaluation framework with comprehensive functionality.
- **OpenCompass**: Focuses on text-only evaluation. [π Usage Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/opencompass_backend.html)
- **VLMEvalKit**: Focuses on multi-modal evaluation. [π Usage Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/vlmevalkit_backend.html)
- **RAGEval**: Focuses on RAG evaluation, supporting Embedding and Reranker models. [π Usage Guide](https://evalscope.readthedocs.io/en/latest/user_guides/backend/rageval_backend/index.html)
- **Third-party Evaluation Tools**: Supports evaluation tasks like [ToolBench](https://evalscope.readthedocs.io/en/latest/third_party/toolbench.html).
ποΈ Overall Architecture
EvalScope Overall Architecture.
1. **Input Layer**
- **Model Sources**: API models (OpenAI API), Local models (ModelScope)
- **Datasets**: Standard evaluation benchmarks (MMLU/GSM8k etc.), Custom data (MCQ/QA)
2. **Core Functions**
- **Multi-backend Evaluation**: Native backend, OpenCompass, MTEB, VLMEvalKit, RAGAS
- **Performance Monitoring**: Supports multiple model service APIs and data formats, tracking TTFT/TPOP and other metrics
- **Tool Extensions**: Integrates Tool-Bench, Needle-in-a-Haystack, etc.
3. **Output Layer**
- **Structured Reports**: Supports JSON, Table, Logs
- **Visualization Platform**: Supports Web Dashboard, Wandb, SwanLab
| 
| 
## π·ββοΈ Contributing
We welcome any contributions from the community! If you want to add new evaluation benchmarks, models, or features, please refer to our [Contributing Guide](https://evalscope.readthedocs.io/en/latest/advanced_guides/add_benchmark.html).
Thanks to all developers who have contributed to EvalScope!
## π Citation
If you use EvalScope in your research, please cite our work:
```bibtex
@misc{evalscope_2024,
title={{EvalScope}: Evaluation Framework for Large Models},
author={ModelScope Team},
year={2024},
url={https://github.com/modelscope/evalscope}
}
```
## β Star History
[](https://star-history.com/#modelscope/evalscope&Date)