Table of Contents

Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on Large Language Models and exploring the boundaries and limits of Generative AI. Use it to build a structured path from fundamentals to hands-on practice.

Editorial note: curated source snapshot published by Collider.club under the MIT License. Source attribution is preserved in the front matter.

Source snapshot

English | 中文

Awesome-LLM-Eval: a curated list of tools, datasets/benchmark, demos, leaderboard, papers, docs and models, mainly for Evaluation on Large Language Models and exploring the boundaries and limits of Generative AI.

The is the official project of our survey: Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap.

NOTE: As we cannot update the arXiv paper in real time, please refer to this repo for the latest updates and the paper may be updated later. We also welcome any pull request or issues to help us improve this work. Your contributions will be acknowledged in acknowledgements.

If you find our survey useful, please kindly cite our paper:

@misc{wang2025llmevalroadmap,
      title={Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap}, 
      author={Jun Wang and Ninglun Gu and Kailai Zhang and Zijiao Zhang and Yelun Bao and Jin Yang and Xu Yin and Liwei Liu and Yihuan Liu and Pengyong Li and Gary G. Yen and Junchi Yan},
      year={2025},
      eprint={2508.18646},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2508.18646}, 
}

Table of Contents

• News
• Tools
• Datasets / Benchmark
  • General
  • Domain
  • RAG-Evaluation
  • Agent-Capabilities
  • Coding-Capabilities
  • Multimodal/Cross-modal
  • Long-Context
  • Inference-Speed
  • Quantization-and-Compression
• Demos
• Leaderboards
• Papers
• LLM-List
  • Pre-trained LLM
  • Instruction Fine-tuned LLM
  • Aligned LLM
  • Open LLM
  • Popular LLM
• LLMOps
• Frameworks for Training
• Courses
• Others
• Other Awesome Lists
• Licenses
• Citation

News

• [2025/08/20] We added the Anthropomorphic-Taxonomy section.
• [2024/04/26] We added the Inference-Speed section.
• [2024/02/26] We added the Coding-Evaluation section.
• [2024/02/08] We added the lighteval tool from Huggingface.
• [2024/01/15] We added CRUXEval, DebugBench, OpenFinData, and LAiW.
• [2023/12/20] We added the RAG-Evaluation section.
• [2023/11/15] We added Instruction-Following-Evaluation and LLMBar for evaluating the instruction following capabilities of LLMs.
• [2023/10/20] We added SuperCLUE-Agent for LLM agent evaluation.
• [2023/09/25] We added ColossalEval from Colossal-AI.
• [2023/09/22] We added the LeaderboardFinder chapter.
• [2023/09/20] We added DeepEval, FinEval, and SuperCLUE-Safety from CLUEbenchmark.
• [2023/09/18] We added OpenCompass from Shanghai AI Lab.
• [2023/08/03] We added new Chinese LLMs: Baichuan and Qwen.
• [2023/06/28] We added AlpacaEval and multiple tools.
• [2023/04/26] We released the V0.1 evaluation list with multiple benchmarks.

Anthropomorphic-Taxonomy

Typical Intelligence Quotient (IQ)-General Intelligence evaluation benchmarks

Typical Professional Quotient (PQ)-Professional Expertise evaluation benchmarks

Typical Emotional Quotient (EQ)-Alignment Ability evaluation benchmarks

Tools

Datasets or Benchmarks

General