Holistic Evaluation of Language Models (HELM)

Holistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models. This framework includes the following features:

• Datasets and benchmarks in a standardized format (e.g. MMLU-Pro, GPQA, IFEval, WildBench)
• Models from various providers accessible through a unified interface (e.g. OpenAI models, Anthropic Claude, Google Gemini)
• Metrics for measuring various aspects beyond accuracy (e.g. efficiency, bias, toxicity)
• Web UI for inspecting individual prompts and responses
• Web leaderboard for comparing results across models and benchmarks

Documentation

Please refer to the documentation on Read the Docs for instructions on how to install and run HELM.

Quick Start

Install the package from PyPI:

pip install crfm-helm

Run the following in your shell:

# Run benchmark
helm-run --run-entries mmlu:subject=philosophy,model=openai/gpt2 --suite my-suite --max-eval-instances 10

# Summarize benchmark results
helm-summarize --suite my-suite

# Start a web server to display benchmark results
helm-server --suite my-suite

Then go to http://localhost:8000/ in your browser.

Leaderboards

We maintain offical leaderboards with results from evaluating recent models on notable benchmarks using this framework. Our current flagship leaderboards are:

• HELM Capabilities
• HELM Safety
• Holistic Evaluation of Vision-Language Models (VHELM)

We also maintain leaderboards for a diverse range of domains (e.g. medicine, finance) and aspects (e.g. multi-linguality, world knowledge, regulation compliance). Refer to the HELM website for a full list of leaderboards.

Papers

The HELM framework was used in the following papers for evaluating models.

• Holistic Evaluation of Language Models - paper, leaderboard
• Holistic Evaluation of Vision-Language Models (VHELM) - paper, leaderboard, documentation
• Holistic Evaluation of Text-To-Image Models (HEIM) - paper, leaderboard, documentation
• Image2Struct: Benchmarking Structure Extraction for Vision-Language Models - paper
• Enterprise Benchmarks for Large Language Model Evaluation - paper, documentation
• The Mighty ToRR: A Benchmark for Table Reasoning and Robustness - paper
• Reliable and Efficient Amortized Model-based Evaluation - paper, documentation

The HELM framework can be used to reproduce the published model evaluation results from these papers. To get started, refer to the documentation links above for the corresponding paper, or the main Reproducing Leaderboards documentation.

Citation

If you use this software in your research, please cite the Holistic Evaluation of Language Models paper as below.

@article{
liang2023holistic,
title={Holistic Evaluation of Language Models},
author={Percy Liang and Rishi Bommasani and Tony Lee and Dimitris Tsipras and Dilara Soylu and Michihiro Yasunaga and Yian Zhang and Deepak Narayanan and Yuhuai Wu and Ananya Kumar and Benjamin Newman and Binhang Yuan and Bobby Yan and Ce Zhang and Christian Alexander Cosgrove and Christopher D Manning and Christopher Re and Diana Acosta-Navas and Drew Arad Hudson and Eric Zelikman and Esin Durmus and Faisal Ladhak and Frieda Rong and Hongyu Ren and Huaxiu Yao and Jue WANG and Keshav Santhanam and Laurel Orr and Lucia Zheng and Mert Yuksekgonul and Mirac Suzgun and Nathan Kim and Neel Guha and Niladri S. Chatterji and Omar Khattab and Peter Henderson and Qian Huang and Ryan Andrew Chi and Sang Michael Xie and Shibani Santurkar and Surya Ganguli and Tatsunori Hashimoto and Thomas Icard and Tianyi Zhang and Vishrav Chaudhary and William Wang and Xuechen Li and Yifan Mai and Yuhui Zhang and Yuta Koreeda},
journal={Transactions on Machine Learning Research},
issn={2835-8856},
year={2023},
url={https://openreview.net/forum?id=iO4LZibEqW},
note={Featured Certification, Expert Certification}
}

Tutorial

This tutorial will explain how to use the HELM command line tools to run benchmarks, aggregate statistics, and visualize results.

We will run two runs using the mmlu scenario on the openai/gpt2 model. The mmlu scenario implements the Massive Multitask Language (MMLU) benchmark from this paper, and consists of a Question Answering (QA) task using a dataset with questions from 57 subjects such as elementary mathematics, US history, computer science, law, and more. Note that GPT-2 performs poorly on MMLU, so this is just a proof of concept. We will run two runs: the first using questions about anatomy, and the second using questions about philosophy.

Using helm-run

helm-run is a command line tool for running benchmarks.

To run this benchmark using the HELM command-line tools, we need to specify run entries that describes the desired runs. For this example, the run entries are mmlu:subject=anatomy,model=openai/gpt2 (for anatomy) and mmlu:subject=philosophy,model=openai/gpt2 (for philosophy).

We will now use helm-run to execute the runs. Run this command:

helm-run --run-entries mmlu:subject=anatomy,model=openai/gpt2 mmlu:subject=philosophy,model=openai/gpt2 --suite my-suite --max-eval-instances 10

The meaning of the arguments are as follows:

• --run-entries specifies the run entries from the desired runs.
• --suite specifies a subdirectory under the output directory in which all the output will be placed.
• --max-eval-instances limits evaluation to only N instances (i.e. items) from the benchmark, using a randomly shuffled order of instances.

helm-run creates an environment directory environment and an output directory by default.

• The environment directory is prod_env/ by default and can be set using --local-path. Credentials for making API calls should be added to a credentials.conf file in this directory.
• The output directory is benchmark_output/ by default and can be set using --output-path.

After running this command, navigate to the benchmark_output/runs/my-suite/ directory. This should contain a two sub-directories named mmlu:subject=anatomy,model=openai_gpt2 and mmlu:subject=philosophy,model=openai_gpt2. Note that the names of these sub-directories is based on the run entries we used earlier, but with / replaced with _.

Each output sub-directory will contain several JSON files that were generated during the corresponding run:

• run_spec.json contains the RunSpec, which specifies the scenario, adapter and metrics for the run.
• scenario.json contains a serialized Scenario, which contains the scenario for the run and specifies the instances (i.e. inputs) used.
• scenario_state.json contains a serialized ScenarioState, which contains every request to and response from the model.
• per_instance_stats.json contains a serialized list of PerInstanceStats, which contains the statistics produced for the metrics for each instance (i.e. input).
• stats.json contains a serialized list of PerInstanceStats, which contains the statistics produced for the metrics, aggregated across all instances (i.e. inputs).

Using helm-summarize

The helm-summarize reads the output files of helm-run and computes aggregate statistics across runs. Run the following:

helm-summarize --suite my-suite

This reads the pre-existing files in benchmark_output/runs/my-suite/ that were written by helm-run previously, and writes the following new files back to benchmark_output/runs/my-suite/:

• summary.json contains a serialized ExecutiveSummary with a date and suite name.
• run_specs.json contains the run entries for all the runs.
• runs.json contains serialized list of Run, which contains the run path, run spec and adapter spec and statistics for each run.
• groups.json contains a serialized list of Table, each containing information about groups in a group category.
• groups_metadata.json contains a list of all the groups along with a human-readable description and a taxonomy.

Additionally, for each group and group-relavent metric, it will output a pair of files: benchmark_output/runs/my-suite/groups/latex/<group_name>_<metric_name>.tex and benchmark_output/runs/my-suite/groups/json/<group_name>_<metric_name>.json. These files contain the statistics for that metric from each run within the group.

Using helm-server

Finally, the helm-server command launches a web server to visualize the output files of helm-run and helm-benchmark. Run:

helm-server --suite my-suite

Open a browser and go to http://localhost:8000/ to view the visualization. You should see a similar view as live website for the paper, but for the data from your benchmark runs. The website has the following sections accessible from the top menu bar:

• Leaderboards contains the leaderboards with aggregate metrics.
• Models contains a list of models and their descriptions
• Scenarios contains a list of scenarios and their descriptions.
• Predictions contains a searchable list of runs.