#
Lighteval

🤗 Lighteval is your all-in-one toolkit for evaluating Large Language Models (LLMs) across multiple backends with ease. Dive deep into your model’s performance by saving and exploring detailed, sample-by-sample results to debug and see how your models stack up.

##
Key Features

###
🚀 Multi-Backend Support

Evaluate your models using the most popular and efficient inference backends:

- `transformers`: Evaluate models on CPU or one or more GPUs using 🤗 Accelerate
- `nanotron`: Evaluate models in distributed settings using ⚡️ Nanotron
- `vllm`: Evaluate models on one or more GPUs using 🚀 VLLM
- `custom`: Evaluate custom models (can be anything)
- `sglang`: Evaluate models using SGLang as backend
- `inference-endpoint`: Evaluate models using Hugging Face’s Inference Endpoints API
- `tgi`: Evaluate models using 🔗 Text Generation Inference running locally
- `litellm`: Evaluate models on any compatible API using LiteLLM
- `inference-providers`: Evaluate models using HuggingFace’s inference providers as backend**: Distributed training and evaluation

###
📊 Comprehensive Evaluation

- Extensive Task Library: 1000s pre-built evaluation tasks
- Custom Task Creation: Build your own evaluation tasks
- Flexible Metrics: Support for custom metrics and scoring
- Detailed Analysis: Sample-by-sample results for deep insights

###
🔧 Easy Customization

Customization at your fingertips: create new tasks, metrics or model tailored to your needs, or browse all our existing tasks and metrics.

###
☁️ Seamless Integration

Seamlessly experiment, benchmark, and store your results on the Hugging Face Hub, S3, or locally.

##
Quick Start

###
Installation

    pip install lighteval

###
Basic Usage

    # Evaluate a model using Transformers backend
    lighteval accelerate \
        "model_name=openai-community/gpt2" \
        "leaderboard|truthfulqa:mc|0"

###
Save Results

    # Save locally
    lighteval accelerate \
        "model_name=openai-community/gpt2" \
        "leaderboard|truthfulqa:mc|0" \
        --output-dir ./results

    # Push to Hugging Face Hub
    lighteval accelerate \
        "model_name=openai-community/gpt2" \
        "leaderboard|truthfulqa:mc|0" \
        --push-to-hub \
        --results-org your-username

Update on GitHub