#
Custom Domain Evaluation

While standard benchmarks are useful, you might need custom evaluation tasks for your specific domain or use case. `lighteval` makes it easy to create and run custom evaluation tasks.

Custom tasks are for applications where you need to test capabilities that standard benchmarks don’t cover:

- Domain-specific terminology and knowledge
- Custom response formats or requirements
- Business-specific scenarios and edge cases

In reality, this covers most applications and so you should expect to create custom tasks if you’re using an LLM in production.

##
Creating a Custom Task

Following the official LightEval documentation, here’s how to create a simple custom task:

###
1. Create Your Task File

Create a Python file in your project directory (e.g., `my_custom_task.py`):

    from lighteval.tasks.lighteval_task import LightevalTaskConfig
    from lighteval.tasks.requests import Doc
    from lighteval.metrics.metrics_sample import SampleLevelMetric
    from lighteval.metrics.utils import MetricCategory, MetricUseCase
    import numpy as np

    def prompt_fn(line, task_name: str = None):
        """Convert dataset line to evaluation document"""
        return Doc(
            task_name=task_name,
            query=line["question"],
            choices=[f" {c}" for c in line["choices"]],
            gold_index=line["gold"],
            instruction="",
        )

    # Create your custom task
    custom_task = LightevalTaskConfig(
        name="my_custom_task",
        prompt_function=prompt_fn,
        suite=["community"],
        hf_repo="your-username/your-dataset",
        hf_subset="default",
        hf_avail_splits=["test"],
        evaluation_splits=["test"],
        few_shots_split=None,
        few_shots_select=None,
        metric=["exact_match"],
        generation_size=-1,  # Use -1 for multiple choice tasks
    )

    # Store your task
    TASKS_TABLE = [custom_task]

Let’s break down the code:

- `prompt_fn`: This is a function that takes a line of data and returns a `Doc` object.
- `custom_task`: This is a `LightevalTaskConfig` object that defines the task.
- `TASKS_TABLE`: This is a list of `LightevalTaskConfig` objects that defines the tasks.

###
2. Create Your Dataset

Create a simple JSON dataset file (`my_dataset.jsonl`):

    {"question": "What is machine learning?", "choices": ["A", "B", "C", "D"], "gold": 0}
    {"question": "How does a neural network work?", "choices": ["A", "B", "C", "D"], "gold": 1}
    {"question": "What is supervised learning?", "choices": ["A", "B", "C", "D"], "gold": 2}

You can upload your dataset to the Hugging Face Hub and reference it in your task configuration.

You might also want to consider using visual tools to create this evaluation dataset. For example, AI Sheets is a tool that allows you to create and expand datasets using AI.

###
3. Run Your Custom Task

Use the CLI to run your custom task with `vllm`:

    # Run your custom task
    lighteval vllm \
        "model_name=HuggingFaceTB/SmolLM3-3B,dtype=bfloat16" \
        "community|my_custom_task|0|0" \
        --custom-tasks my_custom_task.py

Just like the previous exercises, you can use `HF Jobs` to run your custom task.

    hf jobs uv run \
        --flavor a10g-large \
        --with "lighteval[vllm]" \
        --secrets HF_TOKEN \
        lighteval vllm "model_name=your-model" "community|my_custom_task|0|0" --custom-tasks my_custom_task.py

##
Creating Custom Metrics

`lighteval` comes with built-in metrics for most use cases. However, you can also create custom metrics. For example, you can create a custom metric that returns multiple values per sample:

If you need custom evaluation metrics, you can add them to your task file:

    from lighteval.metrics.metrics_sample import SampleLevelMetric
    from lighteval.metrics.utils import MetricCategory, MetricUseCase
    import numpy as np

    def my_custom_metric(predictions, formatted_doc, **kwargs):
        """Simple custom metric example"""
        response = predictions[0].strip()
        expected = formatted_doc.choices[formatted_doc.gold_index].strip()

        # Your custom logic here
        return float(response.lower() == expected.lower())

    # Register the metric
    custom_metric = SampleLevelMetric(
        metric_name="my_custom_metric",
        higher_is_better=True,
        category=MetricCategory.CUSTOM,
        use_case=MetricUseCase.SCORING,
        sample_level_fn=my_custom_metric,
        corpus_level_fn=np.mean
    )

Then use it in your task configuration:

    custom_task = LightevalTaskConfig(
        name="my_custom_task",
        # ... other parameters ...
        metric=[custom_metric],  # Use your custom metric
    )

Let’s break down the code:

- `my_custom_metric`: This is a function that takes a list of predictions and a formatted document and returns a dictionary of metrics.
- `custom_metric`: This is a `SampleLevelMetric` object that defines the metric.

###
4. Upload Your Dataset to Hugging Face Hub

For easier sharing and reproducibility:

    # Upload your dataset
    huggingface-cli upload your-username/my-custom-dataset ./my_dataset.jsonl

Then reference it in your task:

    custom_task = LightevalTaskConfig(
        name="my_custom_task",
        hf_repo="your-username/my-custom-dataset",
        # ... other parameters
    )

Here are some best practices to keep things simple:

Begin with a small dataset and basic metrics:

- Start with 10-50 examples to test your task setup
- Use built-in metrics like `exact_match` initially
- Add complexity gradually as you validate your approach

##
What You’ve Learned

You now know how to:

- Create custom evaluation tasks following LightEval’s documentation
- Set up simple datasets for domain-specific evaluation
- Run custom tasks using the CLI with vLLM
- Use HF Jobs for cloud-based custom evaluation
- Create basic custom metrics when needed

For more advanced features and examples, check out the official LightEval documentation and the community tasks directory.

< > Update on GitHub