#
Evaluate LLMs with Hugging Face Lighteval on Amazon SageMaker

In this sagemaker example, we are going to learn how to evaluate LLMs using Hugging Face lighteval. LightEval is a lightweight LLM evaluation suite that powers Hugging Face Open LLM Leaderboard.

Evaluating LLMs is crucial for understanding their capabilities and limitations, yet it poses significant challenges due to their complex and opaque nature. LightEval facilitates this evaluation process by enabling LLMs to be assessed on acamedic benchmarks like MMLU or IFEval, providing a structured approach to gauge their performance across diverse tasks.

In Detail you will learn how to:

1. Setup Development Environment
2. Prepare the evaluation configuraiton
3. Evaluate Zephyr 7B on TruthfulQA on Amazon SageMaker

    !pip install 'sagemaker<3.0.0' --upgrade --quiet

> SageMaker Python SDK v3 has been recently released, so unless specified otherwise, all the documentation and tutorials are still using the SageMaker Python SDK v2. We are actively working on updating all the tutorials and examples, but in the meantime make sure to install the SageMaker SDK as `pip install "sagemaker<3.0.0"`.

If you are going to use Sagemaker in a local environment. You need access to an IAM Role with the required permissions for Sagemaker. You can find here more about it.

    import sagemaker
    import boto3
    sess = sagemaker.Session()
    # sagemaker session bucket -> used for uploading data, models and logs
    # sagemaker will automatically create this bucket if it not exists
    sagemaker_session_bucket=None
    if sagemaker_session_bucket is None and sess is not None:
        # set to default bucket if a bucket name is not given
        sagemaker_session_bucket = sess.default_bucket()

    try:
        role = sagemaker.get_execution_role()
    except ValueError:
        iam = boto3.client('iam')
        role = iam.get_role(RoleName='sagemaker_execution_role')['Role']['Arn']

    sess = sagemaker.Session(default_bucket=sagemaker_session_bucket)

    print(f"sagemaker role arn: {role}")
    print(f"sagemaker bucket: {sess.default_bucket()}")
    print(f"sagemaker session region: {sess.boto_region_name}")

##
2. Prepare the evaluation configuraiton

LightEval includes script to evaluate LLMs on common benchmarks like MMLU, Truthfulqa, IFEval, and more. It is used to evaluate models on the Hugging Face Open LLM Leaderboard. lighteval isy built on top of the great Eleuther AI Harness with some additional features and improvements.

You can find all available benchmarks here.

We are going to use Amazon SageMaker Managed Training to evaluate the model. Therefore we will leverage the script available in lighteval. The Hugging Face DLC is not having lighteval installed. This means need to provide a `requirements.txt` file to install the required dependencies.

First lets load the `run_evals_accelerate.py` script and create a `requirements.txt` file with the required dependencies.

    import os
    import requests as r

    lighteval_version = "0.2.0"

    # create scripts directory if not exists
    os.makedirs("scripts", exist_ok=True)

    # load custom scripts from git
    raw_github_url = f"https://raw.githubusercontent.com/huggingface/lighteval/v{lighteval_version}/run_evals_accelerate.py"
    res = r.get(raw_github_url)
    with open("scripts/run_evals_accelerate.py", "w") as f:
        f.write(res.text)

    # write requirements.txt
    with open("scripts/requirements.txt", "w") as f:
        f.write(f"lighteval=={lighteval_version}")

In lighteval, the evaluation is done by running the `run_evals_accelerate.py` script. The script takes a `task` argument which is defined as `suite|task|num_few_shot|{0 or 1 to automatically reduce num_few_shot if prompt is too long}`. Alternatively, you can also provide a path to a txt file with the tasks you want to evaluate the model on, which we are going to do. This makes it easier for you to extend the evaluation to other benchmarks.

We are going to evaluate the model on the Truthfulqa benchmark with 0 few-shot examples. TruthfulQA is a benchmark designed to measure whether a language model generates truthful answers to questions, encompassing 817 questions across 38 categories including health, law, finance, and politics​​.

    with open("scripts/tasks.txt", "w") as f:
        f.write(f"lighteval|truthfulqa:mc|0|0")

To evaluate a model on all the benchmarks of the Open LLM Leaderboard you can copy this file

##
3. Evaluate Zephyr 7B on TruthfulQA on Amazon SageMaker

In this example we are going to evaluate the HuggingFaceH4/zephyr-7b-beta on the MMLU benchmark, which is part of the Open LLM Leaderboard.

In addition to the `task` argument we need to define:

- `model_args`: Hugging Face Model ID or path, defined as `pretrained=HuggingFaceH4/zephyr-7b-beta`
- `model_dtype`: The model data type, defined as `bfloat16`, `float16` or `float32`
- `output_dir`: The directory where the evaluation results will be saved, e.g. `/opt/ml/model`

Lightevals can also evaluat peft models or use `chat_templates` you find more about it here.

    from sagemaker.huggingface import HuggingFace

    # hyperparameters, which are passed into the training job
    hyperparameters = {
      'model_args': "pretrained=HuggingFaceH4/zephyr-7b-beta", # Hugging Face Model ID
      'task': 'tasks.txt',  # 'lighteval|truthfulqa:mc|0|0',
      'model_dtype': 'bfloat16', # Torch dtype to load model weights
      'output_dir': '/opt/ml/model' # Directory, which sagemaker uploads to s3 after training
    }

    # create the Estimator
    huggingface_estimator = HuggingFace(
        entry_point          = 'run_evals_accelerate.py',      # train script
        source_dir           = 'scripts',         # directory which includes all the files needed for training
        instance_type        = 'ml.g5.4xlarge',   # instances type used for the training job
        instance_count       = 1,                 # the number of instances used for training
        base_job_name        = "lighteval",          # the name of the training job
        role                 = role,              # Iam role used in training job to access AWS ressources, e.g. S3
        volume_size          = 300,               # the size of the EBS volume in GB
        transformers_version = '4.36',            # the transformers version used in the training job
        pytorch_version      = '2.1',            # the pytorch_version version used in the training job
        py_version           = 'py310',            # the python version used in the training job
        hyperparameters      =  hyperparameters,
        environment          = {
                                "HUGGINGFACE_HUB_CACHE": "/tmp/.cache",
                                # "HF_TOKEN": "REPALCE_WITH_YOUR_TOKEN" # needed for private models
                                }, # set env variable to cache models in /tmp
    )

We can now start our evaluation job, with the `.fit()`.

    # starting the train job with our uploaded datasets as input
    huggingface_estimator.fit()

After the evaluation job is finished, we can download the evaluation results from the S3 bucket. Lighteval will save the results and generations in the `output_dir`. The results are savedas json and include detailed information about each task and the model’s performance. The results are available in the `results` key.

    import tarfile
    import json
    import io
    import os
    from sagemaker.s3 import S3Downloader

    # download results from s3
    results_tar = S3Downloader.read_bytes(huggingface_estimator.model_data)
    model_id = hyperparameters["model_args"].split("=")[1]
    result={}

    # Use tarfile to open the tar content directly from bytes
    with tarfile.open(fileobj=io.BytesIO(results_tar), mode="r:gz") as tar:
        # Iterate over items in tar archive to find your json file by its path
        for member in tar.getmembers():
            # get path of results based on model id used to evaluate
            if os.path.join("details", model_id) in member.name and member.name.endswith('.json'):
                # Extract the file content
                f = tar.extractfile(member)
                if f is not None:
                    content = f.read()
                    result = json.loads(content)
                    break

    # print results
    print(result["results"])
    # {'lighteval|truthfulqa:mc|0': {'truthfulqa_mc1': 0.40636474908200737, 'truthfulqa_mc1_stderr': 0.017193835812093897, 'truthfulqa_mc2': 0.5747003398184238, 'truthfulqa_mc2_stderr': 0.015742356478301463}}

In our test we achieved a `mc1` score of 40.6% and an `mc2` score of 57.47%. The `mc2` is the score used in the Open LLM Leaderboard. Zephyr 7B achieved a `mc2` score of 57.47% on the TruthfulQA benchmark, which is identical to the score on the Open LLM Leaderboard. The evaluation on Truthfulqa took `999 seconds`. The ml.g5.4xlarge instance we used costs `$2.03 per hour` for on-demand usage. As a result, the total cost for evaluating Zephyr 7B on Truthfulqa was `$0.56`.

> 📍 Find the complete example on GitHub here!

Update on GitHub