#
Llama-3.1-8b performance on AWS Inferentia2 (Latency & Throughput)

How fast is Llama-3.1-8b on Inferentia2? Let’s figure out!

For this benchmark we will use the following configurations:

Model type | batch_size | sequence_length
--- | --- | ---
Llama3.1 8b BS1 | 1 | 4096
Llama3.1 8b BS4 | 4 | 4096
Llama3.1 8b BS8 | 8 | 4096
Llama3.1 8b BS16 | 16 | 4096
Llama3.1 8b BS32 | 32 | 4096
Llama3.1 8b BS48 | 48 | 4096

Note: all models are compiled to use 4 devices corresponding to 8 cores on the `inf2.48xlarge` instance.

Note: please refer to the inferentia2 product page for details on the available instances.

##
Time to first token

The time to first token is the time required to process the input tokens and generate the first output token. It is a very important metric, as it corresponds to the latency directly perceived by the user when streaming generated tokens.

We test the time to first token for increasing context sizes, from a typical Q/A usage, to heavy Retrieval Augmented Generation (RAG) use-cases.

Time to first token is expressed in seconds.

##
Inter-token Latency

The inter-token latency corresponds to the average time elapsed between two generated tokens.

It is expressed in milliseconds.

###
Throughput

Unlike some other benchmarks, we evaluate the throughput using generated tokens only, by dividing their number by the end-to-end latency.

Throughput is expressed in tokens/second.