Google releases Gemma 2 2B, ShieldGemma and Gemma Scope
Xenova, pcuenq, reach-vb, joaogante
One month after the release of [Gemma 2](https://huggingface.co/blog/gemma2), Google has expanded their set of Gemma models to include the following new additions:
- [Gemma 2 2B](https://huggingface.co/collections/google/gemma-2-2b-release-66a20f3796a2ff2a7c76f98f) - The 2.6B parameter version of Gemma 2, making it a great candidate for on-device use.
- [ShieldGemma](https://huggingface.co/collections/google/shieldgemma-release-66a20efe3c10ef2bd5808c79) - A series of safety classifiers, trained on top of Gemma 2, for developers to filter inputs and outputs of their applications.
- [Gemma Scope](https://huggingface.co/collections/google/gemma-scope-release-66a4271f6f0b4d4a9d5e04e2) - A comprehensive, open suite of sparse autoencoders for Gemma 2 2B and 9B.

Let’s take a look at each of these in turn!

## Gemma 2 2B

For those who missed the previous launches, Gemma is a family of lightweight, state-of-the-art open models from Google, built from the same research and technology used to create the Gemini models. They are text-to-text, decoder-only large language models, available in English, with open weights for both pre-trained variants and instruction-tuned variants. This release introduces the 2.6B parameter version of Gemma 2 ([base](https://huggingface.co/google/gemma-2-2b) and [instruction-tuned](https://huggingface.co/google/gemma-2-2b-it)), complementing the existing 9B and 27B variants.

Gemma 2 2B shares the same architecture as the other models in the Gemma 2 family, and therefore leverages technical features like sliding attention and logit soft-capping. You can check more details in [this section of our previous blog post](https://huggingface.co/blog/gemma2#technical-advances-in-gemma-2). Like in the other Gemma 2 models, we recommend you use `bfloat16` for inference.

### Use with Transformers

With Transformers, you can use Gemma and leverage all the tools within the Hugging Face ecosystem. To use Gemma models with transformers, make sure to use `transformers` from `main` for the latest fixes and optimizations:

```bash
pip install git+https://github.com/huggingface/transformers.git --upgrade
```

You can then use `gemma-2-2b-it` with `transformers` as follows:

```python
from transformers import pipeline
import torch

pipe = pipeline(
    "text-generation",
    model="google/gemma-2-2b-it",
    model_kwargs={"torch_dtype": torch.bfloat16},
    device="cuda", # use “mps” for running it on Mac
)

messages = [
    {"role": "user", "content": "Who are you? Please, answer in pirate-speak."},
]
outputs = pipe(messages, max_new_tokens=256)
assistant_response = outputs[0]["generated_text"][-1]["content"].strip()
print(assistant_response)
```

> Ahoy, matey! I be Gemma, a digital scallywag, a language-slingin' parrot of the digital seas. I be here to help ye with yer wordy woes, answer yer questions, and spin ye yarns of the digital world.  So, what be yer pleasure, eh? 🦜

For more details on using the models with `transformers`, please check [the model cards](https://huggingface.co/google/gemma-2-2b-it).


### Use with llama.cpp

You can run Gemma 2 on-device (on your Mac, Windows, Linux and more) using llama.cpp in just a few minutes.

Step 1: Install llama.cpp

On a Mac you can directly install llama.cpp with brew. To set up llama.cpp on other devices, please take a look here: https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage

```bash
brew install llama.cpp
```
Note: if you are building llama.cpp from scratch then remember to pass the `LLAMA_CURL=1` flag.

Step 2: Run inference

```bash
./llama-cli
  --hf-repo google/gemma-2-2b-it-GGUF \
  --hf-file 2b_it_v2.gguf \
  -p "Write a poem about cats as a labrador" -cnv

```
Additionally, you can run a local llama.cpp server that complies with the OpenAI chat specs:

```bash
./llama-server \
  --hf-repo google/gemma-2-2b-it-GGUF \
  --hf-file 2b_it_v2.gguf
```
After running the server you can simply invoke the endpoint as below:

```bash
curl http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer no-key" \
-d '{
"messages": [
{	
    "role": "system",
    "content": "You are an AI assistant. Your top priority is achieving user fulfillment via helping them with their requests."
},
{
    "role": "user",
    "content": "Write a limerick about Python exceptions"
}
]
}'
```

Note: The above example runs the inference using the official GGUF weights provided by Google in `fp32`. You can create and share custom quants using the [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space!

### Demo

You can chat with the Gemma 2 2B Instruct model on Hugging Face Spaces! [Check it out here](https://huggingface.co/spaces/huggingface-projects/gemma-2-2b-it).

In addition to this you can run the Gemma 2 2B Instruct model directly from a [colab here](https://github.com/Vaibhavs10/gpu-poor-llm-notebooks/blob/main/Gemma_2_2B_colab.ipynb)

### How to prompt Gemma 2

The base model has no prompt format. Like other base models, it can be used to continue an input sequence with a plausible continuation or for zero-shot/few-shot inference. The instruct version has a very simple conversation structure:

```
<start_of_turn>user
knock knock<end_of_turn>
<start_of_turn>model
who is there<end_of_turn>
<start_of_turn>user
LaMDA<end_of_turn>
<start_of_turn>model
LaMDA who?<end_of_turn><eos>
```

This format has to be exactly reproduced for effective use. In [a previous section](#use-with-transformers) we showed how easy it is to reproduce the instruct prompt with the chat template available in `transformers`. 

### Open LLM Leaderboard v2 Evaluation

| Benchmark | google/gemma-2-2B-it | google/gemma-2-2B | [microsoft/Phi-2](https://huggingface.co/microsoft/phi-2) | [Qwen/Qwen2-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2-1.5B-Instruct) |
| :