Missing BOS tokens for HF tokenizer
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
**Describe the bug**

In the current implementation of HF AutoTokenizer, the tokenizer does not behave in the same way as the HF version. I noticed the problem with the BOS token (which is extremely important for the Gemma 2 model). Specifically, the problem occurs with `text_to_ids` function, https://github.com/NVIDIA/NeMo/blob/ed244d943948b105982f5cb17b51991e8acebfb7/nemo/collections/common/tokenizers/huggingface/auto_tokenizer.py#L222-L225 which is relevant for `preprocess_data_for_megatron.py`. The problem is that this function first transforms text to tokens and then tokens to IDs and consequently does not append the BOS token at the beginning as the HF encode method does (I assume the problem applies to other special tokens as well).

**Steps/Code to reproduce bug**

```
from nemo.collections.common.tokenizers.huggingface.auto_tokenizer import AutoTokenizer as ATNemo
from transformers import AutoTokenizer as ATHF


tokenizer_path = "google/gemma-2-9b"
hf_tokenizer = ATHF.from_pretrained(tokenizer_path)
nemo_tokenizer = ATNemo(tokenizer_path)

text = "Text to tokenize"

# Common HF tokenization
ids = hf_tokenizer.encode(text)
print("HF tokenization:", ids)

# NeMo tokenization
ids = nemo_tokenizer.text_to_ids(text)
print("NeMo tokenization:", ids)

# HF tokenization using NeMo steps
tokens = hf_tokenizer.tokenize(text)
ids = hf_tokenizer.convert_tokens_to_ids(tokens)
print("HF tokenization using NeMo steps:", ids)
```

The outputs for provided code snippet are:
```
HF tokenization: [2, 1637, 577, 223491]
NeMo tokenization: [1637, 577, 223491]
HF tokenization using NeMo steps: [1637, 577, 223491]
```

Notice how HF tokenization have token ID 2 at the beginning, which corresponds to the BOS token.

**Expected behavior**

I expect NeMo HF wrapper to behave the same way as the HF tokenizer. The solution for this case would be to change the `text_to_ids` function to:
```
 def text_to_ids(self, text):
     ids = self.tokenizer.encode(text) 
     return ids 
```

Is there any reason why it isn't implemented this way?

**Environment overview (please complete the following information)**

I was using official NeMo container (24.09). However, I see that code for HF wrapper is still the same on main branch, so the issue should still apply.
