New extended prompt format for Canary, short utterances inference fix, and training micro-optimizations
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. <!-- pylint-output-strict-mode-false -->

beep boop 🤖: 🙏 The following files have warnings. In case you are familiar with these, please try helping us to improve the code base.

---

Your code was analyzed with PyLint. The following annotations have been identified:

```
************* Module nemo.collections.common.prompts.canary2
nemo/collections/common/prompts/canary2.py:37:0: C0301: Line too long (136/119) (line-too-long)
nemo/collections/common/prompts/canary2.py:30:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/common/prompts/canary2.py:121:0: C0116: Missing function or method docstring (missing-function-docstring)
************* Module scripts.speech_recognition.canary.build_canary_1b_special_tokenizer
scripts/speech_recognition/canary/build_canary_1b_special_tokenizer.py:15:0: W0611: Unused Path imported from pathlib (unused-import)
************* Module scripts.speech_recognition.canary.build_canary_2_special_tokenizer
scripts/speech_recognition/canary/build_canary_2_special_tokenizer.py:86:0: C0116: Missing function or method docstring (missing-function-docstring)
scripts/speech_recognition/canary/build_canary_2_special_tokenizer.py:15:0: W0611: Unused import math (unused-import)
************* Module nemo.collections.asr.data.audio_to_text_lhotse_prompted
nemo/collections/asr/data/audio_to_text_lhotse_prompted.py:60:0: C0301: Line too long (133/119) (line-too-long)
nemo/collections/asr/data/audio_to_text_lhotse_prompted.py:27:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/asr/data/audio_to_text_lhotse_prompted.py:113:0: C0115: Missing class docstring (missing-class-docstring)
************* Module nemo.collections.asr.models.aed_multitask_models
nemo/collections/asr/models/aed_multitask_models.py:695:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/asr/models/aed_multitask_models.py:737:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/asr/models/aed_multitask_models.py:788:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/asr/models/aed_multitask_models.py:796:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/asr/models/aed_multitask_models.py:1070:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/asr/models/aed_multitask_models.py:1136:0: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.collections.asr.modules.transformer.transformer_modules
nemo/collections/asr/modules/transformer/transformer_modules.py:60:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/asr/modules/transformer/transformer_modules.py:104:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/asr/modules/transformer/transformer_modules.py:163:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/asr/modules/transformer/transformer_modules.py:168:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/asr/modules/transformer/transformer_modules.py:219:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/asr/modules/transformer/transformer_modules.py:25:0: W0611: Unused logging imported from nemo.utils (unused-import)
************* Module nemo.collections.common.data.lhotse.dataloader
nemo/collections/common/data/lhotse/dataloader.py:110:0: C0301: Line too long (123/119) (line-too-long)
nemo/collections/common/data/lhotse/dataloader.py:126:0: C0301: Line too long (125/119) (line-too-long)
nemo/collections/common/data/lhotse/dataloader.py:134:0: C0301: Line too long (148/119) (line-too-long)
nemo/collections/common/data/lhotse/dataloader.py:231:0: C0301: Line too long (121/119) (line-too-long)
nemo/collections/common/data/lhotse/dataloader.py:232:0: C0301: Line too long (126/119) (line-too-long)
nemo/collections/common/data/lhotse/dataloader.py:233:0: C0301: Line too long (123/119) (line-too-long)
nemo/collections/common/data/lhotse/dataloader.py:577:0: C0301: Line too long (135/119) (line-too-long)
nemo/collections/common/data/lhotse/dataloader.py:717:0: C0301: Line too long (213/119) (line-too-long)
nemo/collections/common/data/lhotse/dataloader.py:724:0: C0301: Line too long (143/119) (line-too-long)
nemo/collections/common/data/lhotse/dataloader.py:169:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/common/data/lhotse/dataloader.py:426:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/common/data/lhotse/dataloader.py:477:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/common/data/lhotse/dataloader.py:504:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/common/data/lhotse/dataloader.py:510:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/common/data/lhotse/dataloader.py:513:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/common/data/lhotse/dataloader.py:516:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/common/data/lhotse/dataloader.py:519:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/common/data/lhotse/dataloader.py:528:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/common/data/lhotse/dataloader.py:530:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/common/data/lhotse/dataloader.py:533:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/common/data/lhotse/dataloader.py:539:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/common/data/lhotse/dataloader.py:567:0: C0115: Missing class docstring (missing-class-docstring)
nemo/collections/common/data/lhotse/dataloader.py:582:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/common/data/lhotse/dataloader.py:589:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/common/data/lhotse/dataloader.py:604:0: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.collections.common.tokenizers.canary_tokenizer
nemo/collections/common/tokenizers/canary_tokenizer.py:83:0: C0301: Line too long (120/119) (line-too-long)
nemo/collections/common/tokenizers/canary_tokenizer.py:56:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/common/tokenizers/canary_tokenizer.py:60:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/common/tokenizers/canary_tokenizer.py:64:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/common/tokenizers/canary_tokenizer.py:68:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/common/tokenizers/canary_tokenizer.py:108:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/collections/common/tokenizers/canary_tokenizer.py:114:4: C0116: Missing function or method docstring (missing-function-docstring)
************* Module nemo.utils.exp_manager
nemo/utils/exp_manager.py:353:0: C0301: Line too long (120/119) (line-too-long)
nemo/utils/exp_manager.py:378:0: C0301: Line too long (124/119) (line-too-long)
nemo/utils/exp_manager.py:382:0: C0301: Line too long (122/119) (line-too-long)
nemo/utils/exp_manager.py:387:0: C0301: Line too long (127/119) (line-too-long)
nemo/utils/exp_manager.py:401:0: C0301: Line too long (120/119) (line-too-long)
nemo/utils/exp_manager.py:408:0: C0301: Line too long (120/119) (line-too-long)
nemo/utils/exp_manager.py:409:0: C0301: Line too long (126/119) (line-too-long)
nemo/utils/exp_manager.py:411:0: C0301: Line too long (136/119) (line-too-long)
nemo/utils/exp_manager.py:413:0: C0301: Line too long (139/119) (line-too-long)
nemo/utils/exp_manager.py:423:0: C0301: Line too long (123/119) (line-too-long)
nemo/utils/exp_manager.py:424:0: C0301: Line too long (128/119) (line-too-long)
nemo/utils/exp_manager.py:425:0: C0301: Line too long (132/119) (line-too-long)
nemo/utils/exp_manager.py:426:0: C0301: Line too long (127/119) (line-too-long)
nemo/utils/exp_manager.py:514:0: C0301: Line too long (122/119) (line-too-long)
nemo/utils/exp_manager.py:602:0: C0301: Line too long (208/119) (line-too-long)
nemo/utils/exp_manager.py:603:0: C0301: Line too long (230/119) (line-too-long)
nemo/utils/exp_manager.py:605:0: C0301: Line too long (178/119) (line-too-long)
nemo/utils/exp_manager.py:642:0: C0301: Line too long (136/119) (line-too-long)
nemo/utils/exp_manager.py:777:0: C0301: Line too long (121/119) (line-too-long)
nemo/utils/exp_manager.py:785:0: C0301: Line too long (156/119) (line-too-long)
nemo/utils/exp_manager.py:832:0: C0301: Line too long (141/119) (line-too-long)
nemo/utils/exp_manager.py:840:0: C0301: Line too long (152/119) (line-too-long)
nemo/utils/exp_manager.py:1110:0: C0301: Line too long (142/119) (line-too-long)
nemo/utils/exp_manager.py:1134:0: C0301: Line too long (154/119) (line-too-long)
nemo/utils/exp_manager.py:1200:0: C0301: Line too long (120/119) (line-too-long)
nemo/utils/exp_manager.py:1219:0: C0301: Line too long (123/119) (line-too-long)
nemo/utils/exp_manager.py:1245:0: C0301: Line too long (121/119) (line-too-long)
nemo/utils/exp_manager.py:91:0: C0115: Missing class docstring (missing-class-docstring)
nemo/utils/exp_manager.py:106:0: C0115: Missing class docstring (missing-class-docstring)
nemo/utils/exp_manager.py:132:0: C0115: Missing class docstring (missing-class-docstring)
nemo/utils/exp_manager.py:141:0: C0115: Missing class docstring (missing-class-docstring)
nemo/utils/exp_manager.py:150:0: C0115: Missing class docstring (missing-class-docstring)
nemo/utils/exp_manager.py:161:0: C0115: Missing class docstring (missing-class-docstring)
nemo/utils/exp_manager.py:274:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/utils/exp_manager.py:277:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/utils/exp_manager.py:280:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/utils/exp_manager.py:283:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/utils/exp_manager.py:286:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/utils/exp_manager.py:289:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/utils/exp_manager.py:292:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/utils/exp_manager.py:295:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/utils/exp_manager.py:333:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/utils/exp_manager.py:336:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/utils/exp_manager.py:339:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/utils/exp_manager.py:342:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/utils/exp_manager.py:1126:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/utils/exp_manager.py:1227:0: C0116: Missing function or method docstring (missing-function-docstring)
nemo/utils/exp_manager.py:1246:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/utils/exp_manager.py:1249:4: C0116: Missing function or method docstring (missing-function-docstring)
nemo/utils/exp_manager.py:1265:0: C0116: Missing function or method docstring (missing-function-docstring)

-----------------------------------
Your code has been rated at 9.41/10
```

---

Thank you for improving NeMo's documentation! [🤖]: Hi @pzelasko 👋,

We wanted to let you know that a [CICD pipeline](https://github.com/NVIDIA/NeMo/actions/runs/12167255600) for this PR just finished successfully

So it might be time to merge this PR or get some approvals

I'm just a bot so I'll leave it you what to do next.

//cc @pablo-garay @ko3n1g
# What does this PR do ?

* New prompt format 
  * Prompt formatter definition
  * Special tokenizer build script 
  * Unit tests for the new Canary prompt format.
* Micro-optimizations (16% faster training step)
* Option to enable loss mask on prompt tokens
* Fix for inference of very short utterances

**Collection**: ASR

# Changelog 
- Add specific line by line info of high level changes in this PR.

# Usage
* You can potentially add a usage example below

```python
# Add a code snippet demonstrating how to use this 
```

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
