TopIPL -Iterative pseudo labeling
This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. This PR was closed because it has been inactive for 7 days since being marked as stale. @bonham79 looks like you experimented with SlimIPL? Would be good if you can review the PR as well.  This PR is stale because it has been open for 14 days with no activity. Remove stale label or comment or update or this will be closed in 7 days. This PR was closed because it has been inactive for 7 days since being marked as stale.
# What does this PR do ?

Implementation of iterative pseudo-labeling algorithm with cache mechanism like [slimIPL](https://arxiv.org/pdf/2010.11524). Algorithm is available for CTC and Hybrid models.  It supports tarr, non-tarr, lhotse datasets. Also works with data from aistore.

**Collection**: [Note which collection this PR will affect]


# Changelog 

1. Changed Hybrid and CTC models to provide iterative pseudo label generation.
2. Changed dataset classes to do caching of audio files from aistore only in the beginning of training.
3. Added support for datasets which audio files are in aistore and manifests are local(on clusters).
4. Added dropout change functionality for conformer encoder.
5. Added random access for tarred lhotse datasets.
6. Added new learning rate scheduler for slimIPL
7. Changed some functions in NeMo core for integration newly pseudo-labeled dataset during training.

# Usage
To do iterative pseudo labeling new field "ipl" should be given in the config file with following parameters.

```
  ipl:
    m_epochs: 0  - how many epochs to train model before first PL generation.
    restore_pc: false - restore pc by comparing with already existing transcriptions if there are any
    manifest_filepath: /path/to/manifest 
    tarred_audio_filepaths:  /path/to/tarred/
    is_tarred: false - if the datasets are tarred
    dataset_weights: 1  - what part of the dataset to use (applicable with non-tar datasets)
    limit_train_batches: 170  - to what to change limit_train_batches after PLs are added to train set (for lhotse only)
    cache_manifest: /path/for/cache/manifest - optional 
    dropout: 0.1 - to what to change dropout after first PL generation
    n_l_epochs: 0 - how many epochs to train with changed dropout before adding PLs to train set
    p_cache: 0.2  - probability  with which cache will be updated
    cache_prefix:  - prefix for cache manifests files (optional for non-tar datasets)
    batch_size: 128  - batch size with which PLs will be generated
```
Models for which this will work.

- EncDecHybridRNNTCTCModel
- EncDecHybridRNNTCTCBPEModel
- EncDecCTCModel
- EncDecCTCModelBPE

For usage our training scripts can be used with proper config file.
```
python NeMo/examples/asr/asr_hybrid_transducer_ctc/speech_to_text_hybrid_rnnt_ctc_bpe.py  --config-path=/path/to/config/ --config-name='config.yaml'. 
```
****Note****
If IPL part started and the training stopped for some reason, to continue from the same place in config the following parameters should be given m_epochs = -1 and n_l_updates = 0, otherwise change them correspondingly.

# GitHub Actions CI

The Jenkins CI system has been replaced by GitHub Actions self-hosted runners.

The GitHub Actions CI will run automatically when the "Run CICD" label is added to the PR.
To re-run CI remove and add the label again.
To run CI on an untrusted fork, a NeMo user with write access must first click "Approve and run".

# Before your PR is "Ready for review"
**Pre checks**:
- [ ] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [ ] Bugfix
- [ ] Documentation

If you haven't finished some of the above items you can still open "Draft" PR.


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information
* Related to # (issue)
