Can we use ASR pre-trained models with SSL pre-training code?
Hi,
I have used the [SSL pre-training script](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/speech_pretraining/speech_pre_training.py) as follows:
```sh
python speech_pre_training.py \
    --config-path="../conf/ssl/citrinet/" \
    --config-name="citrinet_ssl_1024" \
    model.train_ds.manifest_filepath="./train_LS_manifest.json" \
    model.validation_ds.manifest_filepath="./test_LS_manifest.json" \
    trainer.gpus=1 \
    trainer.accelerator="ddp" \
    trainer.max_epochs=100 \
    trainer.log_every_n_steps=100 \
    trainer.accumulate_grad_batches=1 \
    trainer.check_val_every_n_epoch=1 \
    trainer.precision=32 \
    model.train_ds.batch_size=12 \
    model.validation_ds.batch_size=10 \
    model.spec_augment.freq_masks=4 \
    model.spec_augment.time_masks=10 \
    model.spec_augment.freq_width=27 \
    model.spec_augment.time_width=0.05 \
    model.optim.name="novograd" \
    model.optim.lr=0.05 \
    model.optim.betas=[0.8,0.25] \
    model.optim.weight_decay=0.001 \
    model.optim.sched.warmup_steps=5000 \
    model.optim.sched.min_lr=1e-9 \
    exp_manager.exp_dir="ssl_experiments/" \
    exp_manager.name="SSL-Citrinet1024" \
    exp_manager.create_tensorboard_logger=True \
    exp_manager.resume_if_exists=True \
    exp_manager.resume_ignore_no_checkpoint=True \
    exp_manager.create_wandb_logger=False \
    +init_from_pretrained_model="stt_en_citrinet_1024"
```
However, I have faced an error when the PyTorch lightening attempts to load the pre-trained `stt_en_citrinet_1024` weights (_i.e.,_ `+init_from_pretrained_model="stt_en_citrinet_1024"`). The error refers to the [line 61](https://github.com/NVIDIA/NeMo/blob/main/examples/asr/speech_pretraining/speech_pre_training.py#L61) of code (`maybe_init_from_pretrained_checkpoint` method). I have investigated and I realized that the `maybe_init_from_pretrained_checkpoint` method has the internal attribute of `strict=False` [ref](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/main/core/core.html?highlight=maybe_init_from_pretrained_checkpoint#nemo.core.ModelPT.maybe_init_from_pretrained_checkpoint). So, would you please help me that can I load the pre-trained wights? If yes, How I can do that?
Best 