RuntimeError: shape '[16, -1, 128]' is invalid for input of size 113920
@titu1994 @meghmak13
Dear Team,
I am training conformer based SSL model where I have configured the parameters
min_duration : 3.2
patch_size: 16
mask_patches: 0.5
num_negatives: 40
sample_from_same_utterance_only: true
sample_from_non_masked: false
codebook_size: 300
num_groups: 2
num_classes: 90000

but I am getting following error
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1129, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 987, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1129, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 82, in forward
    output = self.module.training_step(*inputs, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/utils/model_utils.py", line 364, in wrap_training_step
    output_dict = wrapped(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/ssl_models.py", line 468, in training_step
    loss_value, loss_val_dict = self.decoder_loss_step(
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/models/ssl_models.py", line 450, in decoder_loss_step
    current_loss_value = current_loss(
  File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1129, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/core/classes/common.py", line 963, in __call__
    outputs = wrapped(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/nemo/collections/asr/losses/ssl_losses/contrastive.py", line 187, in forward
    out_masked_only = out_masked_only.reshape(bs, -1, out_masked_only.shape[-1])
RuntimeError: shape '[16, -1, 128]' is invalid for input of size 113920

I am using Nemo-22.05 images(ngc) and 48, A100-SXM4-40GB GPU i.e 6 nodes for the training. Kindly suggest me what is need to be done ? Thanks in advance