Reduce memory usage in getMultiScaleCosAffinityMatrix function
Reduce memory usage in getMultiScaleCosAffinityMatrix function

Signed-off-by: gabitza-tech <gabriel.pirlogeanu@gmail.com>

# What does this PR do ?

Replace some tensor multiplication in the function for multi scale cos affinity matrix calculation with a single tensor in which we add the repeated embeddings multiplied with its scale weight.

Results show a decrease from 11GB to 2GB for 1 hour audio. It even managed to reduce from above 60GB memory usage for a [53000,53000] tensor to 28GB.
**Collection**: ASR (collections/asr/parts/utils/offline_clustering.py)

# Changelog 
`multiscale_weights = torch.squeeze(multiscale_weights,dim=0).to(device)`
The tensor was [1,6] shaped and had to squeeze the first dimension.
` fused_sim_d = torch.zeros(len(timestamps_in_scales[-1]),len(timestamps_in_scales[-1]))`
Initialized a tensor with the shape [s,s] with 0 values where s is the number of timestamps for the smallest scale. We will add the tensor for each scale to this tensor
` with torch.no_grad():  `
Gradients are not needed for inference.
`fused_sim_d = fused_sim_d + multiscale_weights[scale_idx] * repeated_tensor_1`
Instead of adding the tensor for each scale to a list, then stacking the tensor list and then matrix multiplying it with the weights, we simply add the repeated tensor to the previously initialized tensor before the for loop.

@tango4j is adding followings:
- Clear memory outside torch.jit.script functions since jit-script does not support torch.cuda.empty_cache()
- Fixed the bug where eigen value decomposition does not use GPU. Now GPU usage hits 100%
- Removed torch.jit.script decorators since it slows down if we run without jit.script export.
- Performance (without torch.jit.script) <!--td {border: 1px solid #cccccc;}br {mso-data-placement:same-cell;}-->

sess len in secs | Num of sessions | total time in secs | sec/Iter. | RTFx |  Peak Memory Usage
-- | -- | -- | -- | -- | --
1800 | 10 | 34 | 3.48 | 517.24138 | 3G
3600 | 10 | 70 | 7.02 | 512.82051 | 8G
7200 | 10 | 319 | 31.98 | 225.14071 | 20G

Now, 2hours of audio usually takes 30~40 seconds.


# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [x] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [x] New Feature
- [ ] Bugfix
- [ ] Documentation

## Who can review?

Anyone in the community is free to review the PR once the checks have passed. I would like to ask @titu1994, @redoctopus, @jbalam-nv, or @okuchaiev to review this pull request.
# Additional Information
* Related to https://github.com/NVIDIA/NeMo/issues/5637