Nemo container for Nemotron 340B inference fails pytorch_lightning import
The error was due to a different pytorch version being installed outside of the container in `~/.local/lib/python3.10/site-packages`. Since my home directory was mounted the container prioritized this version over the one in the container and thus there was a mismatch between `torch` and `torchvision`. 

The local Pytorch version was installed by mistake... So make sure to delete or unmount any python installations that might interfere with packages installed in the container. 
**Describe the bug**

Nvidia recently released Nemotron 340B. The [README on Huggingface provides a guide](https://huggingface.co/nvidia/Nemotron-4-340B-Instruct) for performing inference with this model. I followed the guide using the recommended container `nvcr.io/nvidia/nemo:24.01.framework`. 

However, this container is unable to import `pytorch_lightning` without errors. The inference script tries to import `pytorch_lightning`, which in turn attempts to verify it can import `torchvision` (which it fails to do). I suspect the error is due to the version of `torchvision` installed not being compatible with the `torch` version (`RuntimeError: operator torchvision::nms does not exist`). 

I have encountered similar errors related to importing `pytorch_lightning` in your older containers previously as well. It happens more frequently when alpha/nightly versions of python packages are installed. A lot of containers tend to have alpha/nightly versions of either `torch`, `torchtext` or `torchvision` installed. 

