Question about volume mounts while using Nemo container in Kubernetes
**Describe the bug**

When using nemo container `nvcr.io/nvidia/nemo:1.4.0` on a kubernetes environment,  there seems to be increase ephemeral storage usage. I had mounted the experiment directory and data dir as volumes for the pod running the training on, was just wondering if there are other directories that are expected to grow for eg , storage for base model and intermediate models (?) , model dependencies like vocabularies etc (?) . Really not sure any help would be appreciated.  

**Steps/Code to reproduce bug**

On a strict k8s cluster which enforces ephemeral storage limits, I was trying to run the token classification task. 


**Expected behavior**

Hopefully if the directory that is exepcted to grow can be converted into a volume. 

**Environment overview (please complete the following information)**

 - Environment location: Bare-metal
 - Method of NeMo install: Running `nvcr.io/nvidia/nemo:1.4.0` as a Pod with 10Gi ephemeral storage
 

