NER task failing with CUDA errors with text blocks > 300 tokens
We are using NeMo to do named entity recognition on blocks of text. The NER task works correctly when the text size we apply is under around 250 or 300 tokens. Once the text size goes above that, the task will start to fail. Between 250 tokens and around 600 tokens, the task will randomly either succeed or fail. Above 500-600 tokens or so, the task fails consistently.

The errors given are somewhat varied, depending on circumstance. They are one of the following: 

```
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`

RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

RuntimeError: CUDA error: device-side assert triggered\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.
```

Following the advice in that last message, the error changes to:
```
RuntimeError: CUDA error: device-side assert triggered
```

The model we are using is a dataset trained off of the medmentions data set, and the .nemo file is 1.2 GB and is being run on a 20GB GPU RAM slice of an A100 card which is shared on a kubernetes cluster. Based on our reading of other error reports, we suspected that running out of GPU RAM might be the ultimate issue. However, with 20 GB of GPU RAM we were surprised that only 300 tokens would be too much for it.

We have written code to chop the text into smaller windows and process it successively. However, both performance and accuracy suffer the smaller the window is. We would very much like to be able to run this on much larger blocks of text if possible.