improvements for updated .nemo files for NER and Punctuation models
@yzhang123 after trying out TLT, I found it's cleaner to have the training and evaluation parts separated.
Also, we had multiple issues with exp_manager and PyTorch Lightning updates when multiple trainers were created in the same script.
 > @yzhang123 after trying out TLT, I found it's cleaner to have the training and evaluation parts separated.
> Also, we had multiple issues with exp_manager and PyTorch Lightning updates when multiple trainers were created in the same script.

Fair enough, I am ok with this the other nlp models should follow this pattern. However, could you not create script folder, just put the training and evaluation scripts outside and your scripts can already be identified by their name. Otherwise, this makes the folder structure necessarily deep.  > > @yzhang123 after trying out TLT, I found it's cleaner to have the training and evaluation parts separated.
> > Also, we had multiple issues with exp_manager and PyTorch Lightning updates when multiple trainers were created in the same script.
> 
> Fair enough, I am ok with this the other nlp models should follow this pattern. However, could you not create script folder, just put the training and evaluation scripts outside and your scripts can already be identified by their name. Otherwise, this makes the folder structure necessarily deep.

@yzhang123 done > 
> 
> > > @yzhang123 after trying out TLT, I found it's cleaner to have the training and evaluation parts separated.
> > > Also, we had multiple issues with exp_manager and PyTorch Lightning updates when multiple trainers were created in the same script.
> > 
> > 
> > Fair enough, I am ok with this the other nlp models should follow this pattern. However, could you not create script folder, just put the training and evaluation scripts outside and your scripts can already be identified by their name. Otherwise, this makes the folder structure necessarily deep.
> 
> @yzhang123 done

Are these new scripts for just evaluation or inference or for both? > Are these new scripts for just evaluation or inference or for both?

@VahidooX here I'm just splitting the existing scripts into 2: one for training, and the other one for eval+inference example.

 > 
> 
> > Are these new scripts for just evaluation or inference or for both?
> 
> @VahidooX here I'm just splitting the existing scripts into 2: one for training, and the other one for eval+inference example.

That looks nice! I may do the same for text classification next Monday if I could.
- [X] updated config for new pre-trained checkpoints, included test_ds by default
- [X] split the token_classification script into 2: train and eval+infer
- [X] added more comments on how to use NER scripts
- [X] updated Jenkins tests for NER scripts with new .nemo
- [x] added more comments on how to use Punct scripts
- [x] split the punctuation script into 2: train and eval+infer
- [x] updated Jenkins tests for Punct scripts with new .nemo