[TTS] Fix bugs in E2E TTS, Mixer-TTS and FastPitch
# What does this PR do ?

Add a one line overview of what this PR aims to accomplish.

**Collection**: [TTS]

# Changelog 
- Fix bugs in E2E TTS, Mixer-TTS and FastPitch 

**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation
