Out of memory error
I am training Conformer-medium on Multilingual Librispeech Dataset (~45000 hours).
During the training epoch, RAM is constantly increasing until no free space is left. Is there anything i can do to handle this?

Params:
RAM  = 256G
Batch_size = 32
Num_workers = 8

Also when I experimented with 21000 hours, at the end of the epoch, the memory has released.


Here is the trainer params:

trainer:
  gpus: -1
  num_nodes: 1
  max_epochs: 1000
  max_steps: null
  val_check_interval: 0.025 
  accelerator: ddp
  accumulate_grad_batches: 20
  gradient_clip_val: 0.0
  amp_level: O0 
  precision: 32 
  log_every_n_steps: 1  
  progress_bar_refresh_rate: 1
  resume_from_checkpoint: null
  num_sanity_val_steps: 0
  check_val_every_n_epoch: 1 
  sync_batchnorm: true
  checkpoint_callback: false
  logger: false
 