ESM-1nv model trains for only one epoch
Hi, I am using AWS parallel cluster for implementation of NVDIA’s bionemo framework (GitHub - aws-samples/awsome-distributed-training: Collection of best practices, reference architectures, model training examples and utilities to train large models on AWS.). When I try to pretrain the ESM-1nv model, the training happens only for one epoch and stops no matter what the training parameter we are providing. I attached the trainer configuration and output below for reference:Output:image903×256 12.6 KBTrainer configuration in output file:0: trainer:0: devices: 10: num_nodes: 10: accelerator: gpu0: precision: 16-mixed0: logger: false0: enable_checkpointing: false0: use_distributed_sampler: false0: max_epochs: 100: max_steps: 1000: log_every_n_steps: 100: val_check_interval: 500: limit_val_batches: 500: limit_test_batches: 5000: accumulate_grad_batches: 10: gradient_clip_val: 1.00: benchmark: false0: min_epochs: 2Trainer configuration in base_config.yaml:trainer:devices: 1 # number of GPUs or CPUsnum_nodes: 1accelerator: gpu#gpuor cpuprecision: 16-mixed # 16-mixed, bf16-mixed or 32logger: False # logger is provided by NeMo exp_managerenable_checkpointing: False # checkpointing is done by NeMo exp_manageruse_distributed_sampler: False # use NeMo Megatron samplersmax_epochs: 10 # # use max_steps instead with NeMo Megatron modelmax_steps: 1000 # consumed_samples = global_step * micro_batch_size * data_parallel_size * accumulate_grad_batcheslog_every_n_steps: 10 # number of interations between loggingval_check_interval: 1500limit_val_batches: 50 # number of batches in validation step, use fraction for fraction of data, 0 to disablelimit_test_batches: 500 # number of batches in test step, use fraction for fraction of data, 0 to disableaccumulate_grad_batches: 1gradient_clip_val: 1.0benchmark: False