Deepspeed with Trainer: No. of trainable parameters coming to be 0
With “codegen-2B-multi” model usingDeepspeedandgradient checkpointing,Trainerloop gives no. of trainable parameters as0.  This is where the logging happens:github.comhuggingface/transformers/blob/94b3f544a1f5e04b78d87a2ae32a7ac252e22e31/src/transformers/trainer.py#L1616# Train!logger.info("***** Running training *****")logger.info(f"  Num examples = {num_examples}")logger.info(f"  Num Epochs = {num_train_epochs}")logger.info(f"  Instantaneous batch size per device = {args.per_device_train_batch_size}")logger.info(f"  Total train batch size (w. parallel, distributed & accumulation) = {total_train_batch_size}")logger.info(f"  Gradient Accumulation steps = {args.gradient_accumulation_steps}")logger.info(f"  Total optimization steps = {max_steps}")logger.info(f"  Number of trainable parameters = {sum(p.numel() for p in model.parameters() if p.requires_grad)}")self.state.epoch = 0start_time = time.time()epochs_trained = 0steps_trained_in_current_epoch = 0steps_trained_progress_bar = None# Check if continuing training from a checkpointif resume_from_checkpoint is not None and os.path.isfile(When I am just computing the same without using the Trainer loop (w/o Deepspeed etc.), it is accurately outputting 2B parameters. I am unable to debug this issue, can anyone help in this regard?
2022-12-22T14:54:56Z
I meet the same problem! Could I ask whether you have solved it?
2023-03-22T07:42:19Z
logger.info(f"  Number of trainable parameters = {sum(p.numel() + p.ds_numel for p in model.parameters() if p.requires_grad)}")
2023-11-29T03:39:23Z
but I got an error p does not have attribute ds_numel.
2024-11-21T01:54:04Z
same problem. is there any solution？