bf16=True in TrainingArgument
Is the following code in the official Qlora repogithub.comartidoro/qlora/blob/3da535abdfaa29a2d0757eab0971664ed2cd97e8/qlora.py#L334-L343for name, module in model.named_modules():if isinstance(module, LoraLayer):if args.bf16:module = module.to(torch.bfloat16)if 'norm' in name:module = module.to(torch.float32)if 'lm_head' in name or 'embed_tokens' in name:if hasattr(module, 'weight'):if args.bf16 and module.weight.dtype == torch.float32:module = module.to(torch.bfloat16)similar to passingbf16=Trueintotransformers.TrainingArgumentwhen usingtransformers.TrainingArgumentwithtransformers.Trainer?