Proper way to swap backbones from baseline model
Hi, I’m currently training a segmentation model, and this is how I initiate my model and change my backbone before the training:if config[“model_architecture”].lower() == “upernet”:model_args[“loss_ignore_index”] = model_args.pop(“semantic_loss_ignore_index”)del model_args[“image_size”]MODEL_INITIALIZER = UperNetForSemanticSegmentationLoad the new backbone from the checkpointif config[“backbone_checkpoint_model”] is not None:new_backbone = AutoBackbone.from_pretrained(new_backbone_dir, out_indices=(1,2,3,4))# Replace the existing backbone with the new onemodel.backbone = new_backbone# Optionally, if you need to adjust the configurationmodel.config.backbone_config = new_backbone.configFor the other backbones (convnext, convnextv2) this approach seems to work without any issues, but for swinv2, even though the training runs without any errors (I guess this means at least the dimensions match correctly) swinv2 model does not seem to learn under this setting.Is this the way how the backbones are usually swapped? or am I doing it wrong?Any sort of input would be appreciated! Thanks!