Upload MoLA-LM: Mixture of LoRA Adapters Language Model

README.md

@@ -13,18 +13,13 @@ language:
 pipeline_tag: text-generation
 ---
 
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/630f3e4002ce39336c411048/3gVVmArsXVoogpkXvsBs7.png)
+Image here
 
 # MoLA-LM: Mixture of LoRA Adapters LLM
 
 MoLA-LM combines multiple LoRA adapters with an intelligent router to automatically select the best adapter for each input prompt. This approach enables specialized performance across different tasks while maintaining efficiency.
 
-[**Click for evals**](https://github.com/alkinun/MoLA/blob/main/README.md)
-
-**Important Note**: *The v0.5 had issues with the lora applying part of the custom lm class and its router was a bit too small with little generalization.
-In v0.6 and future models, all of these issues are/will be resolved.*
-
-**TLDR:** *Dont use v0.5, use v0.6 and above.*
+Evals are coming...
 
 ## Model Details
 
@@ -70,7 +65,7 @@ print(response)
 The MoLA-LM architecture consists of:
 
 1. **Base Model**: Qwen/Qwen3-4B-Thinking-2507
-2. **Router Network**: Frozen encoder as Sentence transformer + decoder as MLP for adapter selection  
+2. **Router Network**: Frozen encoder as Sentence transformer + decoder as one layer MLP for adapter selection  
 3. **LoRA Adapters**: 9 task-specific fine-tuned adapters
 4. **Dynamic Switching**: Automatic adapter application based on input
 

model-00002-of-00002.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:bed174ccd40be44260f2d0433dd2e27f54994f06823929385883f0c913f26bfe
+oid sha256:6e2a333e0334f80a76b455abfd4db6b05779f440fc980a4bec61259c2ddb8371
 size 3176026404

modeling_mola_lm.py

@@ -217,12 +217,16 @@ class MoLAForCausalLM(PreTrainedModel, GenerationMixin):
             else:
                 # Hub path - download first adapter
                 try:
-                    # Download first adapter to get local path
-                    adapter_file = hf_hub_download(
+                    # Download both required files for first adapter
+                    adapter_weights_file = hf_hub_download(
                         repo_id=self.model_path, 
                         filename=f"loras/{first_adapter}/adapter_model.safetensors"
                     )
-                    first_lora_path = os.path.dirname(adapter_file)
+                    adapter_config_file = hf_hub_download(
+                        repo_id=self.model_path, 
+                        filename=f"loras/{first_adapter}/adapter_config.json"
+                    )
+                    first_lora_path = os.path.dirname(adapter_weights_file)
                     print(f"Downloaded first adapter to: {first_lora_path}")
                 except Exception as e:
                     raise Exception(f"Failed to download first adapter {first_adapter}: {e}")
@@ -249,11 +253,15 @@ class MoLAForCausalLM(PreTrainedModel, GenerationMixin):
                     else:
                         # Hub path - download adapter
                         try:
-                            adapter_file = hf_hub_download(
+                            adapter_weights_file = hf_hub_download(
                                 repo_id=self.model_path, 
                                 filename=f"loras/{task_name}/adapter_model.safetensors"
                             )
-                            lora_path = os.path.dirname(adapter_file)
+                            adapter_config_file = hf_hub_download(
+                                repo_id=self.model_path, 
+                                filename=f"loras/{task_name}/adapter_config.json"
+                            )
+                            lora_path = os.path.dirname(adapter_weights_file)
                         except Exception as e:
                             print(f"❌ Failed to download LoRA {task_name}: {e}")
                             continue

router_weights.pth

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:95494775139afe1350f5bb1f37f5a283ac47ed9d52baf1764dfff16cd6d561ed
+oid sha256:e912e595c3e4543b8057747d8032d8c220664aef6a28cd34fc538b7ff89c739a
 size 7395773