merge_method: arcee_fusion ?
Hello!
I just saw you used a new merging method to have this model, what is arcee_fusion ?
I checked the mergekit github page, but I don't see anything about that, I'm curious!
Thank you!
If I remember correctly it's a method to merge model weights by computing dynamic thresholds to identify important elements, then it's selectively merging these elements to create a fused model. I might be wrong though, you should check the source code: https://github.com/arcee-ai/mergekit/blob/main/mergekit/merge_methods/arcee_fusion.py
You can read more about it here - https://www.arcee.ai/blog/meet-mergekit-v0-1-arcee-fusion-expanded-model-support-multi-gpu-acceleration
We're writing a paper on it currently, but the link above from @MaziyarPanahi is accurate as to the implementations used.
Thank you very much to you two!