TC-MoE: Augmenting Mixture of Experts with Ternary Expert Choice

Model Description

TC-MoE is a novel Mixture-of-Experts (MoE) architecture that enhances traditional MoE models through expert space expansion. By applying the ternary set {-1, 0, 1} to each original expert, TC-MoE achieves:

9% reduction in activated experts compared to Top-K routing
1.1% average performance gain on language understanding benchmarks
Flexible efficiency-effectiveness trade-off via reward mechanism

Key innovations:

🎯 Ternary Expert Expansion: Creates parameter-sharing expert variants (-1, 0, +1) without significant computational overhead
⚖️ Adaptive Load Balancing: Novel load balance loss for expert workload distribution
🎮 Reward-Driven Routing: Dynamic control of expert activation ratios

Model Overview

Architecture: Decoder-only transformer based on LLaMA
Pretraining Data:
- RedPajama (100B tokens)
Model Size:
- Base (681M/2.3B params)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("stiger1000/TC-MoE")
tokenizer = AutoTokenizer.from_pretrained("stiger1000/TC-MoE")

inputs = tokenizer("The capital of France is", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0]))

Training Details

Optimizer: AdamW (β₁=0.9, β₂=0.95)
Learning Rate: 1e-4 with cosine decay
Batch Size: 4M tokens
Loss Components:
- Language Modeling Loss
- Load Balance Loss (α₁=0.01)
- Reward Loss (α₂=0.0)

Citation

@inproceedings{yan2025tcmoe,
  title={TC-MoE: Augmenting Mixture of Experts with Ternary Expert Choice},
  author={Yan, Shen and Bin, Xingyan and Zhang, Sijun and Wang, Yisen and Lin, Zhouchen},
  booktitle={The Thirteenth International Conference on Learning Representations},
  year={2025}
}

📚 Repository: GitHub