Collections

1

BlackMamba: Mixture of Experts for State-Space Models

Paper • 2402.01771 • Published Feb 1, 2024 • 24
OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

Paper • 2402.01739 • Published Jan 29, 2024 • 27
MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

Paper • 2401.15947 • Published Jan 29, 2024 • 51
DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Paper • 2401.06066 • Published Jan 11, 2024 • 49

2

Simple linear attention language models balance the recall-throughput tradeoff

Paper • 2402.18668 • Published Feb 28, 2024 • 19
Linear Transformers with Learnable Kernel Functions are Better In-Context Models

Paper • 2402.10644 • Published Feb 16, 2024 • 80
Repeat After Me: Transformers are Better than State Space Models at Copying

Paper • 2402.01032 • Published Feb 1, 2024 • 24
Zoology: Measuring and Improving Recall in Efficient Language Models

Paper • 2312.04927 • Published Dec 8, 2023 • 2

BlackMamba: Mixture of Experts for State-Space Models

OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Simple linear attention language models balance the recall-throughput tradeoff

Linear Transformers with Learnable Kernel Functions are Better In-Context Models

Repeat After Me: Transformers are Better than State Space Models at Copying

Zoology: Measuring and Improving Recall in Efficient Language Models

BlackMamba: Mixture of Experts for State-Space Models

OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts

SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Self-Rewarding Language Models

ReFT: Reasoning with Reinforced Fine-Tuning

Tuning Language Models by Proxy

TrustLLM: Trustworthiness in Large Language Models

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Sparse Networks from Scratch: Faster Training without Losing Performance

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

A Mixture of h-1 Heads is Better than h Heads

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Mixtral of Experts

Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM

LLM Augmented LLMs: Expanding Capabilities through Composition

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Sparse Networks from Scratch: Faster Training without Losing Performance

ZeRO: Memory Optimizations Toward Training Trillion Parameter Models

A Mixture of h-1 Heads is Better than h Heads

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding

OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

BlackMamba: Mixture of Experts for State-Space Models