DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion
  Deep network architectures struggle to continually learn new tasks without
forgetting the previous tasks. A recent trend indicates that dynamic
architectures based on an expansion of the parameters can reduce catastrophic
forgetting efficiently in continual learning. However, existing approaches
often require a task identifier at test-time, need complex tuning to balance
the growing number of parameters, and barely share any information across
tasks. As a result, they struggle to scale to a large number of tasks without
significant overhead. In this paper, we propose a transformer architecture
based on a dedicated encoder/decoder framework. Critically, the encoder and
decoder are shared among all tasks. Through a dynamic expansion of special
tokens, we specialize each forward of our decoder network on a task
distribution. Our strategy scales to a large number of tasks while having
negligible memory and time overheads due to strict control of the parameters
expansion. Moreover, this efficient strategy doesn't need any hyperparameter
tuning to control the network's expansion. Our model reaches excellent results
on CIFAR100 and state-of-the-art performances on the large-scale ImageNet100
and ImageNet1000 while having less parameters than concurrent dynamic
frameworks.
