Way to train a basic Transformer
Is there any way to train a basic Transformer (the transformer architecture presented in the original Attention Is All You Need paper). I’m trying to set up a translation baseline on my custom dataset (previously trained models do not exist for the language). If anyone can point out a ready-made implementation of just the basic Transformer it would be very helpful.Thanks!
2020-11-20T12:04:43Z
I think OpenNMT is the better framework to do something like that. It allows to train a model that you describe easily.transformersis more directed towards specialized, specific transformer architectures.
2020-11-20T14:59:18Z
Thanks! I usedfairseqa few times. I think OpenNMT is kinda similar, they seem more black box like not full control on the entire pipeline. That’s why I wanted to know if such a thing is possible with transformers because this is native Python and not command-line tools.
2020-11-20T22:36:29Z
Have you seen this:https://pytorch.org/tutorials/beginner/transformer_tutorial.html?
2020-11-21T03:06:22Z
And this is TF2-Keras equivalent to@abhishek’s recommendationTensorFlowNeural machine translation with a Transformer and Keras  |  Text...
2020-11-21T06:20:43Z
Thank you! I implemented mine from scratch in Pytorch and Lightning with the help ofbentrevett’s notebookandsource code of BARTin the transformers library.
2020-11-21T06:21:16Z
Thank you for the resources.