TransformerXL on Custom Language
I am looking to train a Transformer model on biological sequence data where a “sentence” may be represented as follows: [“A”, “LLGR”, “V”, “GD” …]In order to do so, I need a word-level tokenizer that doesn’t split up words, so I’ve opted for TransformerXL.Thetrain()function fromByteLevelBPETokenizer()(as described in this blog:https://huggingface.co/blog/how-to-train) is not available forTransfoXLTokenizer()Is there a way to trainTransfoXLTokenizer()on my custom “language”? Or do I simply train theTransfoXLModeland it’ll take care of custom tokenization? (I can’t see this being the case as its a completely unseen “language”)
2020-10-21T14:58:26Z
No need to train.TransfoXLTokenizer()doesn’t merge words likeByteLevelBPETokenizer().Simply create your vocab file as a txt file with each word on a new line, and feed it’s path as a parameter toTransfoXLTokenizer():tokenizer = TransfoXLTokenizer(vocab_file=vocab_path)