Data shape needed for training TransformerXL from scratch
Hello everyone,I’m having trouble getting my data into the proper shape to train a TransformerXL model from scratch. I have a custom Pytorch Dataset that returns a dict from__getitem__with the keysinput_ids, andlabelsboth assigned the same 1-D Tensor of ids for a particular sequence. When I pass this Dataset to the Trainer object with the default_collator I get agrad can be implicitly created only for scalar outputserror. What am I missing here? Am I missing a needed field for theTransfoXLLMHeadModel'sforwardmethod? I’ve tried just about everything and cannot figure it out.
2021-01-11T06:45:53Z
Hi@jodiakWould be hard to answer this without looking at the code, could you post a small code snippet ?AlsoTransformerXLis a language model so the required inputs areinput_idsandlabels, which are of shape[batch_size, seq_len].
2021-01-12T05:07:15Z
Hello@valhallaThanks for the reply. After wrestling with this for a little longer I realized thatTransfoXLLMHeadModel'sforwardmethod isn’t compatible with theTrainerclass because it returns a 1D Tensor oflossesrather than a scalarloss. This is confirmed inthisissue I created on Github. I’m going to try and create a customTrainerto reduce this loss and post my results here.