Popping `inputs[labels]` when self.label_smoother is not None (in trainer.py)
Hi,I was training my seq2seq model (I’m usingSeq2seqTrainer) with label-smoothing and have encountered an error thatinput_idswas required in my training dataset, whereas I checked that I put them in the dataset.While debugging it, I found that whenself.label_smootheris notNone,  thelabelsitem was popped out frominputsand the error came fromoutputs = model(**inputs)as shown in the following lines intrainer.py:1872     def compute_loss(self, model, inputs, return_outputs=False):
1873         """
1874         How the loss is computed by Trainer. By default, all models return the loss in the first element.
1875 
1876         Subclass and override for custom behavior.
1877         """
1878         if self.label_smoother is not None and "labels" in inputs:
1879             labels = inputs.pop("labels")
1880         else:
1881             labels = None
1882         outputs = model(**inputs)Question: is the line number 1879 intended? I think it would be eitherlabels = copy.deepcopy(inputs['labels'])orlabels = inputs['labels']I searched for this board but couldn’t find any similar post. That means other people are using the label-smoothing without any problem, which means I incorrectly understand the concept of the seq2seq training and label-smoothing.Any comment would be greatly appreciated.
2021-11-11T10:54:21Z
Hey@jbehcan you share a minimal reproducible example? For example, something simple that just shows:How you load and tokenize the datasetsHow you define the training argumentsHow you define the trainerThat will help us understand better what is causing the issue
2021-11-11T13:26:10Z
The labels are popped because otherwise your model computes the losses twice, so two SoftMaxes, which is a very heavy operation. You need to pass along thedecoder_input_idswhen you want to use label smoothing with theTrainer, as generated byDataCollatorForSeq2Seq.