Which loss function in bertforsequenceclassification regression
BertForSequenceClassification can be used for regression when number of classes is set to 1.  The documentation says that BertForSequenceClassification calculates cross-entropy loss for classification.  What kind of loss does it return for regression?(I’ve been assuming it is root mean square error, but I read recently that there are several other possibilities such as Huber or Negative Log Likelihood.)Which is it?How should I find out / where is the code?
2020-10-09T08:28:56Z
This is theGitHublinkAt line 1354, you have the condition to check the labels (if it is one or more)if self.num_labels == 1:#  We are doing regressionloss_fct = MSELoss()loss = loss_fct(logits.view(-1), labels.view(-1))else:loss_fct = CrossEntropyLoss()loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
2020-10-09T09:12:27Z
You can select the lines that you are interested in on a Github code page, then click on the three dots and select copy permalink:github.comhuggingface/transformers/blob/9aeacb58bab321bc21c24bbdf7a24efdccb1d426/src/transformers/modeling_bert.py#L1353-L1360if labels is not None:if self.num_labels == 1:#  We are doing regressionloss_fct = MSELoss()loss = loss_fct(logits.view(-1), labels.view(-1))else:loss_fct = CrossEntropyLoss()loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
2020-10-09T09:50:54Z
Thanks@BramVanroy, this makes the code easier to read. I will do so, in future.
2020-10-09T13:06:51Z
Thank you !
2021-02-24T10:49:36Z
Can you add parameters to the loss function through transformer? for example, add weights to each of the classes?
2021-02-24T19:25:27Z
Hi theudster,the pytorch docs for CrossEntropyLoss suggest that you can add a weight tensorCrossEntropyLoss — PyTorch 1.7.1 documentation.   What happens if you try it?
2021-02-25T13:25:31Z
I haven’t tried that because I am trying to implement everything through the Trainer method