Class weights for bertForSequenceClassification
I have an unbalanced data with a couple of classes with relatively smaller sample sizes. I am wondering if there is a way to assign the class weights to BertFor SequenceClassification class, maybe in BertConfig ?, as we can do in nn.CrossEntropyLoss.Thank you, in advance!
2020-10-21T14:34:12Z
No, you need to compute the loss outside of the model for this. If you’re usingTrainer, seehereon how to change the loss form the default computed by the model.
2020-10-21T15:00:31Z
Hi Sylvain,Glad to hear from you, outside of FastAIWell, I am here as “Beginner” and will have to study more about Trainer. In the meantime, I tried the following:run BertForSequenceClassification as usualTake out logits from output (discard the loss from Bert run)calculate new loss from nn.CrossEntropyLossand then calculate loss.backward()Model runs okay, but I am not sure if this is a legitimate approach…Thanks.
2020-10-21T15:03:25Z
That is the correct approach!
2020-10-21T15:22:06Z
Fantastic!!! This means a lot that I got support from somebody like you, Sylvain! Thanks!!
2021-08-26T15:25:08Z
How do I take out the logits and calculate the new loss? Do you have a code example/snippet?
2022-01-24T04:02:17Z
Hi there,I’m assuming you’re using the standardBertForSequenceClassification. So, instead of doingoutputs = model(**inputs)
loss = outputs['loss']you dooutputs = model(**inputs)
logits = outputs['logits']
criterion = torch.nn.CrossEntropyLoss(weights=class_weights)
loss = criterion(logits, inputs['labels'])assuminginputsis the dictionary that feeds the model. The loss function will leverage the class weights. Documentation forCrossEntropyLosscan be foundhere.If, instead, you’re usingTrainer, you’ll have to change thecompute_lossmethod as informed previously:Class weights for bertForSequenceClassificationBeginnersNo, you need to compute the loss outside of the model for this. If you’re using Trainer, seehereon how to change the loss form the default computed by the model.As an example, modifying theoriginal implementation, it’d be something likefrom transformers import Trainer
import torch

class MyTrainer(Trainer):
    def __init__(self, class_weights, *args, **kwargs):
        super().__init__(*args, **kwargs)
        # You pass the class weights when instantiating the Trainer
        self.class_weights = class_weights

    def compute_loss(self, model, inputs, return_outputs=False):
        """
        How the loss is computed by Trainer. By default, all models return the loss in the first element.
        Subclass and override for custom behavior.
        """
        if self.label_smoother is not None and "labels" in inputs:
            labels = inputs.pop("labels")
        else:
            labels = None
        outputs = model(**inputs)
        # Save past state if it exists
        # TODO: this needs to be fixed and made cleaner later.
        if self.args.past_index >= 0:
            self._past = outputs[self.args.past_index]

        if labels is not None:
            loss = self.label_smoother(outputs, labels)
        else:
            # We don't use .loss here since the model may return tuples instead of ModelOutput.

            # Changes start here
            # loss = outputs["loss"] if isinstance(outputs, dict) else outputs[0]
            logits = outputs['logits']
            criterion = torch.nn.CrossEntropyLoss(weights=self.class_weights)
            loss = criterion(logits, inputs['labels'])
            # Changes end here

        return (loss, outputs) if return_outputs else lossif you’re not using label smoothing.
2022-05-24T19:40:03Z
@lucasresckas per your suggestion, I did the followingtrain_losses = []
num_mb_train = len(train_dataloader)

import torch.nn as nn
import numpy as np




total_preds=[]

for epoch in range(num_epochs):
  train_loss = 0


  
  for step, batch in enumerate(train_dataloader):
    
    model.train()
    
    batch = tuple(t.to(device) for t in batch)
    b_input_ids, b_input_mask,b_token_type, b_labels = batch

    outputs = model(b_input_ids, token_type_ids=b_token_type, attention_mask=b_input_mask, labels=b_labels)
    
    criterion = torch.nn.CrossEntropyLoss(weight=weights,reduction='mean')
    loss = criterion(outputs[1], b_labels)
    
    loss.backward()
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    optimizer.step()
    scheduler.step()
        
    train_loss = loss.item()
    optimizer.zero_grad()
    
    train_losses.append(train_loss)

    if (step) % 50 == 0:
      print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}' 
             .format(epoch+1, num_epochs, step+1, total_steps, loss.item()))Am I doing the right way or not?
2022-05-28T22:31:41Z
lucasresck:torch.nn.CrossEntropyLossheads up, the param fortorch.nn.CrossEntropyLossisweight(notweights)
2022-05-29T00:42:39Z
Supposingoutputs[1]refers tooutputs['logits'], I believe it’s correct.
2022-05-29T00:45:47Z
Thanks for the correction. I would edit the answer if possible.