Custom metrics with extra data?
I have a multi class model that I want to evaluate (during training on the eval set using additional data that is contained in the dataset. The extra data is a column that provides a grouping - a document id.These are options I am considering.Thecompute_metricfunction you can pass into theTraineris only passed in anEvalPredictionwhich  contains the predictions and labels, not the extra data, so this is not possible.I can subclass theTrainerand then override theevaluation_loop. It would be a very much a copy paste effort with overriding the call to compute_metrics to pass in extra data. I don’t know if this would also affect the training time?I can utilise a callback at the end of epoch and calculate the metrics I want outside of the evaluation loop. I’m not sure whether I will have access to all data here though.Any other options that I am missing? What is the best way of accessing extra data in the evaluation metrics?
2022-07-18T15:27:16Z
cc@lvwerrado you have an idea ?
2022-07-18T19:36:18Z
Thanks for calling attention to it, I ended up choosing the last option, implement an end of epoch override. Worked well, unfortunately, it also meant I had to implement saving the best model myself too as it’s the same “metric” being used.
2022-07-19T07:52:12Z
An alternative way might be to either define a global variable or create something like acompute_metricfactory:def compute_metric_with_extra(extra_data):
    def compute_metric(preds, refs):
        # do something with preds, refs and extra_data
        return score
    return compute_metric

compute_metric = compute_metric_with_extra(extra_data)You should be able to pass this function to the trainer. Sorry I was too late
2023-08-25T14:10:44Z
@lvwerraThanks for the workaround. I have a related followup. In my usecase, the metric depends on each individual datapoint (and let’s say each datapoint has a unique “datapoint ID”) and hence, while calculating the metric for a particular result in “preds”, I also need to know which datapoint ID it corresponds to. Hence, in “preds”, I would possibly like to have “input_ids”, “labels”, “predictions” and “datapoint_id”. Is there any way to achieve this ?
2023-10-17T15:34:29Z
@Ivwerra’s suggestion can be made to work the way you want by passing datapoint_id aligned with labels.
2024-03-14T09:29:46Z
Hi@anmolagarwal999@AgnivaI wonder if you could share more details on how you implement this feature? In my case, aligning either predictions or input_ids to “datapoint ID” (with some sort of a dictionary look-up?) would be too much for a large set of long sequences. Alternatively, I am thinking if I can “hack” the EvalPrediction.inputs (i.e. input_ids if i’m not mistaken) or somehow take advantage of the index order. However, I am not confident if either of the two alternatives would work. May I get your opinions on this? Many thanks!
2024-03-14T09:42:52Z
Is there a way to add an external variable but that changes during training (like model for eg.)I want to evaluate a clip model on zero_shot classification, so on labels that are not in EvalPred.label_ids.I can give these labels as extra_data like explained above in the answers, but their embeddings will change as the model is training.Is there a way to do this ?Thanks !
2024-04-12T01:31:09Z
nkw1234:EvalPredictionis the order of EvalPrediction that goes intocompute_metricsthe same as in the evaluation dataset ?