How to Log Accuracy with Metadata in a Sentence Regression Task?
I’m working on asentence regression task, where each sample consists of asentence paired with a numerical scalar. However,each sample also includes metadata(e.g., project name, task type). I want to compute and log theaccuracy with respect to this metadataduring evaluation.What I’ve TriedSo far, I’ve successfully:Modified the dataset classto return adata_dictcontaining:input_idslabels(for loss calculation, just like in a language modeling task)numerical_score(the scalar target for regression)Metadata(a string field likeProjectorTask)Updated the collate functionto pass along all these elements correctly to the model’sforwardfunction.confirmed that this is passed along the model outputs as an additional metadata attribute.Problem: Metadata Gets Lost  before reachingcompute_metricsThe issue arises during the evaluation_loop. I need access tometadatainside thecompute_metricsfunction, but the Trainer class doesn’t seem to provide a clean way to pass it along.for exampleif args.include_inputs_for_metrics:
    metrics = self.compute_metrics(
        EvalPrediction(predictions=all_preds, label_ids=all_labels, inputs=all_inputs)
    )
else:
    metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))I enabledinclude_inputs_for_metrics=Truehoping themetadata would be passed as part ofall_inputs, but itgets strippedby this line in the evaluation loop:inputs_decode = self._prepare_input(inputs[main_input_name]) if args.include_inputs_for_metrics else NoneWhat I Want to AchieveI need a way tolog the accuracy or any custom metric with respect to the metadata. Ideally, I don’t want tooverride the entire evaluation loopor write my own Trainer class, as that feels cumbersome and difficult to maintain, especially with parallelism.What I’m Looking ForIs there aclean way to pass metadatathrough the evaluation loop, without hacking the Trainer class or completely rewriting the evaluation logic? I’m likely not the first person facing this issue, and I suspect there’s an elegant solution I might be missing.Thanks in advance for your help!