How to test pretrained model with already separated-from-whole dataset
Dear@John6666I triedthisbut not working with this error---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-38-9b0dbdbf42dd> in <cell line: 0>()
     17 )
     18 
---> 19 trainer_after_train.evaluate()
     20 
     21 

8 frames

/usr/local/lib/python3.11/dist-packages/transformers/data/data_collator.py in torch_default_data_collator(features)
    139         label = first["label"].item() if isinstance(first["label"], torch.Tensor) else first["label"]
    140         dtype = torch.long if isinstance(label, int) else torch.float
--> 141         batch["labels"] = torch.tensor([f["label"] for f in features], dtype=dtype)
    142     elif "label_ids" in first and first["label_ids"] is not None:
    143         if isinstance(first["label_ids"], torch.Tensor):

TypeError: must be real number, not PngImageFileAlready trained with the same shape dataset that I have divided for training, validation, and test. Now I need to test the trained model with the test_dataset.Separation codes:John6666 previous codedataset_repo = "seand0101/segformer-b0-finetuned-ade-512-512-manggarai-watergate"
ds = load_dataset(dataset_repo, split="train")
newdsl = []
for i in range(ds.num_rows):
    newdsl.append({"pixel_values": ds[i]["pixel_values"], "label": im.fromarray(np.array(ds[i]["label"].convert("L")) / 255).convert("L")})
newds = Dataset.from_list(newdsl) # normalized datasetDividing the dataset# 70% train, 30% test + validation

train_testvalid = newds.train_test_split(test_size=0.3)

train_testvalidDividing it for validation# 70% train, 30% test + validation
train_testvalid = newds.train_test_split(test_size=0.3)
train_testvalidNow for the test# Split the 10% test + valid in half test, half valid

test_valid = train_testvalid['test'].train_test_split(test_size=0.3)

test_validResult# gather everyone if you want to have a single DatasetDict
train_test_valid_dataset = DatasetDict({
    'train': train_testvalid['train'],
    'test': test_valid['test'],
    'valid': test_valid['train']})Output from resultimage599×582 14.5 KBCodes to validate with the test part of the dataset that came fromhereand produce such errorhub_model_id_after_train= "seand0101/segformer-b0-finetuned-ade20k-manggarai_rivergate_6"

model_after_train = SegformerForSemanticSegmentation.from_pretrained(
    hub_model_id_after_train,
    num_labels=num_labels,
    id2label=id2label,
    label2id=label2id,
    ignore_mismatched_sizes=True, # Will ensure the segmentation specific components are reinitialized.
)

trainer_after_train = Trainer(
    model=model_after_train,
    args=training_args,
    train_dataset=train_ds,
    eval_dataset=np.array(test_ds),
    compute_metrics=compute_metrics,
)

trainer_after_train.evaluate()Did I evaluate it wrongly?Btw if its not clear, I wanna test the model with part of the dataset so it outputs its own loss plot and accuracy or miou to the test model if that makes anything? or maybe just loss?, is this done correctly?
2025-01-21T06:31:28Z
Hello. If you use the same method you used to pass your dataset to Trainer (specifying a custom collate_fn), it should probably work without any problems.You probably didn’t get an error in metrics and crash during Trainer execution.That’s because DataCollator (collate_fn) was passing the dataset successfully.Well, you could also convert the dataset itself to match the default collate_fn in advance, and you should get the same result, so that’s fine too…Another issue is that whether the evaluation is correct or not ultimately depends on whether the functions related to compute_metrics and Evaluation are suitable for your use case. The computer is just calculating the score according to the instructions.If the scoring method or target is wrong, the wrong score will be given.So, once you have got it working, you need to test it and check whether the numbers are reasonable compared to your understanding.
2025-01-21T07:38:04Z
Oh I forgot to erase the np.array part, it’s one of my experiment which of course didn’t work. Previously without np.array part, both train and validation works for training. Now I want to use the testing third of the dataset.So what happened here---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-29-a577301e3d2f> in <cell line: 0>()
     17 )
     18 
---> 19 trainer_after_train.evaluate()
     20 
     21 

7 frames

/usr/local/lib/python3.11/dist-packages/transformers/data/data_collator.py in torch_default_data_collator(features)
    139         label = first["label"].item() if isinstance(first["label"], torch.Tensor) else first["label"]
    140         dtype = torch.long if isinstance(label, int) else torch.float
--> 141         batch["labels"] = torch.tensor([f["label"] for f in features], dtype=dtype)
    142     elif "label_ids" in first and first["label_ids"] is not None:
    143         if isinstance(first["label_ids"], torch.Tensor):

TypeError: must be real number, not PngImageFile
2025-01-21T07:41:12Z
When I extract a subset of the dataset using a slice other than the built-in one, it seems to become a Python dict type rather than the Dataset class type.Is that the reason?Oh I forgot to erase the np.array partlolEdit:Anyway, the error message is clear, and the process of converting PIL images to PyTorch tensors is missing. It should be the job of collate_fn. It seems that only the data set where the error occurs is not in the format for the default collate_fn.
2025-01-21T07:44:49Z
It’s a dataset data type last time I checked.image898×266 14.1 KBIt’s confusing lolEdit:I don’t remember anything about collate_fn, is it because I usetrain_test_split(test_size)twice?
2025-01-21T07:47:44Z
I just re-read it, and the label_id seems suspicious. Is it mixed datasets up with that?Edit:is it because I usetrain_test_split(test_size)twice?Possible.
2025-01-21T07:51:47Z
Could you elaborate mixed datasets up, like which one.Btw I just wanna plot losses of model to the testing dataset, isn’t it just like inference but with array of images at this point? Should I just loop each and plot the loss one by one?
2025-01-21T08:15:31Z
It’s just this part. The error is occurring because f[“label”] is not a tensor or a number. Sorry, I was wrong about label_id. I think just the structure of the dataset is still wrong.--> 141         batch["labels"] = torch.tensor([f["label"] for f in features], dtype=dtype)

TypeError: must be real number, not PngImageFile
2025-01-21T08:18:34Z
I found same symptom.stackoverflow.comLocal HuggingFace dataset of images and segmentation is crashed when try to load datahuggingface-transformers, huggingface, huggingface-datasetsasked byidooon11:17AM - 10 Oct 23 UTCIt’s quicker to write collate_fn…Edit:I don’t have much experience training models, but I think that the dataset and collate_fn are effectively a set when using Trainer. If either of them changes, it won’t work in the same way.This is because the format of the data that collate_fn passes to Trainer must be a specific format as Trainer expects. I guess it would be easier if you always made a dataset that works with the default collate_fn, but I don’t know how to do that…
2025-01-21T10:38:18Z
I see thanks then, I will look that up link real quick.