Inaccurate bboxes after finetuning DETR
I followed theObject Detectionguide to fine-tune a DETR model. However, the predicted bboxes for objects in the upper left corner in an image tend to be more accurate than the bottom right corner (the further away from 0,0, the more inaccurate it gets). I’m not sure what could be wrong – whether it’s the transformations, the visualization code, or something else. The original bounding boxes for the dataset seem correct. I’d appreciate any pointers, thanks!Notebook link:Google ColabExample training image:example-chickens1574×885 107 KBImage that demonstrates the bbox issue:inferred-chickens1574×885 13.8 KB
2024-10-02T02:16:58Z
It doesn’t look like a bug in the generative AI, or a normal bug…It looks like the kind of bug that happens when the calculation is simply not precise enough, but in this case, isn’tround()suspicious?for score, label, box in zip(results["scores"], results["labels"], results["boxes"]):
      box = [round(i, 2) for i in box.tolist()]
      x, y, x2, y2 = tuple(box)
      draw.rectangle((x, y, x2, y2), outline="red", width=1)
      draw.text((x, y), model.config.id2label[label.item()], fill="white")stackoverflow.comPython 3.x rounding behaviorpython, python-3.x, roundingasked byLevonon12:11AM - 31 May 12 UTC
2024-10-02T03:33:03Z
Thanks for the suggestion. I also thought it looked like a rounding error, but visually, it still looks the same after increasing the precision. If the rounding was done to normalized coordinates, then it would be more likely the cause, but I can’t see  where that could be happening.
2024-10-02T05:47:52Z
Detected chicken with confidence 0.999 at location [906.24420166, 463.305633545, 952.193481445, 512.982299805]Hmmm, for example, if these numbers are wrong, then you must have missed something in your training.They’re resized, and there’s a good chance that information about the coordinates is lost.If this number is right, then the visualization program is buggy.In any case, it is not such a serious error, and it is probably just a multiplication or rounding error at some point, whether in training or visualization. I guess your program is not doing everything wrong, as it is not backwards or disjointed and it is only a trend.Or, have your AI become anAI that recognizes chicken buttsbecause only chicken butts are educated during training?
2024-10-03T19:13:08Z
Ok, switching from Albumentations toDetrImageProcessorto resize the images fixes it. I am still confused by the following:If I follow the Object Detection guide and setmax_heightandmax_widthtoMAX_SIZE, I still get inaccurate coordinates. It only works when I setheightandwidthtoMAX_SIZE, disregarding the aspect ratio. Does this suggest that there’s a bug in the padding or collate function?Since I’m settingheightandwidth, it seems identical to what I was doing with Albumentations. Why do they yield different results?cc@nielsr@qubvel-hfdo you have any insights?
2024-10-03T19:33:08Z
Hi, it’s indeed looks like some bug and maybe related to wrong box resize to the image coordinates. You can take inputs.pixel_values and it’s shape as a target size and try to visualize boxes on it
2024-10-05T07:09:36Z
Not sure if related - but in the guide’saugment_and_transform_batchfunction, there’s this snippet – why do we want to remove the mask? I tried removing this code, thinking that it could have caused the model to think the chickens are only in the top left section of the image (since the bottom / right sections are always padded), but it actually made things worse, now the coordinates are even more squished.if not return_pixel_mask:
        result.pop("pixel_mask", None)Squished boxes:image1574×885 112 KBI updated the notebook with more visualization, visualizing what a collated batch with a mask looks like. It looks correct, and I can’t find an issue with the inference visualization code either. Any pointers?
2024-10-05T08:08:45Z
Those two lines are the body of the mask disable option, so don’t delete them.I think the resizing is when the chicken is passed to the preprocessor.Here’s an examplehuggingface.coPreprocessWe’re on a journey to advance and democratize artificial intelligence through open source and open science.huggingface.coUtilities for Image ProcessorsWe’re on a journey to advance and democratize artificial intelligence through open source and open science.There is a way to do it manually with torch, but if you can do it with transformers, I recommend letting the transformers do it because it is easier, especially during batch processing.Sparrow Computing – 20 Mar 21TorchVision Transforms: Image Preprocessing in PyTorchThis post explains the torchvision.transforms module by describing the API and showing you how to create custom image transforms.Est. reading time: 6 minutesMedium – 23 Oct 23The Complete Guide to Image Preprocessing Techniques in PythonHave you ever struggled with poor quality images in your machine learning or computer vision projects? Images are the lifeblood of many Al…Reading time: 11 min read
2024-10-05T21:34:28Z
Why might we want to disable the mask?Detr’s documentation:Due to this resizing, images in a batch can have different sizes. DETR solves this by padding images up to the largest size in a batch, and by creating a pixel mask that indicates which pixels are real/which are padding.Anyhow, the preprocessed images look correct, at least thesizes and the bounding boxes do.Preprocessed batch without mask:Preprocessed batch with mask:Image processor:from transformers import AutoImageProcessor
import albumentations
import numpy as np
import torch

IMAGE_SIZE = 480
MAX_SIZE = IMAGE_SIZE
checkpoint = "facebook/detr-resnet-50"
image_processor = AutoImageProcessor.from_pretrained(
    checkpoint,
    do_resize=True,
    size={"max_height": MAX_SIZE, "max_width": MAX_SIZE},
    do_pad=True,
    pad_size={"height": MAX_SIZE, "width": MAX_SIZE},
)

train_transform = albumentations.Compose(
    [
        albumentations.NoOp(),
    ],
    bbox_params=albumentations.BboxParams(format="coco", label_fields=["category"]),
)Visualization of a preprocessed batch:import numpy as np
import cv2

def pixel_values_to_img(pixel_values):
  npimg = pixel_values.numpy()
  npimg = np.transpose(npimg, (1,2,0))
  npimg = (npimg * image_processor.image_std + image_processor.image_mean) * 255
  npimg = (npimg).astype(np.uint8)
  return npimg

def pixel_mask_to_img(pixel_mask):
  npimg = pixel_mask.numpy()
  npimg = npimg[0]
  rgb = np.zeros((npimg.shape[0], npimg.shape[1], 3), dtype=np.uint8)
  rgb[:,:,0] = npimg * 255
  rgb[:,:,1] = npimg * 255
  rgb[:,:,2] = npimg * 255
  return rgb

# translate the preprocessed image batch into a PIL image
transformed_img = collate_fn([train_dataset_transformed[0]])
pixel_values = transformed_img['pixel_values']
if len(pixel_values.shape) > 3:
  pixel_values = pixel_values[0]
img=pixel_values_to_img(pixel_values)
img_w, img_h = img.shape[0], img.shape[1]

# if there's a mask, add a magenta tint to the valid, non-padded pixels
if 'pixel_mask' in transformed_img:
  mask=pixel_mask_to_img(transformed_img['pixel_mask'])
  masked_image = img.copy()
  masked_image = np.where(mask.astype(int),
                          np.array([255,0,255], dtype='uint8'),
                          masked_image)

  masked_image = masked_image.astype(np.uint8)
  masked_image = cv2.addWeighted(img, 0.6, masked_image, 0.4, 0)
  img = masked_image

img = Image.fromarray(img)

# draw bboxes
draw = ImageDraw.Draw(img)
for box in transformed_img['labels'][0]['boxes']:
  cx,cy,w,h=img_w*box[0],img_h*box[1],img_w*box[2],img_h*box[3]
  x,y=cx-w/2,cy-h/2
  x2,y2=x+w,y+h
  draw.rectangle((x,y,x2,y2), outline="white", width=1)

img
2024-10-06T05:29:53Z
If you’re stewing over a bug that can’t be fixed, you can try itwith different data that seems to work. A square one would be even better.Once we know there is a correct answer, we can then track down the bug by process of elimination. If there is no correct answer, it is a bug in the library, but that is not likely to happen this time.Anyway, if you want to disable the mask, you should change it here, not in the function. Whether you do or not is beside the point.transform_batch = partial(
    #augment_and_transform_batch, transform=train_transform, image_processor=image_processor, return_pixel_mask=True
    augment_and_transform_batch, transform=train_transform, image_processor=image_processor, return_pixel_mask=False
)Does this suggest that there’s a bug in the padding or collate function?I think the behavior of the resizing process is stubborn.github.com/huggingface/transformersUnable to disable the `do_resize` option in the CLIPImageProcessoropened08:11AM - 14 Feb 23 UTCclosed12:33PM - 15 Feb 23 UTCadhakal224### System Info

```
- `transformers` version: 4.26.1
- Platform: Linux-3.10….0-1160.53.1.el7.x86_64-x86_64-with-glibc2.31
- Python version: 3.10.8
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
```

### Who can help?
@amyeroberts @NielsRogge @arthur


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I have a situation where I convert my inputs to tensor and resize them before passing them to CLIPImageProcessor. Hence, I want to disable this resize operation inside the CLIPImageProcessor. However, when i pass `False` to the `do_resize` flag, the tensors that it returns are still resized to the default 224x224 size. Here is a reproducible code:

```
from transformers import CLIPImageProcessor
batched_tensor = tensor.rand((2,3,512,512))
processor = CLIPImageProcessor.from_pretrained("openai/clip-vit-base-patch32")
processed_image = processor(list(batched_tensor), return_tensors='pt', padding=True, do_resize=False)

print(processed_image['pixel_values'].shape)

Output:
torch.Size([2, 3, 224, 224])
```
I need to use a `DataLoader` which does not accept variable sizes input (which is the case with my data) and hence I need to resize them before sending it to the CLIPImageProcessor.  
### Expected behavior

I would expect the output of the last line to be `torch.Size([2,3,512,512])`There are various functions for various models in transformers, but most of them are inherited in the class inheritance, so in essence, they are reusable.If a problem occurs in one model, it can occur in any model. If it is a bug, fixing it will fix everything.