Why does ViTForMaskedImageModeling not construct the original image correctly?
I was trying to use masked image modeling in huggingface and I sawViTForMaskedImageModelingin the documentation but I did not understand how it reconstructs the original imageloss, reconstructed_pixel_values = outputs.loss, outputs.reconstructionalso, it doesn’t reconstruct the original image correctly. It gives me noise.url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

image_processor = AutoImageProcessor.from_pretrained("google/vit-base-patch16-224-in21k")
model = ViTForMaskedImageModeling.from_pretrained("google/vit-base-patch16-224-in21k")

num_patches = (model.config.image_size // model.config.patch_size) ** 2
pixel_values = image_processor(images=image, return_tensors="pt").pixel_values
# create random boolean mask of shape (batch_size, num_patches)
bool_masked_pos = torch.randint(low=0, high=2, size=(1, num_patches)).bool()

outputs = model(pixel_values, bool_masked_pos=bool_masked_pos)
loss, reconstructed_pixel_values = outputs.loss, outputs.reconstruction

reconstructed_pixel_values = reconstructed_pixel_values.detach().numpy()
reconstructed_pixel_values = np.transpose(reconstructed_pixel_values[0], (1, 2, 0))

plt.imshow(reconstructed_pixel_values)
plt.show()Screen Shot 2023-05-17 at 7.31.51 AM900×854 65.5 KB