LayoutLMv3 missing visual tokenizer?
Hi,How come LayoutLMv3 does not have a visual tokenizer in its code? The image goes straight from pixel values to embeddings.
2023-01-02T11:34:00Z
Hi,The “tokenization” happens inside the model, using a convolutional 2D layer. One just needs to providepixel_valuesto the model, which will be turned into embedded patches.
2023-01-02T13:07:19Z
Hey@nielsr, first, thanks for the reply.So you’re saying that we do not need discrete integers to serve as ground-truth labels? Reading the paper the authors seem to use a VAE trained on document data. How come HF decided not to do the same? What’s the theoretical baggage behind it?Any help is appreciatedEDIT:If possible can you fundament why text needs to be tokenized and discrete while images do not?
2023-01-02T13:32:22Z
Some relevant literature for the discussion:FromBEiT v2: Masked Image Modeling with Vector-Quantized Visual TokenizersExisting MIM approaches can be coarsely categorized to three according to the reconstruction targets:low-level image elements (e.g., raw pixels; He et al. 2022; Fang et al. 2022; Liu et al. 2022), hand-crafted features (e.g., HOG features; Wei et al. 2021), and visual tokens; Bao et al. 2022; Wang et al.2022; Dong et al. 2021; El-Nouby et al. 2021; Chen et al. 2022. However, all the reconstructiontargets are about, explicitly or implicitly, low-level image elements while underestimating high-levelsemantics. In comparison, the masked words in language modeling (Devlin et al., 2019) are all abouthigh-level semantics, which motivates us to tap the potential of MIM by exploiting semantic-awaresupervision during pretraining.I’d still love to know the reasoning behind HF departure from the OG paper.
2023-01-03T13:37:08Z
Hi,We haven’t deviated from the original paper, Microsoft just didn’t open-source this visual tokenizer used for pre-training (the masked image modeling objective).Note that after pre-training, the masked image modeling head is thrown away, and replaced by a classification head for downstream tasks.
2023-01-03T15:02:29Z
Oh, I see. For my purposes, I’ve been using a random VAE from DALL-E to learn the inner workings of pre-training a model. Can you shed some light on the effects of not using a visual tokenizer?Also, thanks btw. I’ve been using a lot of your stuff to wrap my head around LayoutLMv3.
2023-01-04T17:09:07Z
mvalente:Oh, I see. For my purposes, I’ve been using a random VAE from DALL-E to learn the inner workings of pre-training a model. Can you shed some light on the effects of not using a visual tokenizer?Pre-training using a masked image modeling (MIM) objective where the model needs to predict tokens of a VAE for masked patches is pretty powerful, since the VAE contains useful information in its codebook.BEiTwas the first work to do so, and got state-of-the-art performance on ImageNet after fine-tuning. This was then improved inBEiTv2.LayoutLMv3 adopts the same pre-training objective (among other objectives), but with a VAE especially trained to reconstruct document images.Other models, likeMAEandSimMIMalso do masked image modeling, but instead of predicting tokens of a VAE for masked patches, they directly predict the raw pixel values. This has the benefit of not requiring a separate VAE.BEiTv2 seem to be better than MAE at the moment (as shown in the figure below).Screenshot 2023-01-04 at 18.07.441920×964 110 KB
2023-01-04T17:22:27Z
Thank you for contextualizing the answer.