Multi-input classification (images + Texts)
Hi folks,I am pretty new here. So here is my problem description: I am trying to classify a sequence of document images. These documents could be between 1 and 10 pages long. I noticed there are some models such as LayoutLM are designed explicitly for document images, however, it seems it can only intake one image at a time. In our setting, we need multiple images since two different documents could contain similar images somewhere in them respectively. I can also use OCR to convert the images to texts with corresponding coordinates.I came from Tensorflow world. In the past, I have been using its functional API to train a single model that can intake multiple inputs. But I am not sure how to do that with HuggingFace. Does anyone have experience in this type of problem?Thank you!
2022-09-02T08:45:02Z
Uhm HuggingFace Transformers simply is a collection of models written with PyTorch, Tensorflow, and Flax. So yeah you just take a model source code you want then plug in whichever mode you want.
2022-09-02T18:42:53Z
Thank you! Is it possible to load the pre-trained weight that way?
2022-09-03T09:47:44Z
Yeah but you might have to mind about keys in the state dict (assuming you use PyTorch)
2022-09-04T06:25:34Z
Actually, I use Tensorflow. Do you have any tutorials I can read about? Sorry I am completely new to Huggingface. Appreciate your help and patience.
2022-09-04T07:15:45Z
Sorry I am not familiar with Tensorflow enough try to read their doc more
2024-02-18T16:20:21Z
Hey@uniflowdo you found a solution yet?