Image Features as Model Input
Hello All. I apologize for the basic question but for some reason I am having difficulty using image features as input to a huggingface model.My data comes in the form of an image feature numpy array extracted by a 2D CNN but all the models seem to be built for text based input.If anybody could point me in the right direction of an example or a code snippet I would greatly appreciate it!
2020-11-18T12:43:05Z
Hi CMPE-PL,the transformers models are designed for Natural Language Processing, ie text.   What makes you think they would be good for image features?I expect you could bypass the tokenizers and input numbers directly, but I’m not sure it would do anything useful.   If you did want to do that, you would need to ensure that your numbers were in the right format.  For example, a BERT model expects input that is a vector of 768 real numbers for each word-token, or rather a matrix of 768 x N real numbers, where N is number of word-tokens in a text.What size is your image feature array?The main trick of transformers is to use Attention mechanisms.  It is certainly possible to use Attention in image recognition models, but without using transformers. See this article for an examplehttps://arxiv.org/abs/2004.13621
2020-11-18T12:51:36Z
If you haven’t seen them already, you might find these articles useful.arXiv.orgAttention Is All You NeedThe dominant sequence transduction models are based on complex recurrent or
convolutional neural networks in an encoder-decoder configuration. The best
performing models also connect the encoder and decoder through an attention
mechanism. We propose...jalammar.github.ioThe Illustrated TransformerDiscussions:
Hacker News (65 points, 4 comments), Reddit r/MachineLearning (29 points, 3 comments)


Translations: Chinese (Simplified), French, Japanese, Korean, Russian, Spanish

Watch: MIT’s Deep Learning State of the Art lecture referencing this...