How to optimize performance of CLIP when searching 10_000 images
I am using CLIP to predict from 1000 images based on 1 sentence.text="some sentence"
images=[] # a list of images
CHECKPOINT = "openai/clip-vit-base-patch32"
processor = CLIPProcessor.from_pretrained(CHECKPOINT)
model = CLIPModel.from_pretrained(CHECKPOINT).to("cuda")
inputs = processor(text=text, images=images, return_tensors="pt", padding=True, truncation=True)
outputs = model(**inputs.to("cuda"))What if I want to scale to 10_000 images? I can vertically scale, but can I do it more cost-effective?Now I load all the images into the processor for each sentence, can I do this more efficiently?
2022-10-14T14:38:48Z
Do you mean you want to build a retrieval system, where you want to find the image that matches best with the text?It’s probably beneficial to embed all your images of your database beforehand using CLIP’s image encoder.Then, you can use a library likeFaissto efficiently retrieve the image embedding that is closest to the text embedding.
2022-10-15T05:28:09Z
Thanks for the fast response!Yes, I am currently using theCLIPProcessor, an easy-to-use abstraction layer.I’ll have a look at what it’s doing under the hood …Thanks for Faiss. I should have a look at that!Already found some interesting stuff:GitHubGitHub - rom1504/clip-retrieval: Easily compute clip embeddings and build a...Easily compute clip embeddings and build a clip retrieval system with them - GitHub - rom1504/clip-retrieval: Easily compute clip embeddings and build a clip retrieval system with themMedium – 9 Apr 22Building Image search with OpenAI ClipOpenAI’s Clip is a neural network that was trained on a huge number of image and text pairs and has therefore learned the “connection”…Reading time: 5 min readMedium – 31 Mar 22Semantic search at billions scaleHow to build Semantic search distributed systems using python, pyspark, faiss and clip! A walk through on building a laion5B semantic…Reading time: 7 min read