How do I use Text-Image to Text models with Huggingface Inference?
Hi all, I want to usephi-3.5-text-img-textfor a multi modal task which takes the image of a page and converts the contents of the page into html tags and if there is an image found on the page, it will convert that image into text and add ‘illustration’ tag beside it.How do I use the huggingface inference api forphi3.5instructso that it generates the output.
2024-10-12T13:24:28Z
It’s in the manual, but it’s a newly implemented pipeline, so I don’t know if it really works.huggingface.coWhat is Image-Text-to-Text? - Hugging FaceLearn about Image-Text-to-Text using Machine Learninghuggingface.coImage-text-to-textWe’re on a journey to advance and democratize artificial intelligence through open source and open science.
2024-10-12T15:12:36Z
Hi@perceptron-743the modelmicrosoft/Phi-3.5-vision-instructcontainscustom-code.By default, inference api is disabled for such models, this is done for security reasons since you can’t know what code the model is running in the back.another way to see if the model has an inference api disabled is to go to the model page and check on the rightimage810×391 15.6 KBif the logo has a broken ligthening logo it means the inference api is disabled for that model.To filter the hub by models that have inference api enabled you can checkout the warm and cold ones as shown in the picture belowimage942×577 123 KB
2024-10-12T22:56:08Z
You’re right, but isn’t the server-side behavior a bit buggy?When Inference is turned off, the input field itself on the GUI page doesn’t appear, which is normal behavior for other inferences.Moreover, the author has not explicitly turned off Inference, and he appears to have done the user-enabled settings to turn it on.---
license: mit
license_link: https://huggingface.co/microsoft/Phi-3.5-vision-instruct/resolve/main/LICENSE
language:
- multilingual
pipeline_tag: image-text-to-text
tags:
- nlp
- code
- vision
inference:
  parameters:
    temperature: 0.7
widget:
- messages:
  - role: user
    content: <|image_1|>Can you describe what you see in the image?
library_name: transformers
---