Paid API Service
Is it possible to run a paid inference API at huggingface for large LLMs? Like I want to evaluate a few custom datasets on BLOOM/OPT models. I understand that it’s not possible for huggingface to provide thousands of free API calls. But can I just pay for it and get inference output on publicly available models at huggingface?
2022-12-30T20:33:31Z
cc@jeffboudier
2022-12-30T20:57:30Z
Thanks for asking! To evaluate models beyond the free tier of the Inference API, you can use our paid inference solutionInference Endpointsand deploy any model on dedicated infrastructure for your use (billed by capacity, not by requests). Note that for >10Bn models the available hardware instances may not fit the model (e.g. BLOOM 176Bn, OPT 175Bn), which may require you to request quota for large instances, or a custom quote / deployment.
2022-12-31T02:22:49Z
Thanks a lot for the fast reply.It seems to me that right now it’s not possible to deploy the 176B model according to this page. Or is there any other way we can do that?huggingface.coHugging Face – PricingAnnouncing our new Plans for community and organizationsI can already see that the BLOOM is deployed at azure,bigscience/bloom · Hugging Face.Is it on CPU?Right now as a researcher, I only need inference results for a few datasets which may not require more than a couple of hours each. Is it ok to request a custom quote/deployment for that?Note that I want to host “google/flan-t5-xxl” (11B), “bigscience/bloomz” (176B) or OPT (175B) (depending on perf.). But not sure which device to use since A100 nodes are not available and judgding by thep4de.24xlarge’s availability at aws, not sure when we will get that.
2023-01-04T06:56:06Z
@jeffboudierAny suggestion on the device selection for “google/flan-t5-xxl” (11B), “bigscience/bloomz” (176B) and OPT (175B) models?
2023-01-05T14:45:52Z
Hi - for 11B model you can check out this tutorial to run T5-11B on a T4 via Inference Endpoints:Deploy T5 11B for inference for less than $500BLOOMZ and OPT would require a 8xA100 80GB (e.g. p4de.24xlarge on AWS) instance which wouldn’t make sense to setup / grant for a couple hours, maybe your usage falls within the free limit of the Inference API? Azure is sponsoring free inference for BLOOM (not BLOOMZ) in the Inference API - hosted on a 8xA100 80GB.
2023-01-06T02:42:17Z
Thanks a lot for the suggestion.@jeffboudier