How does the GPT-J inference API work?
Hi All.I started a 7 day trial for the startup plan. I need to use GPT-J through HF inference API. I pinned it in on org account to work on GPU and, after sending a request, all I get back is a single generated word. The max token param is set to 100.Could you please let me know how should I make it generate more than one word
2021-09-28T07:46:19Z
Hi,Normally it should become available. I’ll ask the team and get back to you.
2021-09-28T08:03:16Z
GPT-J is supported by the inference API. You can try it out here:EleutherAI/gpt-j-6B · Hugging FaceCan you verify whether you are able to generate more than a given word?
2021-09-29T07:03:39Z
Thanks for your reply! I verified and it still generates only one word for me. I’m sending the below mentioned json in my request tohttps://api-inference.huggingface.co/models/EleutherAI/gpt-j-6B{“inputs”: “Nowadays, many users would like to upgrade old hard drive to SSD with Windows installed, or reinstall Windows 10 on SSD afterward. The faster boot speed and reading & writing speed make it known as a better boot drive.\r\n\r\nTo be specifc, after installing the operating system on a new SSD, you’ll find the computer boots up faster, and runs smoothly even with muliple programs in the background.\r\n\r\nSo let’s”,“parameters”: {“return_full_text”: false,“max_new_tokens”: 100,“temperature”: 0.8},“options”: {“use_cache”: false}}And I’m getting back this:HTTP/1.1 200 OKdate: Wed, 29 Sep 2021 06:56:12 GMT,Wed, 29 Sep 2021 06:56:21 GMTserver: istio-envoyx-compute-time: 0.1596x-compute-type: gpuaccess-control-expose-headers: x-compute-type, x-compute-timex-compute-characters: 406content-length: 27content-type: application/jsonx-envoy-upstream-service-time: 177[{“generated_text”:" get"}]Is something wrong on my side?
2021-09-29T07:21:47Z
wittyseller:Nowadays, many users would like to upgrade old hard drive to SSD with Windows installed, or reinstall Windows 10 on SSD afterward.It’s strange: if I send only one sentence as input, it works:So this is my request:{“inputs”: “Nowadays, many users would like to upgrade old hard drive to SSD with Windows installed, or reinstall Windows 10 on SSD afterward.”,“parameters”: {“return_full_text”: false,“max_new_tokens”: 100,“temperature”: 0.8},“options”: {“use_cache”: false}}And I get back this:HTTP/1.1 200 OKdate: Wed, 29 Sep 2021 07:06:20 GMT,Wed, 29 Sep 2021 07:06:29 GMTserver: istio-envoyx-compute-time: 2.1995999999999998x-compute-type: gpuaccess-control-expose-headers: x-compute-type, x-compute-timex-compute-characters: 130content-length: 130content-type: application/jsonx-envoy-upstream-service-time: 2244[{“generated_text”:" In this way, they can enjoy the performance of SSD and the convenience of Windows. In the first place, the"}]To me, it looks like there is some sort of a limit on the input tokens
2021-10-08T15:48:26Z
Try using “max_length” parameter instead of “max_new_tokens”. The documentation suggests they serve the same purpose and you should not use both simultaneously. Worked for me.