Is it possible to do inference on gpt-j-6B via Colab?
When I use thepipelineAPI, it crashes Colab with an out of memory error (fills 25.5GB of RAM). I think it should be possible to do the inference on TPUv2? But how do I tell thepipelineto start using the TPUs from the start?from transformers import pipeline
model_name = 'EleutherAI/gpt-j-6B'
generator = pipeline('text-generation', model=model_name)
out = generator("I am Harry Potter.", do_sample=True, min_length=50)
2021-12-21T13:41:58Z
Hi,Inference is only possible on Colab Pro. You can check my notebookherefor more info.