How to get GPTJ 8bit working in a Jupiter notebook?
Hi everyone.I’m trying to get this model:https://huggingface.co/hivemind/gpt-j-6B-8bitworking in a notebook on paper space. It seems when I try this 8bit model with the followingtokenizer = AutoTokenizer.from_pretrained("hivemind/gpt-j-6B-8bit")the tokenizer line crashes the cell with an error saying that the tokenizer for this cannot be found.when I try to load the model with the following. The first time I run it it downloads the model and moves on to the tokenizer line, but the second time I run the cell this line seems to crash the cell and I don’t get any output after this line.model = AutoModelForCausalLM.from_pretrained("hivemind/gpt-j-6B-8bit")I’m not sure what other trouble shooting steps I should try? I can load the 16bit model just fine by following the docs, but the 8 bit model does not work.Thanks.
2022-12-01T08:14:40Z
Hey !You can now load and use 8-bit models easily ontransformers! Check for example this demo:Google Colabthat uses 8bit on T5