megablocks warning and poetry
I"m getting warning !!!!!megablocks not available, using torch.matmul instead when using this model. Trying to install megablocks works well but poetry add git+https://github.com/nomic-ai/megablocks.git failes with ModuleNotFoundError: No module named 'torch'.
Is there any way to add megablocks with poetry?
I don't have any experience with poetry but you probably need to install megablocks after you have torch and run the equivalent of
pip install git+https://github.com/nomic-ai/megablocks.git --no-build-isolation
Got you. In fact, I'm getting faster inference with no megablocks installed in my case. As far as I see from modeling_hf_nomic_bert.py it's not critical to have this module installed. Maybe I'll just leave this for now :-)
that’s surprising! out of curiosity what type of hardware and batch size are you running this on?
i'm testing on RTX 3080 with bach size 64, CUDA 12.4