How to perform parallel inference using multiple GPU
Hi, is there a way to create an instance of LLM and load that model into two different GPUs? Note that the instance will be created in two different celery tasks (asynchronous task/job)
2024-04-10T10:54:17Z
Distributed inference with multiple GPUs (huggingface.co)
2024-04-10T13:02:37Z
I went through the documentation, but I still don’t know, how exactly will I be able to handle both the response from the “result”?