How to enable NCCL P2P with Accelerate?
I have a 3 GPU config 1x4080 and 2x3090. The 3090s are connected via NVLink, tested it with NCCL test all works fine. I have an Accelerate config where I set to use only the 2 3090s. When I runaccelerate test:Output “Using RTX 4000 series which doesn’t support faster communication speedups. Ensuring P2P and IB communications are disabled.” even is I’ve specified to use only the 2 3090s. Does it always “looks” at all the GPUs not only the ones set to use?I have NCCL_DEBUG set to INFO and see “NCCL_P2P_LEVEL set by environment to LOC” though I’d like to use ‘NVL’. It’s not clear for me how can I configure this with Accelerate. I have export NCCL_P2P_LEVEL=NVL but seemingly has no effect.How can I configure Accelerate to use NVL?stdout: debian:34193:34210 [1] NCCL INFO NCCL_P2P_LEVEL set by environment to LOC
stdout: debian:34193:34210 [1] NCCL INFO === System : maxBw 6.0 totalBw 48.0 ===
stdout: debian:34193:34210 [1] NCCL INFO CPU/FFFFFFFFFFFFFFFF (1/2/-1)
stdout: debian:34193:34210 [1] NCCL INFO + PCI[12.0] - GPU/B000 (1)
stdout: debian:34193:34210 [1] NCCL INFO               + NVL[48.0] - GPU/A000
stdout: debian:34193:34210 [1] NCCL INFO + PCI[6.0] - GPU/A000 (0)
stdout: debian:34193:34210 [1] NCCL INFO              + NVL[48.0] - GPU/B000