Load checkpoints on CPU mem to prevent CUDA mem allocation
I was benchmarking Megatron-LM and ran into this issue when recording memory values. This means on the main process there is a lot of unnecessary CUDA memory allocated when loading the state dictionary. Using the proposed fix remedied the issue!