Speeds up copying of necessary artifact files with SaveRestoreConnector
I'm looking into the test failures. Very bizarre since running the complete suite locally hangs and I cannot reproduce the failures by running the failed cases individually I figured out the test failure issue. It turned out in my unit test I `os.chdir` into a temporary directory and don't `os.chdir` back into the CWD. This causes failures for tests in different scopes. So I wasn't seeing the issue when running just the failed tests by themselves, but instead saw it only when running the entire suite together.
# What does this PR do ?

Previously, the SaveRestoreConnector would copy and untar entire
checkpoints just to copy out a tokenizer. For models in the >100GB, this
led to timeouts since only rank=0 did this work, while other ranks moved
on and waited at an all-gather barrier (observed NCCL timeout at 10min).

Revision of #9299 that keeps most of the core logic surrounding tempdir creation