Why isn't FSDP supported by DistributedCheckpointIO?
This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. I'm also interested in this. This issue is stale because it has been open for 30 days with no activity. Remove stale label or comment or this will be closed in 7 days. This issue was closed because it has been inactive for 7 days since being marked as stale.
Hi team,

Thank you for your efforts in adding async_save support to DistributedCheckpointIO. I am currently testing async_save with DistributedCheckpointIO and found that it only supports [non-FSDP MCore models](https://github.com/NVIDIA/NeMo/blob/ae06138a5012375e39a93b512d76b5fbc4d8c229/nemo/collections/nlp/parts/megatron_trainer_builder.py#L145). I have reviewed the source code, including [Megatron-LM](https://github.com/NVIDIA/Megatron-LM), but I am unclear on why FSDP is not supported by DistributedCheckpointIO. Could the team provide more information regarding this limitation? Thank you.