“No space left on device” when using Robust Speech Challenge + AzureML
Hi thereI have been trying to train a model as per the[Robust Speech Event](https://github.com/neelan-elucidate-ai/transformers/tree/main/examples/research_projects/robust-speech-event)Training begins but seems to fill up/dev/sdb1location which has a total of 119 GB. This happens even when i force the cache to be on a different drive as suggestedhere: both manually in the functions and via env var.This happens regardless of the dataset used (at fist I thought it was an issue with"load_dataset("mozilla-foundation/common_voice_7_0", "en")I suspect the very large model and data need a very big tmp location but it should be clearing the tmp not filling it up.Any ideas@patrickvonplaten@osanseviero@anton-l?
2022-06-09T17:48:42Z
seems it was theoutput dirnot thecache_dircausing the issue. seem to be running now after changingoutput dir