{"action":"created","issue":{"url":"https://api.github.com/repos/ncw/rclone/issues/1606","repository_url":"https://api.github.com/repos/ncw/rclone","labels_url":"https://api.github.com/repos/ncw/rclone/issues/1606/labels{/name}","comments_url":"https://api.github.com/repos/ncw/rclone/issues/1606/comments","events_url":"https://api.github.com/repos/ncw/rclone/issues/1606/events","html_url":"https://github.com/ncw/rclone/issues/1606","id":250646901,"number":1606,"title":"Dreaded userRateLimitExceeded","user":{"login":"pcarrollarch","id":30906845,"avatar_url":"https://avatars0.githubusercontent.com/u/30906845?v=4","gravatar_id":"","url":"https://api.github.com/users/pcarrollarch","html_url":"https://github.com/pcarrollarch","followers_url":"https://api.github.com/users/pcarrollarch/followers","following_url":"https://api.github.com/users/pcarrollarch/following{/other_user}","gists_url":"https://api.github.com/users/pcarrollarch/gists{/gist_id}","starred_url":"https://api.github.com/users/pcarrollarch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pcarrollarch/subscriptions","organizations_url":"https://api.github.com/users/pcarrollarch/orgs","repos_url":"https://api.github.com/users/pcarrollarch/repos","events_url":"https://api.github.com/users/pcarrollarch/events{/privacy}","received_events_url":"https://api.github.com/users/pcarrollarch/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-08-16T14:27:22Z","updated_at":"2017-08-23T10:10:06Z","closed_at":null,"body":"Using the latest beta, I cannot get dedupe to act on a larger data set. Works fine on a smal amount of test data. But on a larger set, it constantly times out. You can leave it for hours and nothing will happen. Left it alone for the weekend. First attempt on Monday failed. It's running on a Celeron linux based QNAP NAS. \r\n\r\nrclone rmdirs exhibits the same behaviour on the large data set. Running an rclone sync on the same data set works perfectly. But of course it skips duplicate folders so I do not get all the data...\r\n\r\nTo me, this does not look like a rate limit problem. Otherwise I would expect all calls to fail.\r\n\r\n# rclone -vv dedupe newest gdrive:test\r\n2017/08/16 10:18:46 DEBUG : rclone: Version \"v1.37-090-g9c47b767β\" starting with parameters [\"rclone\" \"-vv\" \"dedupe\" \"newest\" \"gdrive:test\"]\r\n2017/08/16 10:18:47 DEBUG : gdrive: Saved new token in config file\r\n2017/08/16 10:18:47 INFO  : Google drive root 'test': Modify window is 1ms\r\n2017/08/16 10:18:47 INFO  : Google drive root 'test': Looking for duplicates using newest mode.\r\n2017/08/16 10:18:49 DEBUG : Go routines at exit 6\r\n2017/08/16 10:18:49 DEBUG : rclone: Version \"v1.37-090-g9c47b767β\" finishing with parameters [\"rclone\" \"-vv\" \"dedupe\" \"newest\" \"gdrive:test\"]\r\n# rclone -vv dedupe newest gdrive:Something/Bigger\r\n2017/08/16 10:19:26 DEBUG : rclone: Version \"v1.37-090-g9c47b767β\" starting with parameters [\"rclone\" \"-vv\" \"dedupe\" \"newest\" \"gdrive:Something/Else\"]\r\n2017/08/16 10:19:27 INFO  : Google drive root 'Something/Bigger': Modify window is 1ms\r\n2017/08/16 10:19:27 INFO  : Google drive root 'Something/Bigger': Looking for duplicates using newest mode.\r\n2017/08/16 10:19:34 DEBUG : pacer: Rate limited, sleeping for 1.74886657s (1 consecutive low level retries)\r\n2017/08/16 10:19:34 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)\r\n2017/08/16 10:19:34 DEBUG : pacer: Resetting sleep to minimum 10ms on success\r\n2017/08/16 10:19:34 DEBUG : pacer: Rate limited, sleeping for 1.649146272s (1 consecutive low level retries)\r\n2017/08/16 10:19:34 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)\r\n2017/08/16 10:19:34 DEBUG : pacer: Resetting sleep to minimum 10ms on success\r\n2017/08/16 10:19:36 DEBUG : pacer: Rate limited, sleeping for 1.454259606s (1 consecutive low level retries)\r\n2017/08/16 10:19:36 DEBUG : pacer: low level retry 3/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)\r\n2017/08/16 10:19:36 DEBUG : pacer: Rate limited, sleeping for 2.214630369s (2 consecutive low level retries)\r\n2017/08/16 10:19:36 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)\r\n2017/08/16 10:19:36 DEBUG : pacer: Rate limited, sleeping for 4.17680991s (3 consecutive low level retries)\r\n2017/08/16 10:19:36 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)\r\n2017/08/16 10:19:36 DEBUG : pacer: Rate limited, sleeping for 8.859277607s (4 consecutive low level retries)\r\n2017/08/16 10:19:36 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)\r\n2017/08/16 10:19:36 DEBUG : pacer: Rate limited, sleeping for 16.978727474s (5 consecutive low level retries)\r\n2017/08/16 10:19:36 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)\r\n2017/08/16 10:19:36 DEBUG : pacer: Resetting sleep to minimum 10ms on success\r\n2017/08/16 10:19:42 DEBUG : pacer: Rate limited, sleeping for 1.844537542s (1 consecutive low level retries)\r\n2017/08/16 10:19:42 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)\r\n2017/08/16 10:19:42 DEBUG : pacer: Resetting sleep to minimum 10ms on success\r\n2017/08/16 10:19:42 DEBUG : pacer: Rate limited, sleeping for 1.228331083s (1 consecutive low level retries)\r\n2017/08/16 10:19:42 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)\r\n2017/08/16 10:19:42 DEBUG : pacer: Rate limited, sleeping for 2.086382607s (2 consecutive low level retries)\r\n2017/08/16 10:19:42 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)\r\n2017/08/16 10:19:42 DEBUG : pacer: Resetting sleep to minimum 10ms on success\r\n2017/08/16 10:19:44 DEBUG : pacer: Rate limited, sleeping for 1.590990518s (1 consecutive low level retries)\r\n2017/08/16 10:19:44 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)\r\n2017/08/16 10:19:44 DEBUG : pacer: Resetting sleep to minimum 10ms on success\r\n2017/08/16 10:19:44 DEBUG : pacer: Rate limited, sleeping for 1.29803824s (1 consecutive low level retries)\r\n2017/08/16 10:19:44 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded, userRateLimitExceeded)\r\n2017/08/16 10:19:44 DEBUG : pacer: Resetting sleep to minimum 10ms on success\r\n^C\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/ncw/rclone/issues/comments/324284503","html_url":"https://github.com/ncw/rclone/issues/1606#issuecomment-324284503","issue_url":"https://api.github.com/repos/ncw/rclone/issues/1606","id":324284503,"user":{"login":"bittaurus","id":29684716,"avatar_url":"https://avatars0.githubusercontent.com/u/29684716?v=4","gravatar_id":"","url":"https://api.github.com/users/bittaurus","html_url":"https://github.com/bittaurus","followers_url":"https://api.github.com/users/bittaurus/followers","following_url":"https://api.github.com/users/bittaurus/following{/other_user}","gists_url":"https://api.github.com/users/bittaurus/gists{/gist_id}","starred_url":"https://api.github.com/users/bittaurus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bittaurus/subscriptions","organizations_url":"https://api.github.com/users/bittaurus/orgs","repos_url":"https://api.github.com/users/bittaurus/repos","events_url":"https://api.github.com/users/bittaurus/events{/privacy}","received_events_url":"https://api.github.com/users/bittaurus/received_events","type":"User","site_admin":false},"created_at":"2017-08-23T10:10:06Z","updated_at":"2017-08-23T10:10:06Z","body":"As soon as you see:\r\n\r\n2017/08/16 10:19:36 DEBUG : pacer: Resetting sleep to minimum 10ms on success\r\n\r\nIt means the current transaction succeeded and it's on to the next.  Google drive seems to throttle based on current global usage level, so if it's a busier time of day you wait longer.\r\n\r\nIf you wait long enough it will eventually finish."}}
{"actor":{"display_login":"bittaurus"}}