{"action":"created","issue":{"url":"https://api.github.com/repos/aws/aws-cli/issues/2661","repository_url":"https://api.github.com/repos/aws/aws-cli","labels_url":"https://api.github.com/repos/aws/aws-cli/issues/2661/labels{/name}","comments_url":"https://api.github.com/repos/aws/aws-cli/issues/2661/comments","events_url":"https://api.github.com/repos/aws/aws-cli/issues/2661/events","html_url":"https://github.com/aws/aws-cli/issues/2661","id":235772369,"number":2661,"title":"s3 sync calls open() even if no files need to be uploaded","user":{"login":"steffenweber","id":1138129,"avatar_url":"https://avatars0.githubusercontent.com/u/1138129?v=3","gravatar_id":"","url":"https://api.github.com/users/steffenweber","html_url":"https://github.com/steffenweber","followers_url":"https://api.github.com/users/steffenweber/followers","following_url":"https://api.github.com/users/steffenweber/following{/other_user}","gists_url":"https://api.github.com/users/steffenweber/gists{/gist_id}","starred_url":"https://api.github.com/users/steffenweber/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steffenweber/subscriptions","organizations_url":"https://api.github.com/users/steffenweber/orgs","repos_url":"https://api.github.com/users/steffenweber/repos","events_url":"https://api.github.com/users/steffenweber/events{/privacy}","received_events_url":"https://api.github.com/users/steffenweber/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-06-14T06:13:01Z","updated_at":"2017-06-14T07:56:25Z","closed_at":null,"body":"I'm using `aws s3 sync /local/dir/ s3://mybucket --region eu-central-1 --delete` to sync a local directory hierarchy (totalling 580 GB distributed among 3.4 million files) to an S3 bucket. This currently takes about 35 minutes and I'm wondering whether this could be optimized.\r\n\r\nI've attached `strace` to aws-cli to monitor syscalls. Aside from multiple `stat`calls per file I've noticed that `open` is called on files even if they don't have to be transferred (the local directory and S3 are in sync). For each file I'm seeing the following `strace` output:\r\n\r\n```\r\nstat(\"file.jpg\", {st_mode=S_IFREG|0664, st_size=21646, ...}) = 0\r\nlstat(\"file.jpg\", {st_mode=S_IFREG|0664, st_size=21646, ...}) = 0\r\nstat(\"file.jpg\", {st_mode=S_IFREG|0664, st_size=21646, ...}) = 0\r\nstat(\"file.jpg\", {st_mode=S_IFREG|0664, st_size=21646, ...}) = 0\r\nstat(\"file.jpg\", {st_mode=S_IFREG|0664, st_size=21646, ...}) = 0\r\nopen(\"file.jpg\", O_RDONLY|O_CLOEXEC) = 5\r\nfstat(5, {st_mode=S_IFREG|0664, st_size=21646, ...}) = 0\r\nioctl(5, TCGETS, 0x7ffc971b2330)        = -1 ENOTTY (Inappropriate ioctl for device)\r\nfstat(5, {st_mode=S_IFREG|0664, st_size=21646, ...}) = 0\r\nlseek(5, 0, SEEK_CUR)                   = 0\r\nioctl(5, TCGETS, 0x7ffc971b2290)        = -1 ENOTTY (Inappropriate ioctl for device)\r\nlseek(5, 0, SEEK_CUR)                   = 0\r\nclose(5)                                = 0\r\nstat(\"file.jpg\", {st_mode=S_IFREG|0664, st_size=21646, ...}) = 0\r\n```\r\n\r\nI'm not sure whether this `open` call could have a big impact. From what I'm reading, `open` is more expensive than `stat`. Is the `open` call expected behaviour? It only seems to be used for `fstat` calls, no data is read.\r\n\r\nI've taken a look at the aws-cli source code but the only `open` call I've found is in an error-handling code path in `filegenerator.py` which should not be the issue here. The `open` syscall is probably caused indirectly by Python. But I don't know where to look.\r\n\r\n(I'm looking at I/O syscalls because the sync is not CPU bound, it uses about 70% of a single CPU core according to `top`.)\r\n\r\n```\r\n$ aws --version\r\naws-cli/1.11.81 Python/3.4.5 Linux/4.9.28 botocore/1.5.52\r\n```"},"comment":{"url":"https://api.github.com/repos/aws/aws-cli/issues/comments/308351464","html_url":"https://github.com/aws/aws-cli/issues/2661#issuecomment-308351464","issue_url":"https://api.github.com/repos/aws/aws-cli/issues/2661","id":308351464,"user":{"login":"steffenweber","id":1138129,"avatar_url":"https://avatars0.githubusercontent.com/u/1138129?v=3","gravatar_id":"","url":"https://api.github.com/users/steffenweber","html_url":"https://github.com/steffenweber","followers_url":"https://api.github.com/users/steffenweber/followers","following_url":"https://api.github.com/users/steffenweber/following{/other_user}","gists_url":"https://api.github.com/users/steffenweber/gists{/gist_id}","starred_url":"https://api.github.com/users/steffenweber/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steffenweber/subscriptions","organizations_url":"https://api.github.com/users/steffenweber/orgs","repos_url":"https://api.github.com/users/steffenweber/repos","events_url":"https://api.github.com/users/steffenweber/events{/privacy}","received_events_url":"https://api.github.com/users/steffenweber/received_events","type":"User","site_admin":false},"created_at":"2017-06-14T07:56:25Z","updated_at":"2017-06-14T07:56:25Z","body":"If I'm interpreting the `time` output correctly then at most 14 minutes are spent waiting for I/O.\r\n\r\n```\r\nreal\t35m20.771s\r\nuser\t20m37.250s\r\nsys\t1m26.327s\r\n```\r\n\r\nSo probably the `open` call is not such a huge issue."}}
{"actor":{"display_login":"steffenweber"}}