{"action":"created","issue":{"url":"https://api.github.com/repos/codedellemc/rexray/issues/847","repository_url":"https://api.github.com/repos/codedellemc/rexray","labels_url":"https://api.github.com/repos/codedellemc/rexray/issues/847/labels{/name}","comments_url":"https://api.github.com/repos/codedellemc/rexray/issues/847/comments","events_url":"https://api.github.com/repos/codedellemc/rexray/issues/847/events","html_url":"https://github.com/codedellemc/rexray/issues/847","id":227719355,"number":847,"title":"resource not found with v0.9.0","user":{"login":"gpaul","id":442,"avatar_url":"https://avatars2.githubusercontent.com/u/442?v=3","gravatar_id":"","url":"https://api.github.com/users/gpaul","html_url":"https://github.com/gpaul","followers_url":"https://api.github.com/users/gpaul/followers","following_url":"https://api.github.com/users/gpaul/following{/other_user}","gists_url":"https://api.github.com/users/gpaul/gists{/gist_id}","starred_url":"https://api.github.com/users/gpaul/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gpaul/subscriptions","organizations_url":"https://api.github.com/users/gpaul/orgs","repos_url":"https://api.github.com/users/gpaul/repos","events_url":"https://api.github.com/users/gpaul/events{/privacy}","received_events_url":"https://api.github.com/users/gpaul/received_events","type":"User","site_admin":false},"labels":[{"id":270032141,"url":"https://api.github.com/repos/codedellemc/rexray/labels/triage","name":"triage","color":"e11d21","default":false}],"state":"open","locked":false,"assignee":{"login":"akutz","id":101085,"avatar_url":"https://avatars1.githubusercontent.com/u/101085?v=3","gravatar_id":"","url":"https://api.github.com/users/akutz","html_url":"https://github.com/akutz","followers_url":"https://api.github.com/users/akutz/followers","following_url":"https://api.github.com/users/akutz/following{/other_user}","gists_url":"https://api.github.com/users/akutz/gists{/gist_id}","starred_url":"https://api.github.com/users/akutz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/akutz/subscriptions","organizations_url":"https://api.github.com/users/akutz/orgs","repos_url":"https://api.github.com/users/akutz/repos","events_url":"https://api.github.com/users/akutz/events{/privacy}","received_events_url":"https://api.github.com/users/akutz/received_events","type":"User","site_admin":false},"assignees":[{"login":"akutz","id":101085,"avatar_url":"https://avatars1.githubusercontent.com/u/101085?v=3","gravatar_id":"","url":"https://api.github.com/users/akutz","html_url":"https://github.com/akutz","followers_url":"https://api.github.com/users/akutz/followers","following_url":"https://api.github.com/users/akutz/following{/other_user}","gists_url":"https://api.github.com/users/akutz/gists{/gist_id}","starred_url":"https://api.github.com/users/akutz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/akutz/subscriptions","organizations_url":"https://api.github.com/users/akutz/orgs","repos_url":"https://api.github.com/users/akutz/repos","events_url":"https://api.github.com/users/akutz/events{/privacy}","received_events_url":"https://api.github.com/users/akutz/received_events","type":"User","site_admin":false}],"milestone":null,"comments":8,"created_at":"2017-05-10T15:27:39Z","updated_at":"2017-05-18T12:52:03Z","closed_at":null,"body":"# Summary\r\nWe ran some tests in a loop overnight and found that rexray hiccuped once. From the logs it looks like the AWS API had a glitchy moment, but I figured I'd share the logs just in case this points at a real issue.\r\n\r\n# Bug Reports\r\n\r\n## Version\r\n\r\n```\r\n$ rexray version\r\nINFO[0000] updated log level                             logLevel=debug\r\nDEBU[0000] os.args                                       time=1495111093877 val=[rexray version]\r\nREX-Ray\r\n-------\r\nBinary: /opt/mesosphere/packages/rexray--2e5d0eda71cb7474db386b659bd3709ec98e8c4f/bin/rexray\r\nFlavor: client+agent+controller\r\nSemVer: 0.9.0\r\nOsArch: Linux-x86_64\r\nBranch: (detached from 2a7458d\r\nCommit: 2a7458dd90a79c673463e14094377baf9fc8695e\r\nFormed: Mon, 08 May 2017 09:10:18 UTC\r\n\r\nlibStorage\r\n----------\r\nSemVer: 0.6.0\r\nOsArch: Linux-x86_64\r\nBranch: (detached from fa055d6\r\nCommit: fa055d6da595602715bdfd5541b4aa6d4dcbcbd9\r\nFormed: Mon, 08 May 2017 09:09:53 UTC\r\nDEBU[0000] completed cli execution                       time=1495111093878\r\nINFO[0000] exiting process                               time=1495111093878\r\nDEBU[0000] completed onExit at end of program            time=1495111093878\r\n```\r\n\r\n## Expected Behavior\r\nI expect the same rexray commands that worked hundreds of times in a row to keep working. More specifically, my DC/OS service tasks that are being run in a loop should keep succeeding.\r\n\r\n## Actual Behavior\r\nFor a long time all volume commands succeed on each of my 2 DC/OS agents. After many iterations of starting a task that mounts an external volume via REX-Ray, writes a token string to a file on the volume, and exits, the tasks start failing. All tasks use exactly the same volume. They are run serially. When a task exits, it is re-launched on one of the two agent nodes, randomly.\r\n\r\nWhen the task is scheduled to run on node 1, it always fails, on node 2, it always succeeds.\r\n\r\nMy suspicion is that the AWS API may be glitchy or perhaps that node has exceeded it's number of allowed API requests (ie., being rate limited) or something similar.   \r\n\r\nIf you search the log file for \"resource not found\" you can see that the volume in question is `vol-0319d68458d1f2063` and it certainly exists as can be confirmed by the printed response to `GET /volumes/ebs?attachments=0`.\r\n\r\nThe \"resource not found\" is emitted alongside \"api call failed\", perhaps that is the deeper error.\r\n\r\n## Configuration Files\r\n```\r\n$ cat /etc/rexray/config.yml \r\nrexray:\r\n  logLevel:        debug\r\n  service: ebs\r\nlibstorage:\r\n  logging:\r\n    level:         debug\r\n    httpRequests:  true\r\n    httpResponses: true\r\n  server:\r\n    tasks:\r\n      logTimeout: 5m\r\n  integration:\r\n    volume:\r\n      operations:\r\n        unmount:\r\n          ignoreusedcount: true\r\n```\r\n\r\n## Logs\r\n\r\n### Service Log\r\nThe log file as a gist: https://gist.github.com/gpaul/e29e79a9057ae60c4f13309cc899cd69\r\n"},"comment":{"url":"https://api.github.com/repos/codedellemc/rexray/issues/comments/302394481","html_url":"https://github.com/codedellemc/rexray/issues/847#issuecomment-302394481","issue_url":"https://api.github.com/repos/codedellemc/rexray/issues/847","id":302394481,"user":{"login":"gpaul","id":442,"avatar_url":"https://avatars2.githubusercontent.com/u/442?v=3","gravatar_id":"","url":"https://api.github.com/users/gpaul","html_url":"https://github.com/gpaul","followers_url":"https://api.github.com/users/gpaul/followers","following_url":"https://api.github.com/users/gpaul/following{/other_user}","gists_url":"https://api.github.com/users/gpaul/gists{/gist_id}","starred_url":"https://api.github.com/users/gpaul/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gpaul/subscriptions","organizations_url":"https://api.github.com/users/gpaul/orgs","repos_url":"https://api.github.com/users/gpaul/repos","events_url":"https://api.github.com/users/gpaul/events{/privacy}","received_events_url":"https://api.github.com/users/gpaul/received_events","type":"User","site_admin":false},"created_at":"2017-05-18T12:52:03Z","updated_at":"2017-05-18T12:52:03Z","body":"@akutz Thanks for your help so far, I've updated the issue description to match the template and entered the correct information."}}
{"actor":{"display_login":"gpaul"}}