{"action":"created","issue":{"url":"https://api.github.com/repos/habitat-sh/habitat/issues/2427","repository_url":"https://api.github.com/repos/habitat-sh/habitat","labels_url":"https://api.github.com/repos/habitat-sh/habitat/issues/2427/labels{/name}","comments_url":"https://api.github.com/repos/habitat-sh/habitat/issues/2427/comments","events_url":"https://api.github.com/repos/habitat-sh/habitat/issues/2427/events","html_url":"https://github.com/habitat-sh/habitat/pull/2427","id":230050230,"number":2427,"title":"Ship logs to an S3-compatible object store","user":{"login":"christophermaier","id":207178,"avatar_url":"https://avatars1.githubusercontent.com/u/207178?v=3","gravatar_id":"","url":"https://api.github.com/users/christophermaier","html_url":"https://github.com/christophermaier","followers_url":"https://api.github.com/users/christophermaier/followers","following_url":"https://api.github.com/users/christophermaier/following{/other_user}","gists_url":"https://api.github.com/users/christophermaier/gists{/gist_id}","starred_url":"https://api.github.com/users/christophermaier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/christophermaier/subscriptions","organizations_url":"https://api.github.com/users/christophermaier/orgs","repos_url":"https://api.github.com/users/christophermaier/repos","events_url":"https://api.github.com/users/christophermaier/events{/privacy}","received_events_url":"https://api.github.com/users/christophermaier/received_events","type":"User","site_admin":false},"labels":[{"id":446438964,"url":"https://api.github.com/repos/habitat-sh/habitat/labels/Builder","name":"Builder","color":"b60205","default":false},{"id":450045549,"url":"https://api.github.com/repos/habitat-sh/habitat/labels/DO%20NOT%20MERGE","name":"DO NOT MERGE","color":"5319e7","default":false}],"state":"open","locked":false,"assignee":{"login":"christophermaier","id":207178,"avatar_url":"https://avatars1.githubusercontent.com/u/207178?v=3","gravatar_id":"","url":"https://api.github.com/users/christophermaier","html_url":"https://github.com/christophermaier","followers_url":"https://api.github.com/users/christophermaier/followers","following_url":"https://api.github.com/users/christophermaier/following{/other_user}","gists_url":"https://api.github.com/users/christophermaier/gists{/gist_id}","starred_url":"https://api.github.com/users/christophermaier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/christophermaier/subscriptions","organizations_url":"https://api.github.com/users/christophermaier/orgs","repos_url":"https://api.github.com/users/christophermaier/repos","events_url":"https://api.github.com/users/christophermaier/events{/privacy}","received_events_url":"https://api.github.com/users/christophermaier/received_events","type":"User","site_admin":false},"assignees":[{"login":"christophermaier","id":207178,"avatar_url":"https://avatars1.githubusercontent.com/u/207178?v=3","gravatar_id":"","url":"https://api.github.com/users/christophermaier","html_url":"https://github.com/christophermaier","followers_url":"https://api.github.com/users/christophermaier/followers","following_url":"https://api.github.com/users/christophermaier/following{/other_user}","gists_url":"https://api.github.com/users/christophermaier/gists{/gist_id}","starred_url":"https://api.github.com/users/christophermaier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/christophermaier/subscriptions","organizations_url":"https://api.github.com/users/christophermaier/orgs","repos_url":"https://api.github.com/users/christophermaier/repos","events_url":"https://api.github.com/users/christophermaier/events{/privacy}","received_events_url":"https://api.github.com/users/christophermaier/received_events","type":"User","site_admin":false}],"milestone":null,"comments":0,"created_at":"2017-05-19T18:17:38Z","updated_at":"2017-05-19T18:17:40Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/habitat-sh/habitat/pulls/2427","html_url":"https://github.com/habitat-sh/habitat/pull/2427","diff_url":"https://github.com/habitat-sh/habitat/pull/2427.diff","patch_url":"https://github.com/habitat-sh/habitat/pull/2427.patch"},"body":"*DON'T MERGE YET*: Still working out some kinks around local archiving for dev. The key functionality is present here now, though.\r\n\r\nWhen a job log is complete, we now archive it in an S3 API-compatible\r\nobject store (tested against S3 and [minio][1]), deleting it from the\r\njob server. Correspondingly, when an API request comes in for a given\r\nlog, we retrieve the contents from the archive and forward it to the\r\nuser.\r\n\r\nOne side effect of this implementation is that we no longer record a\r\n`log_url` for a job in the database, but rather simply mark it as\r\n\"archived\". The archiver itself knows how to retrieve the logs given\r\na job ID.\r\n\r\nThe implementation use the `aws-sdk-rust` crate, which, despite the\r\ngeneral name, only interfaces with S3. This was chosen instead of the\r\nmore widely-used `rusoto` library specifically because `aws-sdk-rust`\r\nhas the ability to interact with arbitrary compatible object stores,\r\nand not just AWS S3; this was a constraint of the feature, because we\r\ndid not want to mandate the use of AWS to run builder.\r\n\r\nConfiguration\r\n=============\r\n\r\nThe job server's configuration now has an `[archive]` section. Here is\r\nan example for using a bucket in S3's `us-east-1` region:\r\n\r\n```toml\r\n[archive]\r\nkey = \"THIS_IS_A_KEY\"\r\nsecret = \"THIS_IS_A_SECRET\"\r\nbucket = \"habitat-log-bukkit\"\r\n```\r\n\r\nAlternatively, you may specify a region if you do not use the\r\n`us-east-1` default.\r\n\r\n```toml\r\n[archive]\r\nkey = \"THIS_IS_A_KEY\"\r\nsecret = \"THIS_IS_A_SECRET\"\r\nbucket = \"habitat-log-bukkit\"\r\nregion = \"us-west-2\"\r\n```\r\n\r\nTo use Minio, you must supply an endpoint:\r\n\r\n```toml\r\n[archive]\r\nkey = \"THIS_IS_A_KEY\"\r\nsecret = \"THIS_IS_A_SECRET\"\r\nbucket = \"habitat-log-bukkit\"\r\nendpoint = \"http://minio.mycompany.com:9000\"\r\n```\r\n\r\nBy default, Minio uses `us-east-1` as its region (for API\r\ncompatibility); if you override this in your Minio installation, you\r\nshould also include this region in your configuration.\r\n\r\nThe archiver will access objects in the bucket using the \"virtual\r\nbucket\" model (e.g. `${BUCKET_NAME}.s3.amazonaws.com` for AWS. If\r\nfor some reason you want / need to use the path model\r\n(e.g. `s3.amazonaws.com/${BUCKET_NAME}`), you will need to specify\r\nboth the appropriate region and the corresponding endpoint, such as:\r\n\r\n```toml\r\n   # ...\r\n   region = \"us-west-2\"\r\n   endpoint = \"http://s3-us-west-2.amazonaws.com\"\r\n   # ...\r\n```\r\n(In the future this may become a configuration option.)\r\n\r\nFor the appropriate endpoint name, see\r\nhttp://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region For\r\nMinio, the archiver currently always uses the path access model.\r\n\r\nFailure Modes\r\n=============\r\n\r\nAs soon as the log is recognized as being complete, we archive it. If\r\nthe archiving call fails for whatever reason, we do not delete the\r\nlocal log file on disk. API calls to retrieve the log will still work,\r\nbeing satisfied from the on-disk file, but it won't be marked as\r\n\"complete\" in the API response, which would cause the UI to repeatedly\r\nquery for more lines. This is probably going to be rare, but we should\r\ncome up with a way to try and re-archive things that couldn't get\r\narchived the first time around; we'll have the data on hand to\r\ndetermine which file should be retried.\r\n\r\nFixes #2305\r\n\r\n[1]: https://minio.io\r\n\r\nSigned-off-by: Christopher Maier <cmaier@chef.io>"},"comment":{"url":"https://api.github.com/repos/habitat-sh/habitat/issues/comments/302774958","html_url":"https://github.com/habitat-sh/habitat/pull/2427#issuecomment-302774958","issue_url":"https://api.github.com/repos/habitat-sh/habitat/issues/2427","id":302774958,"user":{"login":"thesentinels","id":18452204,"avatar_url":"https://avatars0.githubusercontent.com/u/18452204?v=3","gravatar_id":"","url":"https://api.github.com/users/thesentinels","html_url":"https://github.com/thesentinels","followers_url":"https://api.github.com/users/thesentinels/followers","following_url":"https://api.github.com/users/thesentinels/following{/other_user}","gists_url":"https://api.github.com/users/thesentinels/gists{/gist_id}","starred_url":"https://api.github.com/users/thesentinels/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thesentinels/subscriptions","organizations_url":"https://api.github.com/users/thesentinels/orgs","repos_url":"https://api.github.com/users/thesentinels/repos","events_url":"https://api.github.com/users/thesentinels/events{/privacy}","received_events_url":"https://api.github.com/users/thesentinels/received_events","type":"User","site_admin":false},"created_at":"2017-05-19T18:17:40Z","updated_at":"2017-05-19T18:17:40Z","body":"Thanks for the pull request! Here is what will happen next:\n 1. Your PR will be reviewed by the maintainers\n 2. If everything looks good, one of them will `approve` it, and your PR will be merged.\n\nThank you for contributing!"}}
{"actor":{"display_login":"thesentinels"}}