{"action":"created","issue":{"url":"https://api.github.com/repos/samtools/htslib/issues/436","repository_url":"https://api.github.com/repos/samtools/htslib","labels_url":"https://api.github.com/repos/samtools/htslib/issues/436/labels{/name}","comments_url":"https://api.github.com/repos/samtools/htslib/issues/436/comments","events_url":"https://api.github.com/repos/samtools/htslib/issues/436/events","html_url":"https://github.com/samtools/htslib/issues/436","id":189886270,"number":436,"title":"support for non-amazonaws endpoint urls (e.g. s3.<company>.com)","user":{"login":"blajoie","id":8889420,"avatar_url":"https://avatars1.githubusercontent.com/u/8889420?v=4","gravatar_id":"","url":"https://api.github.com/users/blajoie","html_url":"https://github.com/blajoie","followers_url":"https://api.github.com/users/blajoie/followers","following_url":"https://api.github.com/users/blajoie/following{/other_user}","gists_url":"https://api.github.com/users/blajoie/gists{/gist_id}","starred_url":"https://api.github.com/users/blajoie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/blajoie/subscriptions","organizations_url":"https://api.github.com/users/blajoie/orgs","repos_url":"https://api.github.com/users/blajoie/repos","events_url":"https://api.github.com/users/blajoie/events{/privacy}","received_events_url":"https://api.github.com/users/blajoie/received_events","type":"User","site_admin":false},"labels":[{"id":5087625,"url":"https://api.github.com/repos/samtools/htslib/labels/enhancement","name":"enhancement","color":"84b6eb","default":true},{"id":334925307,"url":"https://api.github.com/repos/samtools/htslib/labels/S3","name":"S3","color":"fef2c0","default":false}],"state":"open","locked":false,"assignee":{"login":"dkj","id":1033592,"avatar_url":"https://avatars2.githubusercontent.com/u/1033592?v=4","gravatar_id":"","url":"https://api.github.com/users/dkj","html_url":"https://github.com/dkj","followers_url":"https://api.github.com/users/dkj/followers","following_url":"https://api.github.com/users/dkj/following{/other_user}","gists_url":"https://api.github.com/users/dkj/gists{/gist_id}","starred_url":"https://api.github.com/users/dkj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dkj/subscriptions","organizations_url":"https://api.github.com/users/dkj/orgs","repos_url":"https://api.github.com/users/dkj/repos","events_url":"https://api.github.com/users/dkj/events{/privacy}","received_events_url":"https://api.github.com/users/dkj/received_events","type":"User","site_admin":false},"assignees":[{"login":"dkj","id":1033592,"avatar_url":"https://avatars2.githubusercontent.com/u/1033592?v=4","gravatar_id":"","url":"https://api.github.com/users/dkj","html_url":"https://github.com/dkj","followers_url":"https://api.github.com/users/dkj/followers","following_url":"https://api.github.com/users/dkj/following{/other_user}","gists_url":"https://api.github.com/users/dkj/gists{/gist_id}","starred_url":"https://api.github.com/users/dkj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dkj/subscriptions","organizations_url":"https://api.github.com/users/dkj/orgs","repos_url":"https://api.github.com/users/dkj/repos","events_url":"https://api.github.com/users/dkj/events{/privacy}","received_events_url":"https://api.github.com/users/dkj/received_events","type":"User","site_admin":false},{"login":"jenniferliddle","id":1533586,"avatar_url":"https://avatars3.githubusercontent.com/u/1533586?v=4","gravatar_id":"","url":"https://api.github.com/users/jenniferliddle","html_url":"https://github.com/jenniferliddle","followers_url":"https://api.github.com/users/jenniferliddle/followers","following_url":"https://api.github.com/users/jenniferliddle/following{/other_user}","gists_url":"https://api.github.com/users/jenniferliddle/gists{/gist_id}","starred_url":"https://api.github.com/users/jenniferliddle/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jenniferliddle/subscriptions","organizations_url":"https://api.github.com/users/jenniferliddle/orgs","repos_url":"https://api.github.com/users/jenniferliddle/repos","events_url":"https://api.github.com/users/jenniferliddle/events{/privacy}","received_events_url":"https://api.github.com/users/jenniferliddle/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/samtools/htslib/milestones/6","html_url":"https://github.com/samtools/htslib/milestone/6","labels_url":"https://api.github.com/repos/samtools/htslib/milestones/6/labels","id":1194394,"number":6,"title":"1.4","description":"","creator":{"login":"jmarshall","id":70921,"avatar_url":"https://avatars3.githubusercontent.com/u/70921?v=4","gravatar_id":"","url":"https://api.github.com/users/jmarshall","html_url":"https://github.com/jmarshall","followers_url":"https://api.github.com/users/jmarshall/followers","following_url":"https://api.github.com/users/jmarshall/following{/other_user}","gists_url":"https://api.github.com/users/jmarshall/gists{/gist_id}","starred_url":"https://api.github.com/users/jmarshall/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmarshall/subscriptions","organizations_url":"https://api.github.com/users/jmarshall/orgs","repos_url":"https://api.github.com/users/jmarshall/repos","events_url":"https://api.github.com/users/jmarshall/events{/privacy}","received_events_url":"https://api.github.com/users/jmarshall/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":33,"state":"open","created_at":"2015-07-06T08:55:27Z","updated_at":"2017-03-20T09:25:45Z","due_on":null,"closed_at":null},"comments":13,"created_at":"2016-11-16T23:44:06Z","updated_at":"2017-12-05T20:05:15Z","closed_at":null,"author_association":"NONE","body":"We are utilizing EMC on-premise s3-compatible storage.\r\n\r\nHtslib does not accommodate non (.s3.amazonaws.com) endpoint urls.\r\n\r\nWe can successfully modify, re-compile and use htslib/samtools if we swap in our own endpoint url as below:\r\n\r\nline 846 of hfile_libcurl.c\r\n```\r\n// Use virtual hosted-style access if possible, otherwise path-style.\r\n    if (is_dns_compliant(bucket, path)) {\r\n        kputsn(bucket, path - bucket, &url);\r\n        kputs(\"<custom_endpoint_url>\", &url);\r\n    }\r\n    else {\r\n        kputs(\"<custom_endpoint_url>/\", &url);\r\n        kputsn(bucket, path - bucket, &url);\r\n    }\r\n    kputs(path, &url);\r\n```\r\n\r\nCan htslib be modified to accept custom endpoint urls from all potential config locations?\r\ne.g.\r\n1. embeded within link - `s3://id:secret:endpoint@bucket/` (or similar)\r\n2. env variable - `AWS_ENDPOINT_URL=`\r\n3. within ~/.aws/config - `aws_endpoint_url=` (or some other standard, extracted from aws profile?)\r\n\r\nOr modify to utilize a  's3_domain' parameter?\r\n\r\nLooks like you've already touched on a few of these related to gcs:\r\nhttps://github.com/samtools/htslib/pull/390\r\n\r\n-\r\nIt would also be nice to modify the s3 logic to also take in a --profile parameter as used in aws-cli.\r\n"},"comment":{"url":"https://api.github.com/repos/samtools/htslib/issues/comments/349424691","html_url":"https://github.com/samtools/htslib/issues/436#issuecomment-349424691","issue_url":"https://api.github.com/repos/samtools/htslib/issues/436","id":349424691,"user":{"login":"wresch","id":272760,"avatar_url":"https://avatars1.githubusercontent.com/u/272760?v=4","gravatar_id":"","url":"https://api.github.com/users/wresch","html_url":"https://github.com/wresch","followers_url":"https://api.github.com/users/wresch/followers","following_url":"https://api.github.com/users/wresch/following{/other_user}","gists_url":"https://api.github.com/users/wresch/gists{/gist_id}","starred_url":"https://api.github.com/users/wresch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wresch/subscriptions","organizations_url":"https://api.github.com/users/wresch/orgs","repos_url":"https://api.github.com/users/wresch/repos","events_url":"https://api.github.com/users/wresch/events{/privacy}","received_events_url":"https://api.github.com/users/wresch/received_events","type":"User","site_admin":false},"created_at":"2017-12-05T20:05:15Z","updated_at":"2017-12-05T20:05:15Z","author_association":"NONE","body":"We patch our htslib locally to force path_style URLs via a 'url_mode' key in .s3cfg so that we can access a local cleversafe store. It would be nice if this mechanism or one like it could be made official.\r\n"}}
{"actor":{"display_login":"wresch"}}