{"action":"created","issue":{"url":"https://api.github.com/repos/golang/dep/issues/121","repository_url":"https://api.github.com/repos/golang/dep","labels_url":"https://api.github.com/repos/golang/dep/issues/121/labels{/name}","comments_url":"https://api.github.com/repos/golang/dep/issues/121/comments","events_url":"https://api.github.com/repos/golang/dep/issues/121/events","html_url":"https://github.com/golang/dep/issues/121","id":202606537,"number":121,"title":"vendor/ verification","user":{"login":"sdboyer","id":21599,"avatar_url":"https://avatars0.githubusercontent.com/u/21599?v=4","gravatar_id":"","url":"https://api.github.com/users/sdboyer","html_url":"https://github.com/sdboyer","followers_url":"https://api.github.com/users/sdboyer/followers","following_url":"https://api.github.com/users/sdboyer/following{/other_user}","gists_url":"https://api.github.com/users/sdboyer/gists{/gist_id}","starred_url":"https://api.github.com/users/sdboyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sdboyer/subscriptions","organizations_url":"https://api.github.com/users/sdboyer/orgs","repos_url":"https://api.github.com/users/sdboyer/repos","events_url":"https://api.github.com/users/sdboyer/events{/privacy}","received_events_url":"https://api.github.com/users/sdboyer/received_events","type":"User","site_admin":false},"labels":[{"id":456574659,"url":"https://api.github.com/repos/golang/dep/labels/help-wanted","name":"help-wanted","color":"128A0C","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":19,"created_at":"2017-01-23T17:59:10Z","updated_at":"2017-07-31T23:24:23Z","closed_at":null,"body":"dep needs a way of verifying that the contents of `vendor/` are what's expected. This serves three purposes:\r\n\r\n1. **Performance:** without a verification mechanism, the only safe choice dep can make is to drop and recreate vendor in its entirety on every operation. This is wasteful and slow; any operation that avoids unnecessary disk writes to determine if vendor *needs* updating will be faster, and would be a huge boon.\r\n2. **Transparency:** `dep status` is very limited in what it can report about `vendor/` if it can't verify that `vendor` is actually what's expected.\r\n3. **Security:** without a verification mechanism, we can't know that the code we're building with is the code we expect it should be. We haven't articulated a security model yet, but this is obviously a significant part of it.\r\n\r\nThe simple, obvious solution is to deterministically hash (SHA256, I imagine) the file trees of dependencies, then store the resulting digest in the lock. However, the need to perform vendor stripping (#120) complicates this, as that creates perfectly valid situations in which the code under `vendor/` might differ from the upstream source. The more choice around stripping modes we create, the more complex and error-prone this challenge becomes.\r\n\r\nOr, we might simply rely on the hashing performed by the underlying VCS - indeed, we already record \"revisions\" in the lock, so we're already sorta doing this. Relying on it is triply disqualified, though:\r\n\r\n* It suffers from the same issues with vendor pruning as above.\r\n* On the a security front, git, hg, bzr, and svn all rely on the broken SHA1 for basic integrity checks. (Even when git was first released, they made it explicitly clear that their security model is not based on the internal use of SHA1)\r\n* It may bind us to VCS being the underlying source for code, which is not a corner we want to paint ourselves into.\r\n\r\nThe only path forward I've come up with so far is to compute and record hashes for individual files contained in dependencies (perhaps before stripping, perhaps after). This would allow us to be more adaptive about assessing package integrity - e.g., we can know that it's OK if certain files are missing.\r\n\r\nThe downside is that these lists of file hash digests would be potentially huge (e.g., if you import k8s), though that might be mitigated by placing them under vendor/ rather than directly in the lock - sdboyer/gps#69. Also, without the larger security model in place, I don't know if disaggregating hashing to the individual file level might compromise some broader notion of integrity that will prove important.\r\n\r\nWe might *also* record just the project unit-level hash against some future day where we've reinvented GOPATH and are no longer relying on vendor."},"comment":{"url":"https://api.github.com/repos/golang/dep/issues/comments/319224382","html_url":"https://github.com/golang/dep/issues/121#issuecomment-319224382","issue_url":"https://api.github.com/repos/golang/dep/issues/121","id":319224382,"user":{"login":"sdboyer","id":21599,"avatar_url":"https://avatars0.githubusercontent.com/u/21599?v=4","gravatar_id":"","url":"https://api.github.com/users/sdboyer","html_url":"https://github.com/sdboyer","followers_url":"https://api.github.com/users/sdboyer/followers","following_url":"https://api.github.com/users/sdboyer/following{/other_user}","gists_url":"https://api.github.com/users/sdboyer/gists{/gist_id}","starred_url":"https://api.github.com/users/sdboyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sdboyer/subscriptions","organizations_url":"https://api.github.com/users/sdboyer/orgs","repos_url":"https://api.github.com/users/sdboyer/repos","events_url":"https://api.github.com/users/sdboyer/events{/privacy}","received_events_url":"https://api.github.com/users/sdboyer/received_events","type":"User","site_admin":false},"created_at":"2017-07-31T23:24:23Z","updated_at":"2017-07-31T23:24:23Z","body":"> However, this feature involves creating a check that ensures whatever library is written in the `vendor/` directory matches what is in the `Gopkg.lock` file, by checking to ensure each library in `vendor/` has a directory hash that matches some pre-computed value.\r\n\r\nyes - that's pretty much the flow i'm imagining for us :) is there an alternative?\r\n\r\n> I _think_ it is a safe assumption that libraries are the list of directories that are three levels down from the original `vendor/` directory.\r\n\r\nif only it were 🤕 \r\n\r\nlaunchpad, which supports two-level roots (e.g. `launchpad.net/gocheck`) is an obvious exception, but the actual [ruleset](https://github.com/golang/dep/blob/master/internal/gps/deduce.go) makes no guarantee of three levels.\r\n\r\nnevertheless, i don't think you need to rethink implementation much. while we can't know _a priori_ how deep the root is along any given path, the root location is knowledge the rest of the system already has - and it's represented in the `gps.Lock`, via `Lock.Projects()`. the `HashPackageTree(path string)` func i described will need to be called for each of those roots.\r\n\r\nthe only slightly tricky bit would be the `NotInLock` case - when there's directories in `vendor` that are not equal to or logical children of any of the roots given in the locked project list. still, that shouldn't be too bad - gps already makes liberal use of `github.com/armon/go-radix` for tasks like these, and `isPathPrefixOrEqual()` should get you most of the rest of the way. (it's also possible that we may not actually need to hash such dirs at all)."}}
{"actor":{"display_login":"sdboyer"}}