{"action":"created","issue":{"url":"https://api.github.com/repos/golang/dep/issues/213","repository_url":"https://api.github.com/repos/golang/dep","labels_url":"https://api.github.com/repos/golang/dep/issues/213/labels{/name}","comments_url":"https://api.github.com/repos/golang/dep/issues/213/comments","events_url":"https://api.github.com/repos/golang/dep/issues/213/events","html_url":"https://github.com/golang/dep/issues/213","id":204498475,"number":213,"title":"The dep workflow: default constraints, minimal manifest?","user":{"login":"nathany","id":4566,"avatar_url":"https://avatars2.githubusercontent.com/u/4566?v=3","gravatar_id":"","url":"https://api.github.com/users/nathany","html_url":"https://github.com/nathany","followers_url":"https://api.github.com/users/nathany/followers","following_url":"https://api.github.com/users/nathany/following{/other_user}","gists_url":"https://api.github.com/users/nathany/gists{/gist_id}","starred_url":"https://api.github.com/users/nathany/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nathany/subscriptions","organizations_url":"https://api.github.com/users/nathany/orgs","repos_url":"https://api.github.com/users/nathany/repos","events_url":"https://api.github.com/users/nathany/events{/privacy}","received_events_url":"https://api.github.com/users/nathany/received_events","type":"User","site_admin":false},"labels":[{"id":456574661,"url":"https://api.github.com/repos/golang/dep/labels/question","name":"question","color":"cc317c","default":true}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2017-02-01T05:05:23Z","updated_at":"2017-04-10T17:12:48Z","closed_at":"2017-02-10T02:03:59Z","body":"This is a meta-issue after some discussion with @sdboyer on [#vendor](https://invite.slack.golangbridge.org/) to better understand the `dep` tool and to both explain my understanding and present what I think the workflow could be like based on what I currently know. Currently I'm only tackling the simple case of building a new app.\r\n\r\nThe `dep` tool is unique compared to its contemporaries because it is aware of the import graph contained in source code. That means the `import` statements in your source code are the source of truth. Bundler, Hex, NPM, etc. look to a manifest file for a list dependencies you manually maintain. This impacts what the so-named (#168) \"manifest\" file is actually for.\r\n\r\nSo to use dep, you can start by just writing code. At some point you need to `dep init` to create the manifest and lock file. The lock file is a full transitive list of all imported code with commit SHAs that make reproducible builds possible. The \"manifest\" need not list all the dependencies and constraints, in fact it could be essentially empty†. Let me explain.\r\n\r\nSay you `import \"foo/bar\"` and run `dep ensure` to ensure that everything imported is correctly vendored and tracked. \r\n\r\nThe `dep` tool can be smart enough to fetch the latest stable version `v1.3.4` of \"foo/bar\" rather than pulling master (as with `go get`) and rather than pulling `v2.0.0.beta.1`. Pre-releases are always opt-in, as the solver gives preference to stable versions.\r\n\r\nAt this point there is no need for anything to exist in the manifest, but the lock file will contain a specific commit SHA for \"foo/bar\" which you commit along with your code (and optionally the vendored copy of \"foo/bar\" for the _hermetic_ view of the world).\r\n\r\nNow lets say you `import \"baz/qux\"`. Even if \"foo/bar\" v1.3.5 is available, running `dep ensure` should vendor \"baz/qux\" but not touch \"foo/bar\" at all.\r\n\r\nA separate command should exist to `dep update foo/bar`, which would upgrade to `v1.3.5`. If no constraints are manually specified, `dep update x` always gets you the latest stable version, so once `v2.0.0` is out, `dep update foo/bar` would give you that.\r\n\r\nIMO, this should be fine for most cases. Upgrading versions is always done in a controlled manner and builds are reproducible thanks to the lock file. Additionally, you can use `dep status` to see what's going to happen before running `dep update x` and simply choose not to update to new major versions immediately.\r\n\r\nThe \"manifest\" comes into play when the default behaviour isn't what you want.\r\n\r\n* To opt-in to a pre-release version of a dependency `^v2.0.0.beta.1`\r\n* When two downstream dependencies have conflicting constraints, the manifest can specify an override to determine which one wins.\r\n* Or generally to override the preferences of a downstream dependency if you know better.\r\n* To lock to an older version due to an incompatibility, license change, or other consideration.\r\n* To use a temporary fork in place of a given dependency.\r\n\r\nThese are all exceptions to the general rule of latest stable version. In all these cases, it is helpful to include a comment in the manifest indicating **why** the constraint is specified.\r\n\r\n```toml\r\n[dependencies]\r\n\"foo/bar\" = \"^v1.3.5\" # v2 is incompatible with baz/qux\r\n```\r\n\r\nNext time you run `dep ensure`, the constraints in the manifest will override the default behaviour (latest stable version) when solving. However, if `v2.0.0` is already vendored, you may see an error, and need to run `dep update` to _update_ to the older specified version.\r\n\r\nHow does this sound so far?\r\n\r\nIf this makes sense, I'd suggest a few changes that could help direct some open issues:\r\n\r\n* The \"manifest\" should be a human-editable file that is created on `init` and never modified by the tool. Comments and organization of the file are up to the humans maintaining it. The initial file should include a wall of comments as documentation on usage. To support comments and for readability, the format preferably wouldn't be JSON (#119), but the lock file could remain JSON.\r\n* The `dep ensure foo/bar` variant should go away in favour of a separate update command. Neither command should alter the manifest. To add a dependency, write an `import` statement in your Go code, and run `dep ensure` to vendor the latest stable version.\r\n* The \"manifest\" probably shouldn't be called \"manifest\" (#168). It provides a way to override default behaviours. It's not a full list of everything (the lock file is).\r\n* The manifest is for specifying constraints and overrides. It is an error if the manifest specifies a package that isn't ever imported (much like unused variable/import in Go).\r\n* If supported, `dep update foo/bar v1.3.4` could override any constraints specified in the manifest too, vendoring v1.3.4. But assuming this doesn't update the manifest (not worth the trouble), maybe it's not worth supporting constraints on the CLI.\r\n\r\nThe end result of all this is a manifest designed for humans but that rarely needs editing. For the common case, just write your code, run `dep ensure`, and commit your work as usual. That's as simple as it gets.\r\n\r\n† Yet to confirm that the solver can cope with a large number of unspecified constraints (empty manifest) with a preference for the latest stable version.\r\n"},"comment":{"url":"https://api.github.com/repos/golang/dep/issues/comments/293016210","html_url":"https://github.com/golang/dep/issues/213#issuecomment-293016210","issue_url":"https://api.github.com/repos/golang/dep/issues/213","id":293016210,"user":{"login":"sdboyer","id":21599,"avatar_url":"https://avatars3.githubusercontent.com/u/21599?v=3","gravatar_id":"","url":"https://api.github.com/users/sdboyer","html_url":"https://github.com/sdboyer","followers_url":"https://api.github.com/users/sdboyer/followers","following_url":"https://api.github.com/users/sdboyer/following{/other_user}","gists_url":"https://api.github.com/users/sdboyer/gists{/gist_id}","starred_url":"https://api.github.com/users/sdboyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sdboyer/subscriptions","organizations_url":"https://api.github.com/users/sdboyer/orgs","repos_url":"https://api.github.com/users/sdboyer/repos","events_url":"https://api.github.com/users/sdboyer/events{/privacy}","received_events_url":"https://api.github.com/users/sdboyer/received_events","type":"User","site_admin":false},"created_at":"2017-04-10T17:12:48Z","updated_at":"2017-04-10T17:12:48Z","body":"Yes, this thread definitely introduces some new terms and injects some additional uncertainty into the system. I do have formal properties like these in my mind with respect to `dep`'s operation, but have not written them out. I'd say that doing so would be helpful.\r\n\r\nSome notes, some corrections:\r\n\r\n> At any point in time, it’s the code I have on disk that gives me a reproducible build or otherwise \r\n\r\n\"code I have on disk\" is a bit ambiguous - it could refer to, at minimum, either what's in a project's `vendor` or in `GOPATH` more generally. Either way, though, from the point of view of `dep`, this is _not_ what provides reproducibility - `lock.json` does. Committing your `vendor` dir also provides reproducibility, but that's outside `dep`'s scope of responsibility.\r\n\r\n> Version information, constraints etc are to my mind an additional level on top... a mapping onto repository concepts that ultimately resolve to the commits we see in the lock file.\r\n\r\nIf we're being precise here, it's better not to think of these as mappings onto \"repositories\", but a more generalized concept of a \"source\" that can produce N code trees, each of which is a version of a \"project.\"\r\n\r\nBut yes, the basic idea that what's in the manifest is \"an additional level on top\" is generally sound; the manifest is not strictly necessary for running a solve. That fact was largely what motivated @nathany to open this issue. \r\n\r\n> * Set A: ...\r\n> * Set B: ...\r\n\r\nYep, this is how they're defined in gps. The [`*rootdata.externalImportList()`](https://github.com/sdboyer/gps/blob/6db81554de4870d859883c6b6767eceadc22436c/rootdata.go#L53) call performs what amounts to the the union op.\r\n\r\n> Vendored packages are the subset V of D which are then found within d/vendor/ and \"controlled\" by a tool like dep\r\n> Let’s define the set O such that O = D - V. The set O is by definition found elsewhere within our environment and is not relevant for this discussion\r\n\r\n`dep` explicitly manages all of `vendor`, never a subset of it. By this definition, when system state is good, V = D, and O is the empty set. So, it's relevant to the extent that we consider the situation where O is non-empty to be indicative that `vendor` is out of sync with the lock. It's generally not a harmful situation, but it's still incorrect.\r\n\r\n> At any point in time there exists a possibly empty set Q of repositories that can be pruned from beneath d/vendor/, where { r ∈ Q | packagesIn(r) ⊆ E }, i.e. we have no dependencies on the packages contained within those repositories\r\n\r\nSome disambiguation here is needed - there is no valid exit state of `dep` in which Q will be non-empty. The only pruning work that may need to be done is the removal of E, which may be non-empty. (That's the focus of #120)\r\n\r\n> A discrepancy between lock.json and what we have on disk beneath d/vendor/:\r\n> * can only be determined by re-fetching the contents described by lock.json and seeing whether the resulting directory trees are identical or not\r\n\r\nIf the lock contains a hash digest of the code tree, then one-sided verification is possible. This is a useful property, and much of the focus of #121.\r\n\r\n> * can arise for a number of reasons (not trying to list them all here)\r\n\r\nThis is a bit ambiguous, but may be a key point of departure: while such things can happen, none of them are an output state of `dep`. That is, _there is no current `dep` command that can exit 0 and allow such a disk inconsistency to exist._ We may bend a little on this for particular circumstances, but this is a cornerstone of `dep`'s design. To that end:\r\n\r\n> A user is interested in various of the sets listed above, functions of those sets and manipulating these sets. The commands the dep exposes can be defined in terms of those sets.\r\n\r\nCan and should! But it should also be emphasized that `dep` is expressly _not_ a tool for arbitrary manipulation of these sets. Being opinionated is how `dep` can guarantee that, for example, Q and O are always empty, if the system is in a good state, and that it is always possible for `dep` to drive towards that good state during every command execution. That's a useful guarantee, because it is categorically unuseful for Q and O to be non-empty.\r\n\r\n> ```\r\n> # \"control\" a package we will import/have imported. This could (/should?) prompt \r\n> # us to ask whether we want to specify:\r\n> #\r\n> # * whether the commits of the transitive dependencies of github.com/pkg/errors \r\n> #   should also be controlled by dep\r\n> # * assuming yes, whether we want to specify the commit\r\n> #\r\n> ```\r\n> dep add github.com/pkg/errors@ff09b135c25aae272398c51a07235b90a75aa4f0\r\n> ```\r\n\r\nHaving a notion of \"control\" like this would change `dep` from an opt-out model (via `ignored` packages in the manifest) to an opt-in model, where you have to explicitly specify the deps the tool should be managing. This would dramatically increase the amount of maintenance work the user needs to perform, and I'm not seeing the upside to it. I'm sorry, but this is a non-starter. \r\n\r\n> ```\r\n> dep add github.com/pkg/errors@ff09b135c25aae272398c51a07235b90a75aa4f0\r\n> ```\r\n\r\nNow, there may be some benefit to an `add` command, separate from `ensure`, to make the specific case in #303 behave as users expect - but it does not entail a new property like \"control\" to achieve it. The `required` property exists in the manifest, and we can even make it work without explicitly writing that property to the manifest on disk - #36.\r\n\r\n> ```\r\n> # \"control\" a main package we will use at some point - notice the -p flag used\r\n> # to indicate it’s a program dependency, i.e.\r\n> #\r\n> dep add -p github.com/myitcv/gopherjs/cmd/reactGen@648bf1950ae20f0ad155e4faabc276252c7f3ff9\r\n> ```\r\n\r\n> Would such an approach help to pin down the different areas/concerns and make precise what we're trying to do with various commands, the separate areas of concern?\r\n\r\nYes, while these are the terms I think in, I do think it would benefit broader discussion to formalize them. As evidenced by my response here, though, I think we need to be cautious that in defining these terms, we do not get distracted by a sense that we have to give the user access to manipulate all the sets we can define."}}
{"actor":{"display_login":"sdboyer"}}