{"action":"created","issue":{"url":"https://api.github.com/repos/nengo/nengo/issues/1301","repository_url":"https://api.github.com/repos/nengo/nengo","labels_url":"https://api.github.com/repos/nengo/nengo/issues/1301/labels{/name}","comments_url":"https://api.github.com/repos/nengo/nengo/issues/1301/comments","events_url":"https://api.github.com/repos/nengo/nengo/issues/1301/events","html_url":"https://github.com/nengo/nengo/issues/1301","id":222799508,"number":1301,"title":"nengo.utils and nengo_extras","user":{"login":"tbekolay","id":203709,"avatar_url":"https://avatars1.githubusercontent.com/u/203709?v=3","gravatar_id":"","url":"https://api.github.com/users/tbekolay","html_url":"https://github.com/tbekolay","followers_url":"https://api.github.com/users/tbekolay/followers","following_url":"https://api.github.com/users/tbekolay/following{/other_user}","gists_url":"https://api.github.com/users/tbekolay/gists{/gist_id}","starred_url":"https://api.github.com/users/tbekolay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tbekolay/subscriptions","organizations_url":"https://api.github.com/users/tbekolay/orgs","repos_url":"https://api.github.com/users/tbekolay/repos","events_url":"https://api.github.com/users/tbekolay/events{/privacy}","received_events_url":"https://api.github.com/users/tbekolay/received_events","type":"User","site_admin":false},"labels":[{"id":86085168,"url":"https://api.github.com/repos/nengo/nengo/labels/discussion","name":"discussion","color":"207de5","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":22,"created_at":"2017-04-19T16:42:16Z","updated_at":"2017-04-20T17:31:50Z","closed_at":null,"body":"At the last dev meeting, we discussed what should be contained in the Nengo API, `nengo.utils` and `nengo_extras`. To this point, we've been doing this on a case-by-case basis, which has worked for the most part. However, I can understand why people would be frustrated when a review will say \"this should go in nengo_extras,\" so it would be good to be more clear how to make those decisions.\r\n\r\nTo kick off that discussion, I'll put my thoughts down in this initial comment. Hopefully there will be some followup discussion on it; when the discussion dies down, I'll summarize and put those summaries in the documentation (in `docs/frontend_api.rst`, `nengo/utils/__init__.py`, and `nengo_extras/README.rst`) and make the appropriate changes.\r\n\r\n## The Nengo API\r\n\r\nTo start off, it's worth being clear about the Nengo API. At the moment, Nengo Core contains two APIs, the frontend API and the backend API. Only the frontend API is \"official\" (until I refactor the backend API), in the sense that backwards incompatible and compatible changes to the frontend API are what drive the major and minor version numbers up.\r\n\r\nThe Nengo API is defined by what is in [this document](http://pythonhosted.org/nengo/frontend_api.html). Since that's pretty long, it's perhaps more clear to look at [the ReST source instead](http://pythonhosted.org/nengo/_sources/frontend_api.rst.txt) for what the API contains.\r\n\r\nSome parts of the API are straightforward: you definitely need the Nengo objects, and they need the distributions, the base `NeuronType` and `LearningRuleType`, and so on. \r\n\r\nOther parts are not so obvious: we include 8 neurons types, even though the API does not technically need to provide any neurons types, only the base `NeuronType` class. It's reasonable to think that we should just include the most common neuron type, `LIF`. However, we include these additional neuron types in Nengo core for several reasons:\r\n\r\n1. Some are are commonly used (`Direct`, `RectifiedLinear`)\r\n2. History (Nengo 1.4 provided some of these types)\r\n3. Marketing (it's much easier to claim that Nengo works with lots of neuron types if we include a bunch)\r\n4. Low effort (maintaining a neuron type is not much work)\r\n\r\nSimilar arguments can be made for other parts of the Nengo API that are not strictly necessary. The important takeaway is that the Nengo API is _not_ a minimal skeleton that is extended by other projects (as occurs with a plugin architecture). Minimalism is not really the goal of the API; we're looking to strike a balance, covering a large percentage of use cases with a small, stable, and maintainable code base. There are things in there that don't need to be there, but unless they are affecting the stability or maintainability of the code base, it's more effort to remove them than to leave them there.\r\n\r\n## nengo.utils\r\n\r\nI fought against having a `utils.py` file or a `utils` folder for a while, but in my old age I realize that \"utilities\" are a necessary evil for any sufficiently large software project. Typically, utilities are little bits of code that end up being used in many different places, but don't naturally live in any of those places, so they get relegated to a utils file or folder to be used wherever appropriate.\r\n\r\nUnfortunately, the presence of a `utils` file or folder means that it inevitably accumulates things over time. In addition to the description above, we also ended up using the utils folder for lesser-used features that users might want (e.g., [`probe_all`](https://github.com/nengo/nengo/blob/master/nengo/utils/probe.py#L5)). This made sense before we made `nengo_extras`, but I think `nengo_extras` is a better place for things that we expect modelers to use.\r\n\r\nTo crystallize those thoughts into a few concrete rules, I propose that we use `nengo.utils` _only_ for internal utilities used _by_ the Nengo API, and not for helpful utility functions that _use_ the Nengo API. I would further propose that they not be considered part of the API, and therefore can be freely moved and changed as makes sense for the API (without requiring a minor/major version bump). This means that other projects can import and use them if they choose, but they should not rely on them never changing. While this might seem objectionable to those that are maintaining other projects, I think in practice it won't be that difficult to keep up with changes to utils.\r\n\r\n## nengo_extras\r\n\r\n[nengo_extras](https://github.com/nengo/nengo_extras/) was created as a place to put a growing number of things that were being developed using the Nengo API, but didn't seem to be so commonplace that they made sense to include in the API (and therefore incur stability and maintainability costs). Currently it contains some networks, neuron types, distributions, and utilities for downloading data sets from various locations on the internet.\r\n\r\nCurrently, nengo_extras is a separate repository that hasn't been officially released, though it is being used by a few people. It can be installed using pip. It's relatively low activity, so I haven't put in the time to get it organized, documented, and released on PyPI, but if we're moving some of the things from `nengo.utils` to `nengo_extras` then that should be done before the move is finalized.\r\n\r\nThere are two alternatives to having a separate `nengo_extras` repo which might work for us:\r\n\r\n1. Move `nengo_extras` to a `nengo_extras` folder in the Nengo core repo.\r\n\r\n   This folder would be installed when you `pip install nengo`, and would be importable with `import nengo_extras` (many packages do this; if you have pytest installed, for example, you can open an interpreter and do `import pytest` and `import _pytest`, both of which are installed when you do `pip install pytest`). \r\n\r\n    Since `nengo_extras` would be a separate folder, it would limit the visual noise when looking through the source files in the `nengo` folder.\r\n\r\n2. Move `nengo_extras` to `nengo/extras` in the Nengo core library.\r\n\r\n   This shares some of the benefits of 1., but is more familiar to those poking around the source tree. However, it ups the number of files in the `nengo` directory a fair bit.\r\n\r\nThe downsides of both of these approaches, compared to the separate repo, are that they are tricky to version separately, and they will not have a separate issue / pull request tracker. The upside is that they would not require an additional `pip install` step when users find they want to use something in `nengo_extras`, and the Nengo core examples can use `nengo_extras` freely.\r\n\r\n---\r\n\r\nTo sum up the above: `nengo_extras` should contain things that _use_ the Nengo API. `nengo.utils` should contain things are _used by_ the Nengo API.\r\n\r\nIn order to get that done, we need to make some decisions about what to do with `nengo_extras` and what to do about the parts of `nengo.utils` that are currently used in examples and other places. Specifically, the things that should be moved are:\r\n\r\n- `nengo.utils.ensemble.tuning_curves`\r\n- `nengo.utils.ensemble.response_curves`\r\n- `nengo.utils.ensemble.sorted_neurons`\r\n- `nengo.utils.functions.piecewise`\r\n- `nengo.utils.functions.HilbertCurve`\r\n- `nengo.utils.least_squares_solvers`\r\n- `nengo.utils.matplotlib.implot`\r\n- `nengo.utils.matplotlib.rasterplot`\r\n- `nengo.utils.network.with_self`\r\n- `nengo.utils.network.activate_direct_mode`\r\n- `nengo.utils.paths`\r\n- `nengo.utils.progress`\r\n\r\nI think that most of these will end up finding better locations in the frontend API, or be otherwise refactored (#1187 -- which is still open -- already moves utilities that make sense in `nengo_extras`.) However, let's not clutter this discussion with thoughts about what to do about these specific instances, and instead focus on the general gameplan for `nengo.utils` and `nengo_extras`."},"comment":{"url":"https://api.github.com/repos/nengo/nengo/issues/comments/295826173","html_url":"https://github.com/nengo/nengo/issues/1301#issuecomment-295826173","issue_url":"https://api.github.com/repos/nengo/nengo/issues/1301","id":295826173,"user":{"login":"tbekolay","id":203709,"avatar_url":"https://avatars1.githubusercontent.com/u/203709?v=3","gravatar_id":"","url":"https://api.github.com/users/tbekolay","html_url":"https://github.com/tbekolay","followers_url":"https://api.github.com/users/tbekolay/followers","following_url":"https://api.github.com/users/tbekolay/following{/other_user}","gists_url":"https://api.github.com/users/tbekolay/gists{/gist_id}","starred_url":"https://api.github.com/users/tbekolay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tbekolay/subscriptions","organizations_url":"https://api.github.com/users/tbekolay/orgs","repos_url":"https://api.github.com/users/tbekolay/repos","events_url":"https://api.github.com/users/tbekolay/events{/privacy}","received_events_url":"https://api.github.com/users/tbekolay/received_events","type":"User","site_admin":false},"created_at":"2017-04-20T17:31:50Z","updated_at":"2017-04-20T17:31:50Z","body":"> So it feels to me that the cognitive load on the user is much smaller if there are just a collection of utility functions in nengo.utils.\r\n\r\nI'm of the opinion that putting network-related functions in `nengo.network` is easier to remember, so they won't have to refer to the docs as often. I think this is likely not something either of us are going to change our minds on, so would be good to get some other voices in the mix to move the discussion forward."}}
{"actor":{"display_login":"tbekolay"}}