{"action":"created","issue":{"url":"https://api.github.com/repos/hylang/hy/issues/1407","repository_url":"https://api.github.com/repos/hylang/hy","labels_url":"https://api.github.com/repos/hylang/hy/issues/1407/labels{/name}","comments_url":"https://api.github.com/repos/hylang/hy/issues/1407/comments","events_url":"https://api.github.com/repos/hylang/hy/issues/1407/events","html_url":"https://github.com/hylang/hy/issues/1407","id":253523307,"number":1407,"title":"Get rid of autoimports via new namespaces","user":{"login":"gilch","id":3346292,"avatar_url":"https://avatars3.githubusercontent.com/u/3346292?v=4","gravatar_id":"","url":"https://api.github.com/users/gilch","html_url":"https://github.com/gilch","followers_url":"https://api.github.com/users/gilch/followers","following_url":"https://api.github.com/users/gilch/following{/other_user}","gists_url":"https://api.github.com/users/gilch/gists{/gist_id}","starred_url":"https://api.github.com/users/gilch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gilch/subscriptions","organizations_url":"https://api.github.com/users/gilch/orgs","repos_url":"https://api.github.com/users/gilch/repos","events_url":"https://api.github.com/users/gilch/events{/privacy}","received_events_url":"https://api.github.com/users/gilch/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-08-29T04:03:45Z","updated_at":"2017-08-29T15:36:23Z","closed_at":null,"body":"> Namespaces are one honking great idea -- let's do more of those!\r\n> -- The Zen of Python\r\n\r\nAutoimports are causing us headaches. #1367, #791.\r\n\r\nMacros are also brittle due to lack of namespacing and hygene. #277. Even some compiler builtins have this problem--\r\n```Python\r\n=> ((fn [list] (eval `(print ~@list))) [\"Hy\" \"world!\"])\r\nfrom hy.core.language import eval\r\nfrom hy import HyExpression, HySymbol\r\n(lambda list: eval(HyExpression((([] + [HySymbol('print')]) + list((list or []))))))(['Hy', 'world!'])\r\nTraceback (most recent call last):\r\n  File \"c:\\users\\me\\documents\\github\\hy\\hy\\importer.py\", line 201, in hy_eval\r\n    return eval(ast_compile(expr, \"<eval>\", \"eval\"), namespace)\r\n  File \"<eval>\", line 1, in <module>\r\n  File \"<eval>\", line 1, in <lambda>\r\nTypeError: 'list' object is not callable\r\n```\r\nIt seems like that should work. I didn't even use any macros. Can you spot the error in the Python expansion?\r\n\r\nOne might be tempted to scoff and say, \"You should *never* use a builtin name as a local!\". That might work for Python, which has a fairly small number of builtins, but for Clojure, which has ~600 symbols in core, that's an *unreasonable cognitive burden* on the programmer. Hy isn't quite there yet, but it's still got a *much* bigger core than Python's builtins. The `name` builtin is already a problem. #525 Maybe restricting the names of special forms is okay, but we really need to be able to shadow the other core names with a local.\r\n\r\nClojure's namespacing may hold the answer. If the compiler had used `__import__(\"builtins\").list` instead of just `list` in its quasiquote expansion, the above would have worked properly. Clojure automatically namespaces symbols in its syntax-quote forms, so they work more like that.\r\n\r\nAn inline import is not a big deal in Python, since modules are cached. It's basically just an extra dict lookup--a fast action that you do all the time in Python code.\r\n\r\nUsing `__import__` directly is frowned upon in Python--`__import__` is kind of an implementation detail (but we could say the same of the AST itself). The recommendation is to use the import statement, like we're doing now. But that's harder to use than an expression, and requires an extra gensym, and doesn't play nice with `__future__`.  So if you need a more portable import expression, you're supposed to use the one from importlib. But how do we get to *that* without an autoimport? Is `__import__(\"importlib\").import_module(\"foo\").bar` any better than `__import__(\"foo\").bar`? Chicken/egg.\r\n\r\nThe solution to a chicken/egg problem is bootstrapping.\r\n### step 1 new builtin\r\nAs a start, I propose adding a `_#` object to Python's builtins module upon import of Hy. The `_` marks it \"private\" and the `#` makes it unlikely to interfere with any other Python use of the module, even by other libraries, since `#` is not allowed in Python identifiers. This object's class will override `getattribute` to make a dot access a module. Now you can refer to (e.g.) `list` as `_#.builtins.list` in macros, and it will work properly from any module, even in a context where `list` is shadowed.\r\n### step 2 no more autoimports\r\nWe can get rid of all autoimports in the compiler once this is available and we rewrite the compiler and macros to use these namespaced symbols instead of builtins or anything from core we're autoimporting now. This is a a bit less tedious than rewriting them to (e.g.) `__import__(\"builtins\").list` or `(. ~g!builtins list)` after adding `(import [builtins :as g!builtins])` to every macro. Unlike some complicated import expression, `_#.foo.bar` is just a `HySymbol`, and can be treated like one by macros. \r\n### step 3 upgrade quasiquote to syntax-quote\r\nBut it could be automated further like Clojure does in its syntax quote. This way, the compiler would insert the namespace prefix for you. You don't have to write it, but it comes out in the expansion anyway. \r\n\r\nThere could be some namespace macro that alters the compiler state, maybe by setting a `_#.hy.core.ns` object, which would map abbreviations to their expansions, e.g. `{\"list\": HySymbol(\"_#.builtins.list\"), ...}`. We could make the `def` form add the mapping to the current namespace. #911. This way you don't have to build the mapping either. You just set the namespace, and `def` builds it for you as you go. `setv` wouldn't alter the namespace, so adding the abbreviation is optional. There would also be a way to include all the mappings from one namespace in another. You'd often include `builtins` and `hy.core` in a custom namespace, for example.\r\n### step 4 special variables\r\nWe could also use the namespace system to help implement special variables. #1089. We'd update the `def` form to make special variables instead of the normal kind. A dot lookup would go through a special `__getattr__` method that dereferences a dynamic variable, instead of returning the Var object itself. This means that you'd have to spell out the prefix when using them outside of a syntax quote.\r\n\r\nWe can start using special variables to configure things like other Lisps do, like changing how things print at the repl.\r\n### step 5 module-level namespacing\r\nMaking the compiler add the prefix to an abbreviation *outside* of a syntax quote seems like a bad idea. There'd be no way to shadow it with a local. That's not how special variables work in Common Lisp or Clojure.\r\n\r\nSince we want to be able to shadow abbreviations, they need to act like globals, not symbol macros. For most things, it's enough to simply add it to the module dict. (e.g. `(import [foo [*]])` But that's not good enough for a dynamic variable, which needs the dereference magic that lives in the dot access. At the repl, you could replace the `__builtins__` module with an object that does the dereference magic, since if value not found in the `globals()` dict, Python will look for it in `__builtins__`. But this doesn't work in modules, which are executed all at once on import, instead of incrementally. By the time you try to set `__builtins__` it's already fixed.\r\n\r\nThe way around this is to `exec` the code or bytecode on module load using a custom `globals` object that has the magic in its `__getitem__`. This way, an ordinary global lookup can dereference the special variable, even without a dot, even outside a syntax quote. And it can still be shadowed by a local.\r\n### step 6 finally, a sensible let alternative\r\nMake an `_#.auto` object that overrides `__setattr__` to create a Var object when you attempt to bind one with that prefix that doesn't exist yet.\r\n\r\n The `binding` form implemented in **step 4** requires a prefixed name. In **step 5** it would also accept an abbreviation from the currently active namespace. Now, if there's no prefix, and no abbreviation, `_#.auto` is assumed instead. Now you can use `bindings` like a dynamic-scoped `let`. These `bindings` forms seem to do most of we want from `let` short of lexical closures. The binding get released at the end of the last form. You can nest them. You can use short names, like `x` and `y`. You can pass them to functions, if the call is made *before* the binding ends, and they'll get a lexical argument.\r\n### conclusion\r\nThat's a big plan with a lot of steps. Some of the details are tentative. I'm not going to put all of this in one PR. That's too much. But you might not like the changes until they're finished, so you need to know where I'm going with this.\r\n\r\nI'd also like to get general approval of this plan before I start putting in the actual work. Does this seem like a good direction? Any parts that aren't clear? Is there anything you'd want changed?"},"comment":{"url":"https://api.github.com/repos/hylang/hy/issues/comments/325703435","html_url":"https://github.com/hylang/hy/issues/1407#issuecomment-325703435","issue_url":"https://api.github.com/repos/hylang/hy/issues/1407","id":325703435,"user":{"login":"Kodiologist","id":322730,"avatar_url":"https://avatars3.githubusercontent.com/u/322730?v=4","gravatar_id":"","url":"https://api.github.com/users/Kodiologist","html_url":"https://github.com/Kodiologist","followers_url":"https://api.github.com/users/Kodiologist/followers","following_url":"https://api.github.com/users/Kodiologist/following{/other_user}","gists_url":"https://api.github.com/users/Kodiologist/gists{/gist_id}","starred_url":"https://api.github.com/users/Kodiologist/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Kodiologist/subscriptions","organizations_url":"https://api.github.com/users/Kodiologist/orgs","repos_url":"https://api.github.com/users/Kodiologist/repos","events_url":"https://api.github.com/users/Kodiologist/events{/privacy}","received_events_url":"https://api.github.com/users/Kodiologist/received_events","type":"User","site_admin":false},"created_at":"2017-08-29T15:36:23Z","updated_at":"2017-08-29T15:36:23Z","body":"The most straightforward solution to auto-importing woes is to replace the current auto-importing magic with `from hy.core.language import *` at the start of the REPL and the top of each source file. Then Hy core functions would work very much like Python builtins.\r\n\r\nThings like dynamically scoped variables and protection against shadowing of core function names in macro expansions would be nice. But it's hard for me to guess in advance whether your plan would work and what the side-effects would be."}}
{"actor":{"display_login":"Kodiologist"}}