{"action":"created","issue":{"url":"https://api.github.com/repos/nodemcu/nodemcu-firmware/issues/2068","repository_url":"https://api.github.com/repos/nodemcu/nodemcu-firmware","labels_url":"https://api.github.com/repos/nodemcu/nodemcu-firmware/issues/2068/labels{/name}","comments_url":"https://api.github.com/repos/nodemcu/nodemcu-firmware/issues/2068/comments","events_url":"https://api.github.com/repos/nodemcu/nodemcu-firmware/issues/2068/events","html_url":"https://github.com/nodemcu/nodemcu-firmware/issues/2068","id":248424853,"number":2068,"title":"Policy of supporting Lua in ROM","user":{"login":"TerryE","id":690173,"avatar_url":"https://avatars3.githubusercontent.com/u/690173?v=4","gravatar_id":"","url":"https://api.github.com/users/TerryE","html_url":"https://github.com/TerryE","followers_url":"https://api.github.com/users/TerryE/followers","following_url":"https://api.github.com/users/TerryE/following{/other_user}","gists_url":"https://api.github.com/users/TerryE/gists{/gist_id}","starred_url":"https://api.github.com/users/TerryE/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TerryE/subscriptions","organizations_url":"https://api.github.com/users/TerryE/orgs","repos_url":"https://api.github.com/users/TerryE/repos","events_url":"https://api.github.com/users/TerryE/events{/privacy}","received_events_url":"https://api.github.com/users/TerryE/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2017-08-07T14:14:34Z","updated_at":"2017-08-08T17:05:53Z","closed_at":null,"body":"Although this issue links to earlier discussions in #1289 and #1661, I see this as a  _policy_ issue mainly for the committers, so can you all read this and give your comments so we can move forward on the basis of some form of consensus?\r\n\r\nOf the ~45Kb RAM available on the ESP8266, typically half or more of this RAM is Lua compiled code and constant data as opposed to true R/W data.   The facility to move Lua binary code in to Flash _**will more than double the effective RAM available to programmers**_.\r\n\r\n### Do we add support for running Lua directly out of Flash?\r\n\r\n### If so do we add it to the current dev branch soon?\r\n\r\n## Background \r\n\r\nA hierarchy of function prototypes. and their associated vectors (constants, instructions, meta data for debug) are loaded into RAM when any Lua source or `lc` file is loaded into memory.  Because in the Lua architecture, each `Proto` hierarchy can be bound to multiple closures (this closure creation is only done by executing the `CLOSURE` statement at runtime), such hierarchies are intrinsically read-only and therefore in principle ROMable.\r\n\r\nThe main complication here is that, like all other Lua resources, `Proto` hierarchies are garbage-collectable (and advanced Lua programmers exploit this collection).  So IMO, the difficulties arise when devising the details of how any compiled Lua in ROM interacts nicely and stably with the GC: it's fairly straightforward to implement a scheme which work _mostly_: but we need one which works _all of the time_ in a well determined manner if we proceed with this.\r\n\r\nI haven't worked out a robust way of doing an incremental storage system, as Phil discusses in #128, and IMO this will be hard to realise.   What I have worked out how to do is essentially an \"freeze into flash, then reboot\" approach.  \r\n\r\n-  Essentially what this does is to maintain a ROM string table and a set of Proto hierarchies in a fixed (64Kb) flash block within the ICACHE_FLASH address space.   \r\n-  Any Lua files that you want to move into ROM must be preloaded into SPIFFS as `lua` or `lc` files.\r\n-  The flash block can be discarded and rebuilt using a new `node.rebuild_flash()` function supplying a list of lua files that you wanted including into the ROM.  This `rebuild_flash` call should be preferably called just after reboot.  This call then _either_ rebuilds flash block and reboots the ESP immediately on completion, _or_ leaves the flash block unchanged and reboots with an error status.\r\n-  After the reboot, the modules in the flash block are in the `require` path and so can be executed by a `require` statement; the `loadfile` and `dofile` will also parse `rom:module` syntax and return or execute a closure accordingly.\r\n\r\n## Basic approach\r\n\r\n-  This ROM-base lua system uses two string tables as discussed in #1661: the standard RAM-based table and a ROM-based one ahead of it in the search order. The `rebuild_flash` routine unhooks the current ROM table, and does two passes of loading the modules.\r\n-  The first pass is a dummy load.  This serves to purpose: (i) to calculate the total storage requirement for the `Proto hierarchies`; (ii) to fill the RAM string table with all the strings needed to store the hierarchies.\r\n-  The size of the string table and all string resources is then calculated and if the total size of code and string table fit within the flash block, then the strings and string table are copied to flash, then the RAM string table GC pruned. The ROM table is rehooked and the RAM table replaced with an empty table.\r\n-  The second pass is a flash buffer load, with all string constants resolved against the ROM table.  Since all of the strings needed to load the `Proto` hierarchies are now in ROM, these hierarchies can now persist over reboot, and only the closure-based resources will occupy RAM.\r\n\r\nThis process is simple and robust, but the Lua RTS is built around the assumption that collectable objects don't move their location and that strings are interned.  It will be impossible to return control to the invoking Lua after a successful load, and difficult to return control after a failed one, which is why this \"reload flash and immediately reboot\" option is the most robust.  \r\n\r\nThis system would enable Lua programmers to be able to compile and execute significantly larger Lua programs within  the ESP resources.\r\n\r\nThere are some extra wrinkles for the Lua 5.3 environment but I will park these for now.  So comments so far?"},"comment":{"url":"https://api.github.com/repos/nodemcu/nodemcu-firmware/issues/comments/321019639","html_url":"https://github.com/nodemcu/nodemcu-firmware/issues/2068#issuecomment-321019639","issue_url":"https://api.github.com/repos/nodemcu/nodemcu-firmware/issues/2068","id":321019639,"user":{"login":"nwf","id":156882,"avatar_url":"https://avatars3.githubusercontent.com/u/156882?v=4","gravatar_id":"","url":"https://api.github.com/users/nwf","html_url":"https://github.com/nwf","followers_url":"https://api.github.com/users/nwf/followers","following_url":"https://api.github.com/users/nwf/following{/other_user}","gists_url":"https://api.github.com/users/nwf/gists{/gist_id}","starred_url":"https://api.github.com/users/nwf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nwf/subscriptions","organizations_url":"https://api.github.com/users/nwf/orgs","repos_url":"https://api.github.com/users/nwf/repos","events_url":"https://api.github.com/users/nwf/events{/privacy}","received_events_url":"https://api.github.com/users/nwf/received_events","type":"User","site_admin":false},"created_at":"2017-08-08T17:05:53Z","updated_at":"2017-08-08T17:05:53Z","body":"Thanks much for the very detailed response!"}}
{"actor":{"display_login":"nwf"}}