{"action":"created","issue":{"url":"https://api.github.com/repos/jordwalke/esy/issues/88","repository_url":"https://api.github.com/repos/jordwalke/esy","labels_url":"https://api.github.com/repos/jordwalke/esy/issues/88/labels{/name}","comments_url":"https://api.github.com/repos/jordwalke/esy/issues/88/comments","events_url":"https://api.github.com/repos/jordwalke/esy/issues/88/events","html_url":"https://github.com/jordwalke/esy/issues/88","id":214837910,"number":88,"title":"Environment for cross compiling packages","user":{"login":"wokalski","id":1693190,"avatar_url":"https://avatars3.githubusercontent.com/u/1693190?v=3","gravatar_id":"","url":"https://api.github.com/users/wokalski","html_url":"https://github.com/wokalski","followers_url":"https://api.github.com/users/wokalski/followers","following_url":"https://api.github.com/users/wokalski/following{/other_user}","gists_url":"https://api.github.com/users/wokalski/gists{/gist_id}","starred_url":"https://api.github.com/users/wokalski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wokalski/subscriptions","organizations_url":"https://api.github.com/users/wokalski/orgs","repos_url":"https://api.github.com/users/wokalski/repos","events_url":"https://api.github.com/users/wokalski/events{/privacy}","received_events_url":"https://api.github.com/users/wokalski/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-03-16T21:04:07Z","updated_at":"2017-03-17T07:42:15Z","closed_at":null,"body":"These are somewhat messy notes about cross compiling. Please note that I'm not fully confident about anything that's written below. Cross compiling in OCaml is a complex issue for several reasons.\r\n\r\n1. `opam` packages have arbitrary, platform dependent build scripts. Usually made to work on hosts.\r\n2. You need to use different compiler versions for different platforms and architectures\r\n3. You need to manage build artefacts for different platforms\r\n\r\nWhat's more, details will also differ depending on the host platform. The most obvious example that comes to my mind are fat binaries. On Mac we have wonderful support for them but they don't exist on other platforms. So in the Mac case, handling build artefacts could've been a bit easier. However, we cannot make any assumptions about platforms and the intricacies of targets make this task even harder.\r\n\r\nPlease note that I'm going to use word target (instead of architecture) because the product of compilation might be a fat binary containing slices for multiple architectures.\r\n\r\n### Workflow\r\n\r\nFor cross compilation we are going to need additional argument to `esy build`. What it does depends on the implementation but the task is on the intersection of package management, sandboxing, and building. Both build and runtime dependencies are different depending on the target.\r\n\r\n## Building opam packages\r\n\r\nExisting packages use arbitrary commands as the build phase. I've skimmed through some packages and they use variety of tools. I don't want to claim which ones are (not) popular, but I have seen the following: \r\n\r\n- OASIS\r\n- jbuilder\r\n- ocamlbuild\r\n\r\nThose tools in turn use `ocaml*`, C compiler, and probably other tools I'm not aware of.\r\n\r\nWhat I've written below is hard to believe to me since OCaml is highly portable. Therefore take it with a grain of salt.\r\n\r\n### Populating env vars\r\n\r\nIf we assume that most packages use just certain tools, let's say OCaml toolchain and C compilers we might try to create the right environment for a target. I have very low confidence in it though. I skimmed over `opam-cross-ios` and there are packages which need certain environment vars, but most of them also require changes to build commands.\r\n\r\nTo sum up, it's going to be hard to make it work. I would be very interested to hear an opinion from someone with good overview. If the premise that all those tools use similar setup behind the scenes is true, building a shell around `ocaml*` and C compiler might be an interesting solution. The scripts would forward commands from the build to concrete implementations with some adjustments to parameters and with right environment.\r\n\r\n### Package overrides\r\n\r\nHand written package overrides is a logical solution for those problems. `opam-cross-ios` is a *working* example of it. The changes to the packages usually consist of tweaks to build commands and sometimes patches to enable/disable target specific configs (in iOS case these are usually shared libraries). The pain here is that there's no easy way. Patching a package to a target is not very beginner friendly and time consuming.\r\n\r\n### Emulation\r\n\r\n@jordwalke suggested emulation as a possible solution. Technically  it could be possible. However, the cost of emulator, which is (afaik) very slow, eliminates the merits of native. We might as well compile it down to js and run in V8.\r\n\r\n### Structured build tool\r\n\r\nProvided that what I've written above is true another solution might be making esy work with a particular build tool. Reason toolchain is great because it's opinionated. What I feel is needed for cross compilation support is something more predictable than potentially as many ways to build something as packages there are. Of course, reinventing the wheel is the easiest thing to do, so treat this as a frivolous thought. @bsansouci started work on bsb native which might be a fit.\r\n\r\n---\r\n\r\nThere's a whole new category of problems and necessary consideration if we go beyond OCaml, but it's out of scope for this issue.\r\n\r\nAgain, it's hard to believe for me, that it's actually the way I described it so I will be delighted if I'm proven wrong.\r\n\r\nAll questions and thoughts are more than welcome."},"comment":{"url":"https://api.github.com/repos/jordwalke/esy/issues/comments/287287934","html_url":"https://github.com/jordwalke/esy/issues/88#issuecomment-287287934","issue_url":"https://api.github.com/repos/jordwalke/esy/issues/88","id":287287934,"user":{"login":"wokalski","id":1693190,"avatar_url":"https://avatars3.githubusercontent.com/u/1693190?v=3","gravatar_id":"","url":"https://api.github.com/users/wokalski","html_url":"https://github.com/wokalski","followers_url":"https://api.github.com/users/wokalski/followers","following_url":"https://api.github.com/users/wokalski/following{/other_user}","gists_url":"https://api.github.com/users/wokalski/gists{/gist_id}","starred_url":"https://api.github.com/users/wokalski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wokalski/subscriptions","organizations_url":"https://api.github.com/users/wokalski/orgs","repos_url":"https://api.github.com/users/wokalski/repos","events_url":"https://api.github.com/users/wokalski/events{/privacy}","received_events_url":"https://api.github.com/users/wokalski/received_events","type":"User","site_admin":false},"created_at":"2017-03-17T07:42:15Z","updated_at":"2017-03-17T07:42:15Z","body":"@jordwalke re emulation. Yes, it does simplify a bit. I can see three potential issues:\r\n\r\n1. Emulator needs to be able to emulate platform X.\r\n2. All dependencies of the whole build system need to be compatible with emulated host\r\n3. Forking/patching might still be needed when a package makes assumptions about host. Like shared libs and iOS in the example.\r\n\r\nThis is a very interesting solution in general, but it's not interesting in my context. By the time I make Apple ARM + iOS emulate on qemu (or something else) I would've forked all opam packages there are 😛."}}
{"actor":{"display_login":"wokalski"}}