{"action":"created","issue":{"url":"https://api.github.com/repos/dfdx/Spark.jl/issues/15","repository_url":"https://api.github.com/repos/dfdx/Spark.jl","labels_url":"https://api.github.com/repos/dfdx/Spark.jl/issues/15/labels{/name}","comments_url":"https://api.github.com/repos/dfdx/Spark.jl/issues/15/comments","events_url":"https://api.github.com/repos/dfdx/Spark.jl/issues/15/events","html_url":"https://github.com/dfdx/Spark.jl/pull/15","id":216761288,"number":15,"title":"Add Broadcast macro","user":{"login":"JohnAdders","id":16005222,"avatar_url":"https://avatars1.githubusercontent.com/u/16005222?v=3","gravatar_id":"","url":"https://api.github.com/users/JohnAdders","html_url":"https://github.com/JohnAdders","followers_url":"https://api.github.com/users/JohnAdders/followers","following_url":"https://api.github.com/users/JohnAdders/following{/other_user}","gists_url":"https://api.github.com/users/JohnAdders/gists{/gist_id}","starred_url":"https://api.github.com/users/JohnAdders/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JohnAdders/subscriptions","organizations_url":"https://api.github.com/users/JohnAdders/orgs","repos_url":"https://api.github.com/users/JohnAdders/repos","events_url":"https://api.github.com/users/JohnAdders/events{/privacy}","received_events_url":"https://api.github.com/users/JohnAdders/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2017-03-24T11:53:43Z","updated_at":"2017-03-26T12:07:18Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/dfdx/Spark.jl/pulls/15","html_url":"https://github.com/dfdx/Spark.jl/pull/15","diff_url":"https://github.com/dfdx/Spark.jl/pull/15.diff","patch_url":"https://github.com/dfdx/Spark.jl/pull/15.patch"},"body":"Added very preliminary version of broadcast to get variables onto workers"},"comment":{"url":"https://api.github.com/repos/dfdx/Spark.jl/issues/comments/289278947","html_url":"https://github.com/dfdx/Spark.jl/pull/15#issuecomment-289278947","issue_url":"https://api.github.com/repos/dfdx/Spark.jl/issues/15","id":289278947,"user":{"login":"dfdx","id":428171,"avatar_url":"https://avatars1.githubusercontent.com/u/428171?v=3","gravatar_id":"","url":"https://api.github.com/users/dfdx","html_url":"https://github.com/dfdx","followers_url":"https://api.github.com/users/dfdx/followers","following_url":"https://api.github.com/users/dfdx/following{/other_user}","gists_url":"https://api.github.com/users/dfdx/gists{/gist_id}","starred_url":"https://api.github.com/users/dfdx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dfdx/subscriptions","organizations_url":"https://api.github.com/users/dfdx/orgs","repos_url":"https://api.github.com/users/dfdx/repos","events_url":"https://api.github.com/users/dfdx/events{/privacy}","received_events_url":"https://api.github.com/users/dfdx/received_events","type":"User","site_admin":false},"created_at":"2017-03-26T12:07:18Z","updated_at":"2017-03-26T12:07:18Z","body":"Ok, maybe we should take a step back and look at the bigger picture. \r\n\r\nFirst of all, `share_variable` only makes sense for passing data that takes long to generateI. In your tests, you share variable `a` which is a constant integer, but you could have equally written: \r\n\r\n```\r\n@attach a = 2\r\n```\r\n(haven't tested it, but if it doesn't work, it's a bug)\r\n\r\nAt the same time it's unreasonable to write:\r\n\r\n```\r\n@attach a = long_running_functions()\r\n```\r\nbecause in this case `long_running_function` would be executed at each worker,  not saying that it may refer to data local to the driver process. So we want to serialize the _data_ in `a` and simply load it on workers.\r\n\r\nAs I understand it, you do so by distributing a data file to workers and attaching a code to read it on workers. This is a good idea, but so far I don't see why `share_variable` should be a macro. E.g. you could do it with something like this instead:\r\n\r\n```\r\nfunction share_variable(sc::SparkContext, name::Symbol, data::Any)\r\n    temp_filename = generate_temp_filename(\"sharevar__\", \".jbin\")\r\n    path = joinpath(get_temp_dir(sc), temp_filename)\r\n    open(path, \"w\") do io\r\n        serialize(io, data)\r\n    end\r\n    add_file(sc, convert_to_uri(path))\r\n    ex = quote\r\n            open($temp_filename, \"r\") do io\r\n                global $name = deserialize(io)\r\n            end\r\n         end\r\n    save_attachment(ex)\r\nend\r\n\r\n...\r\n\r\nshare_variable(sc, :a, a)\r\n```\r\nThis is the most straightforward way I can think of to achieve the goal. Note that there are no macros or string parsing. However, you can wrap this function into a more convenient macro call, e.g.:\r\n\r\n```\r\n@share a = long_running_process()\r\n```\r\nor \r\n```\r\na = long_running_process()\r\n@share a\r\n```\r\nIn this case, `@share` macro should:\r\n\r\n1. Parse variable name.\r\n2. Evaluate it on the current module (no `esc`) to obtain data.\r\n3. Call `share_variable` function with parsed var name and data.\r\n\r\nTo summarize: \r\n\r\n * code generation doesn't have to happen in a macro\r\n * macros aren't restricted to call  only other macros, but can  equally call functions\r\n * string evaluation is rarely needed"}}
{"actor":{"display_login":"dfdx"}}