{"action":"created","issue":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/21953","repository_url":"https://api.github.com/repos/JuliaLang/julia","labels_url":"https://api.github.com/repos/JuliaLang/julia/issues/21953/labels{/name}","comments_url":"https://api.github.com/repos/JuliaLang/julia/issues/21953/comments","events_url":"https://api.github.com/repos/JuliaLang/julia/issues/21953/events","html_url":"https://github.com/JuliaLang/julia/issues/21953","id":229810007,"number":21953,"title":"Different remote copying behaviour with one process vs multiple","user":{"login":"iamed2","id":3131001,"avatar_url":"https://avatars0.githubusercontent.com/u/3131001?v=3","gravatar_id":"","url":"https://api.github.com/users/iamed2","html_url":"https://github.com/iamed2","followers_url":"https://api.github.com/users/iamed2/followers","following_url":"https://api.github.com/users/iamed2/following{/other_user}","gists_url":"https://api.github.com/users/iamed2/gists{/gist_id}","starred_url":"https://api.github.com/users/iamed2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/iamed2/subscriptions","organizations_url":"https://api.github.com/users/iamed2/orgs","repos_url":"https://api.github.com/users/iamed2/repos","events_url":"https://api.github.com/users/iamed2/events{/privacy}","received_events_url":"https://api.github.com/users/iamed2/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-05-18T21:28:22Z","updated_at":"2017-05-20T04:08:31Z","closed_at":null,"body":"```julia\r\nusing Base.Test\r\n\r\nfunction main()\r\n    @testset begin\r\n        rng = MersenneTwister(3742)\r\n        numbers = pmap(1:1000) do _\r\n            rand(rng)\r\n        end\r\n\r\n        @test all(numbers .== numbers[1])\r\n\r\n        spawn_numbers = [(@fetch rand(rng)) for i = 1:1000]\r\n\r\n        @test all(spawn_numbers .== spawn_numbers[1])\r\n\r\n        pfor_numbers = @parallel vcat for i = 1:1000\r\n            [rand(rng)]\r\n        end\r\n\r\n        @test all(pfor_numbers .== pfor_numbers[1])\r\n    end\r\nend\r\n\r\nmain()\r\n```\r\n\r\nThere are three cases here to compare different behaviours with the different parallel methods but I really just care about the `@fetch`/`@spawn` case.\r\n\r\n```\r\nericdavies@whitacre> julia -p 1 ~/rng_parallel_test.jl\r\ntest set: Test Failed\r\n  Expression: all(pfor_numbers .== pfor_numbers[1])\r\n in record(::Base.Test.DefaultTestSet, ::Base.Test.Fail) at test.jl:431\r\n in do_test(::Base.Test.Returned, ::Expr) at test.jl:281\r\n in macro expansion at rng_parallel_test.jl:20 [inlined]\r\n in macro expansion at test.jl:674 [inlined]\r\n in main() at rng_parallel_test.jl:4\r\n in include_from_node1(::String) at loading.jl:488\r\n in include_from_node1(::String) at sys.dylib:?\r\n in process_options(::Base.JLOptions) at client.jl:265\r\n in _start() at client.jl:321\r\n in _start() at sys.dylib:?\r\nTest Summary: | Pass  Fail  Total\r\n  test set    |    2     1      3\r\nERROR: LoadError: Some tests did not pass: 2 passed, 1 failed, 0 errored, 0 broken.\r\n in finish(::Base.Test.DefaultTestSet) at test.jl:498\r\n in macro expansion at test.jl:681 [inlined]\r\n in main() at rng_parallel_test.jl:4\r\n in include_from_node1(::String) at loading.jl:488\r\n in include_from_node1(::String) at sys.dylib:?\r\n in process_options(::Base.JLOptions) at client.jl:265\r\n in _start() at client.jl:321\r\n in _start() at sys.dylib:?\r\nwhile loading /Users/ericdavies/rng_parallel_test.jl, in expression starting on line 24\r\n```\r\n\r\n```\r\nericdavies@whitacre> julia ~/rng_parallel_test.jl\r\ntest set: Test Failed\r\n  Expression: all(numbers .== numbers[1])\r\n in record(::Base.Test.DefaultTestSet, ::Base.Test.Fail) at test.jl:431\r\n in do_test(::Base.Test.Returned, ::Expr) at test.jl:281\r\n in macro expansion at rng_parallel_test.jl:10 [inlined]\r\n in macro expansion at test.jl:674 [inlined]\r\n in main() at rng_parallel_test.jl:4\r\n in include_from_node1(::String) at loading.jl:488\r\n in include_from_node1(::String) at sys.dylib:?\r\n in process_options(::Base.JLOptions) at client.jl:265\r\n in _start() at client.jl:321\r\n in _start() at sys.dylib:?\r\ntest set: Test Failed\r\n  Expression: all(spawn_numbers .== spawn_numbers[1])\r\n in record(::Base.Test.DefaultTestSet, ::Base.Test.Fail) at test.jl:431\r\n in do_test(::Base.Test.Returned, ::Expr) at test.jl:281\r\n in macro expansion at rng_parallel_test.jl:14 [inlined]\r\n in macro expansion at test.jl:674 [inlined]\r\n in main() at rng_parallel_test.jl:4\r\n in include_from_node1(::String) at loading.jl:488\r\n in include_from_node1(::String) at sys.dylib:?\r\n in process_options(::Base.JLOptions) at client.jl:265\r\n in _start() at client.jl:321\r\n in _start() at sys.dylib:?\r\ntest set: Test Failed\r\n  Expression: all(pfor_numbers .== pfor_numbers[1])\r\n in record(::Base.Test.DefaultTestSet, ::Base.Test.Fail) at test.jl:431\r\n in do_test(::Base.Test.Returned, ::Expr) at test.jl:281\r\n in macro expansion at rng_parallel_test.jl:20 [inlined]\r\n in macro expansion at test.jl:674 [inlined]\r\n in main() at rng_parallel_test.jl:4\r\n in include_from_node1(::String) at loading.jl:488\r\n in include_from_node1(::String) at sys.dylib:?\r\n in process_options(::Base.JLOptions) at client.jl:265\r\n in _start() at client.jl:321\r\n in _start() at sys.dylib:?\r\nTest Summary: | Fail  Total\r\n  test set    |    3      3\r\nERROR: LoadError: Some tests did not pass: 0 passed, 3 failed, 0 errored, 0 broken.\r\n in finish(::Base.Test.DefaultTestSet) at test.jl:498\r\n in macro expansion at test.jl:681 [inlined]\r\n in main() at rng_parallel_test.jl:4\r\n in include_from_node1(::String) at loading.jl:488\r\n in include_from_node1(::String) at sys.dylib:?\r\n in process_options(::Base.JLOptions) at client.jl:265\r\n in _start() at client.jl:321\r\n in _start() at sys.dylib:?\r\nwhile loading /Users/ericdavies/rng_parallel_test.jl, in expression starting on line 24\r\n```\r\n\r\nWhen there are one or more additional processes, `rng` is copied. When there is only one process, it is not.\r\n\r\nThe reason the `@parallel for` case fails both times is due to the use of `CachingPool`, which may also be used for `pmap` in the future via https://github.com/JuliaLang/julia/issues/21946.\r\n\r\nIdeally I would like the behaviours to be consistent (though I don't need them to be deterministic). "},"comment":{"url":"https://api.github.com/repos/JuliaLang/julia/issues/comments/302849128","html_url":"https://github.com/JuliaLang/julia/issues/21953#issuecomment-302849128","issue_url":"https://api.github.com/repos/JuliaLang/julia/issues/21953","id":302849128,"user":{"login":"amitmurthy","id":117461,"avatar_url":"https://avatars2.githubusercontent.com/u/117461?v=3","gravatar_id":"","url":"https://api.github.com/users/amitmurthy","html_url":"https://github.com/amitmurthy","followers_url":"https://api.github.com/users/amitmurthy/followers","following_url":"https://api.github.com/users/amitmurthy/following{/other_user}","gists_url":"https://api.github.com/users/amitmurthy/gists{/gist_id}","starred_url":"https://api.github.com/users/amitmurthy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amitmurthy/subscriptions","organizations_url":"https://api.github.com/users/amitmurthy/orgs","repos_url":"https://api.github.com/users/amitmurthy/repos","events_url":"https://api.github.com/users/amitmurthy/events{/privacy}","received_events_url":"https://api.github.com/users/amitmurthy/received_events","type":"User","site_admin":false},"created_at":"2017-05-20T04:08:31Z","updated_at":"2017-05-20T04:08:31Z","body":"For performance reasons remote calls executing locally short-circuit the entire serialization-deserialization cycle of the closure. I get your view about consistent results between nprocs=1 and nprocs>1 in absolutely all cases.\r\n\r\nThe easier implementation to ensure similar behavior in all cases is to have remote calls to `myid()` go through a loopback connection. But at the cost of inefficiency in remote calls to the local process."}}
{"actor":{"display_login":"amitmurthy"}}