{"action":"created","issue":{"url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/3458","repository_url":"https://api.github.com/repos/PaddlePaddle/Paddle","labels_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/3458/labels{/name}","comments_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/3458/comments","events_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/3458/events","html_url":"https://github.com/PaddlePaddle/Paddle/issues/3458","id":249938577,"number":3458,"title":"When configured use \"--loadsave_parameters_in_pserver\" trainning fails to init parameters","user":{"login":"typhoonzero","id":13348433,"avatar_url":"https://avatars1.githubusercontent.com/u/13348433?v=4","gravatar_id":"","url":"https://api.github.com/users/typhoonzero","html_url":"https://github.com/typhoonzero","followers_url":"https://api.github.com/users/typhoonzero/followers","following_url":"https://api.github.com/users/typhoonzero/following{/other_user}","gists_url":"https://api.github.com/users/typhoonzero/gists{/gist_id}","starred_url":"https://api.github.com/users/typhoonzero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/typhoonzero/subscriptions","organizations_url":"https://api.github.com/users/typhoonzero/orgs","repos_url":"https://api.github.com/users/typhoonzero/repos","events_url":"https://api.github.com/users/typhoonzero/events{/privacy}","received_events_url":"https://api.github.com/users/typhoonzero/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-08-14T05:54:25Z","updated_at":"2017-08-17T08:54:21Z","closed_at":null,"body":"Word embedding using v1 API:\r\n\r\nLog is:\r\n\r\n```\r\nI0814 05:35:20.026533    69 Util.cpp:166] commandline: /usr/local/lib/python2.7/dist-packages/usr/local/opt/paddle/bin/paddle_trainer --loadsave_parameters_in_pserver=true --job=train --use_old_updater=1 --port=7164 --nics=eth0 --ports_num=1 --ports_num_for_sparse=1 --num_passes=1 --trainer_count=1 --saving_period=1 --log_period=20 --local=0 --rdma_tcp=tcp --config=emb_train.py --use_gpu=0 --trainer_id=0 --save_dir=./output --pservers=10.1.90.7,10.1.53.8 --num_gradient_servers=4\r\n[INFO 2017-08-14 05:35:20,519 networks.py:1491] The input order is [w1, w2, w3, w4, label]\r\n[INFO 2017-08-14 05:35:20,519 networks.py:1497] The output order is [__cost_0__]\r\nI0814 05:35:20.521780    69 Trainer.cpp:162] trainer mode: SgdSparseCpuTraining\r\nI0814 05:35:20.583519    69 PyDataProvider2.cpp:243] loading dataprovider dataprovider_emb::process\r\nI0814 05:35:20.584769    69 GradientMachine.cpp:85] Initing parameters..\r\n*** Aborted at 1502688920 (unix time) try \"date -d @1502688920\" if you are using GNU date ***\r\nPC: @                0x0 (unknown)\r\n*** SIGSEGV (@0x30) received by PID 69 (TID 0x7fea192fe740) from PID 48; stack trace: ***\r\n    @     0x7fea18cdb390 (unknown)\r\n    @           0xc95a3a paddle::VectorT<>::getSize()\r\n    @           0xd7057e paddle::Parameter::isFullSize()\r\n    @           0xd700d2 paddle::GradientMachine::randParameters()\r\n    @           0xe85f4f paddle::Trainer::init()\r\n    @           0xbda83e main\r\n    @     0x7fea178ed830 __libc_start_main\r\n    @           0xbda4b9 _start\r\n    @                0x0 (unknown)\r\n/usr/local/bin/paddle: line 113:    69 Segmentation fault      (core dumped) ${DEBUGGER} /usr/local/lib/python2.7/dist-packages/usr/local/opt/paddle/bin/paddle_trainer ${@:2}\r\n```\r\n\r\nSeems that \"PARAMETER_VALUE\" is not properly initialized in this case."},"comment":{"url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/comments/323010003","html_url":"https://github.com/PaddlePaddle/Paddle/issues/3458#issuecomment-323010003","issue_url":"https://api.github.com/repos/PaddlePaddle/Paddle/issues/3458","id":323010003,"user":{"login":"typhoonzero","id":13348433,"avatar_url":"https://avatars1.githubusercontent.com/u/13348433?v=4","gravatar_id":"","url":"https://api.github.com/users/typhoonzero","html_url":"https://github.com/typhoonzero","followers_url":"https://api.github.com/users/typhoonzero/followers","following_url":"https://api.github.com/users/typhoonzero/following{/other_user}","gists_url":"https://api.github.com/users/typhoonzero/gists{/gist_id}","starred_url":"https://api.github.com/users/typhoonzero/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/typhoonzero/subscriptions","organizations_url":"https://api.github.com/users/typhoonzero/orgs","repos_url":"https://api.github.com/users/typhoonzero/repos","events_url":"https://api.github.com/users/typhoonzero/events{/privacy}","received_events_url":"https://api.github.com/users/typhoonzero/received_events","type":"User","site_admin":false},"created_at":"2017-08-17T08:54:21Z","updated_at":"2017-08-17T08:54:21Z","body":"Did a bunch of testing, seems that in `ParameterClient2.cpp:sendParallel` `getLocalRow` is corrupted when running multiple pservers and trainers.\r\n\r\n```\r\n*** SIGABRT (@0x46) received by PID 70 (TID 0x7fd9659d4700) from PID 70; stack trace: ***\r\n    @     0x7fd9ac05b390 (unknown)\r\n    @     0x7fd9aac82428 gsignal\r\n    @     0x7fd9aac8402a abort\r\n    @     0x7fd9aacc47ea (unknown)\r\n    @     0x7fd9aaccd37a (unknown)\r\n    @     0x7fd9aacd153c cfree\r\n    @           0xd2d604 paddle::AlignedAllocator<>::deallocate()\r\n    @           0xd2b33c std::allocator_traits<>::deallocate()\r\n    @           0xd285dc std::_Vector_base<>::_M_deallocate()\r\n    @           0xd2574e std::vector<>::_M_default_append()\r\n    @           0xd228b1 std::vector<>::resize()\r\n    @           0xe9c084 paddle::RowBuffer::getWithAutoGrowth()\r\n    @           0xe9c119 paddle::SparseRowCpuMatrix::getLocalRow()\r\n    @          0x2611120 paddle::ParameterClient2::sendParallel()\r\n    @          0x261223b _ZZN6paddle16ParameterClient223sendAndReceiveParameterENS_19ParameterUpdateModeENS_19enumeration_wrapper13ParameterTypeERKSt6vectorINS_17ParameterSegmentsESaIS5_EElfbS3_S3_ENKUlimE_clEim\r\n    @          0x2616c91 _ZNSt17_Function_handlerIFvimEZN6paddle16ParameterClient223sendAndReceiveParameterENS1_19ParameterUpdateModeENS1_19enumeration_wrapper13ParameterTypeERKSt6vectorINS1_17ParameterSegmentsESaIS7_EElfbS5_S5_EUlimE_E9_M_invokeERKSt9_Any_dataOiOm\r\n    @           0xd22504 std::function<>::operator()()\r\n    @           0xe51720 paddle::SyncThreadPool::run()\r\n    @           0xe513eb _ZZN6paddle14SyncThreadPool5startEvENKUliE_clEi\r\n    @           0xe84f54 _ZNSt12_Bind_simpleIFZN6paddle14SyncThreadPool5startEvEUliE_mEE9_M_invokeIILm0EEEEvSt12_Index_tupleIIXspT_EEE\r\n    @           0xe84e26 _ZNSt12_Bind_simpleIFZN6paddle14SyncThreadPool5startEvEUliE_mEEclEv\r\n    @           0xe84d5a _ZNSt6thread5_ImplISt12_Bind_simpleIFZN6paddle14SyncThreadPool5startEvEUliE_mEEE6_M_runEv\r\n    @     0x7fd9ab5eec80 (unknown)\r\n    @     0x7fd9ac0516ba start_thread\r\n    @     0x7fd9aad543dd clone\r\n    @                0x0 (unknown)\r\n```\r\n\r\nAnd other trainer cores at:\r\n\r\n\r\n```\r\n*** SIGABRT (@0x46) received by PID 70 (TID 0x7f63d91d3700) from PID 70; stack trace: ***\r\n    @     0x7f641f75b390 (unknown)\r\n    @     0x7f641e382428 gsignal\r\n    @     0x7f641e38402a abort\r\n    @     0x7f641e3c47ea (unknown)\r\n    @     0x7f641e3cd37a (unknown)\r\n    @     0x7f641e3d153c cfree\r\n    @           0xd89094 __gnu_cxx::new_allocator<>::deallocate()\r\n    @           0xd87b82 std::allocator_traits<>::deallocate()\r\n    @           0xd868e2 std::_Vector_base<>::_M_deallocate()\r\n    @           0xd85711 std::_Vector_base<>::~_Vector_base()\r\n    @           0xd84c07 std::vector<>::~vector()\r\n    @           0xec08bc paddle::MsgReader::readBlocks()\r\n    @          0x2611180 paddle::ParameterClient2::sendParallel()\r\n    @          0x261223b _ZZN6paddle16ParameterClient223sendAndReceiveParameterENS_19ParameterUpdateModeENS_19enumeration_wrapper13ParameterTypeERKSt6vectorINS_17ParameterSegmentsESaIS5_EElfbS3_S3_ENKUlimE_clEim\r\n    @          0x2616c91 _ZNSt17_Function_handlerIFvimEZN6paddle16ParameterClient223sendAndReceiveParameterENS1_19ParameterUpdateModeENS1_19enumeration_wrapper13ParameterTypeERKSt6vectorINS1_17ParameterSegmentsESaIS7_EElfbS5_S5_EUlimE_E9_M_invokeERKSt9_Any_dataOiOm\r\n    @           0xd22504 std::function<>::operator()()\r\n    @           0xe51720 paddle::SyncThreadPool::run()\r\n    @           0xe513eb _ZZN6paddle14SyncThreadPool5startEvENKUliE_clEi\r\n    @           0xe84f54 _ZNSt12_Bind_simpleIFZN6paddle14SyncThreadPool5startEvEUliE_mEE9_M_invokeIILm0EEEEvSt12_Index_tupleIIXspT_EEE\r\n    @           0xe84e26 _ZNSt12_Bind_simpleIFZN6paddle14SyncThreadPool5startEvEUliE_mEEclEv\r\n    @           0xe84d5a _ZNSt6thread5_ImplISt12_Bind_simpleIFZN6paddle14SyncThreadPool5startEvEUliE_mEEE6_M_runEv\r\n    @     0x7f641eceec80 (unknown)\r\n    @     0x7f641f7516ba start_thread\r\n    @     0x7f641e4543dd clone\r\n    @                0x0 (unknown)\r\n/usr/local/bin/paddle: line 113:    70 Aborted                 (core dumped) ${DEBUGGER} $MYDIR/../opt/paddle/bin/paddle_trainer ${@:2}\r\n```"}}
{"actor":{"display_login":"typhoonzero"}}