{"action":"created","issue":{"url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/57","repository_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark","labels_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/57/labels{/name}","comments_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/57/comments","events_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/57/events","html_url":"https://github.com/yahoo/TensorFlowOnSpark/issues/57","id":220150864,"number":57,"title":"TFoS can't export model","user":{"login":"terryKing1992","id":22629426,"avatar_url":"https://avatars3.githubusercontent.com/u/22629426?v=3","gravatar_id":"","url":"https://api.github.com/users/terryKing1992","html_url":"https://github.com/terryKing1992","followers_url":"https://api.github.com/users/terryKing1992/followers","following_url":"https://api.github.com/users/terryKing1992/following{/other_user}","gists_url":"https://api.github.com/users/terryKing1992/gists{/gist_id}","starred_url":"https://api.github.com/users/terryKing1992/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/terryKing1992/subscriptions","organizations_url":"https://api.github.com/users/terryKing1992/orgs","repos_url":"https://api.github.com/users/terryKing1992/repos","events_url":"https://api.github.com/users/terryKing1992/events{/privacy}","received_events_url":"https://api.github.com/users/terryKing1992/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2017-04-07T08:52:24Z","updated_at":"2017-04-12T13:06:14Z","closed_at":null,"body":"I append some code in mnist_dist.py file, like this:\r\n ```\r\n while not sv.should_stop() and step < args.steps:\r\n    # Run a training step asynchronously.\r\n    # See `tf.train.SyncReplicasOptimizer` for additional details on how to\r\n    # perform *synchronous* training.\r\n\r\n    # using feed_dict\r\n    batch_xs, batch_ys = feed_dict()\r\n    feed = {x: batch_xs, y_: batch_ys}\r\n\r\n    if len(batch_xs) != batch_size:\r\n        print(\"done feeding\")\r\n        break\r\n    else:\r\n        if args.mode == \"train\":\r\n            _, step = sess.run([train_op, global_step], feed_dict=feed)\r\n            # print accuracy and save model checkpoint to HDFS every 100 steps\r\n            if (step % 100 == 0):\r\n                print(\"{0} step: {1} accuracy: {2}\".format(datetime.now().isoformat(), step,\r\n                                                           sess.run(accuracy,\r\n                                                                    {x: batch_xs, y_: batch_ys})))\r\n        else:  # args.mode == \"inference\"\r\n            labels, preds, acc = sess.run([label, prediction, accuracy], feed_dict=feed)\r\n\r\n            results = [\"{0} Label: {1}, Prediction: {2}\".format(datetime.now().isoformat(), l, p) for\r\n                       l, p in zip(labels, preds)]\r\n            TFNode.batch_results(ctx.mgr, results)\r\n            print(\"acc: {0}\".format(acc))\r\n\r\n  if sv.is_chief:\r\n    print(\"save model to:{}=======1\".format(local_model_dir))\r\n    sess.graph._unsafe_unfinalize()\r\n    classification_inputs = utils.build_tensor_info(x)\r\n    classification_outputs_classes = utils.build_tensor_info(y)\r\n    print ('begin exporting!======11111')\r\n    classification_signature = signature_def_utils.build_signature_def(\r\n        inputs={signature_constants.CLASSIFY_INPUTS: classification_inputs},\r\n        outputs={\r\n            signature_constants.CLASSIFY_OUTPUT_CLASSES:\r\n                classification_outputs_classes\r\n        },\r\n        method_name=signature_constants.CLASSIFY_METHOD_NAME)\r\n    print ('begin exporting!======22222')\r\n    tensor_info_x = utils.build_tensor_info(x)\r\n    tensor_info_y = utils.build_tensor_info(y)\r\n    print ('begin exporting!======33333')\r\n    prediction_signature = signature_def_utils.build_signature_def(\r\n        inputs={'images': tensor_info_x},\r\n        outputs={'scores': tensor_info_y},\r\n        method_name=signature_constants.PREDICT_METHOD_NAME)\r\n    print ('begin exporting!======44444')\r\n    legacy_init_op = tf.group(tf.tables_initializer(), name='legacy_init_op')\r\n    print ('begin exporting!======55555')\r\n    builder = saved_model_builder.SavedModelBuilder(local_model_dir)\r\n    print('begin exporting!======66666')\r\n    builder.add_meta_graph_and_variables(\r\n        sess, [tag_constants.SERVING],\r\n        signature_def_map={\r\n            'predict_images':\r\n                prediction_signature,\r\n            signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY:\r\n                classification_signature,\r\n        },clear_devices=True,legacy_init_op=legacy_init_op)\r\n    print ('begin exporting!======77777')\r\n    builder.save()\r\n```\r\n\r\nbut the chief worker only exec to this `print('begin exporting!======66666')` line, the task end without executing the rest code.\r\n\r\nand the chief worker log as bellow:\r\n```\r\n  save model to:/data/mnist/1/0=======1\r\n  begin exporting!======11111\r\n  begin exporting!======22222\r\n  begin exporting!======33333\r\n  begin exporting!======44444\r\n  INFO:tensorflow:No assets to save.2017-04-07 16:31:23,479 INFO (Thread-1-30287) No assets to save.\r\n  INFO:tensorflow:No assets to write.\r\n  2017-04-07 16:31:23,479 INFO (Thread-1-30287) No assets to write.\r\n  begin exporting!======55555\r\n  begin exporting!======66666\r\n  2017-04-07 16:31:23,520 INFO (MainThread-30287) Feeding None into output queue\r\n  2017-04-07 16:31:23,528 INFO (MainThread-30287) Setting mgr.state to 'stopped'17/04/07 16:31:23 INFO python.PythonRunner: Times: total = 123, boot = -28156, init = 28200, finish = 7917/04/07 16:31:23 INFO executor.Executor: Finished task 4.0 in stage 3.0 (TID 36). 2102 bytes result sent to driver17/04/07 16:31:29 INFO executor.CoarseGrainedExecutorBackend: Driver commanded a shutdown\r\n  17/04/07 16:31:29 INFO memory.MemoryStore: MemoryStore cleared\r\n  17/04/07 16:31:29 INFO storage.BlockManager: BlockManager stopped\r\n  17/04/07 16:31:29 INFO util.ShutdownHookManager: Shutdown hook called17/04/07 16:31:29 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-a281a003-75aa-4d20-8f9d-93e3dbe8874a/executor-35ec5b73-87a6-4fa2-b895-b5783cace0d8/spark-957ff0d0-334b-42cc-98fb-4995ab0f871517/04/07 16:31:29 ERROR executor.CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM:29 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-a281a003-75aa-4d20-8f9d-93e3dbe8874a/executor-35ec5b73-87a6-4fa2-b895-b5783cace0d8/spark-957ff0d0-334b-42cc-98fb-4995ab0f8715  \r\n```\r\n\r\nhow can i make the task work well on exporting model. BTW, this export model code can work well on tensorflow distribution. thanks a lot"},"comment":{"url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/comments/293569357","html_url":"https://github.com/yahoo/TensorFlowOnSpark/issues/57#issuecomment-293569357","issue_url":"https://api.github.com/repos/yahoo/TensorFlowOnSpark/issues/57","id":293569357,"user":{"login":"terryKing1992","id":22629426,"avatar_url":"https://avatars3.githubusercontent.com/u/22629426?v=3","gravatar_id":"","url":"https://api.github.com/users/terryKing1992","html_url":"https://github.com/terryKing1992","followers_url":"https://api.github.com/users/terryKing1992/followers","following_url":"https://api.github.com/users/terryKing1992/following{/other_user}","gists_url":"https://api.github.com/users/terryKing1992/gists{/gist_id}","starred_url":"https://api.github.com/users/terryKing1992/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/terryKing1992/subscriptions","organizations_url":"https://api.github.com/users/terryKing1992/orgs","repos_url":"https://api.github.com/users/terryKing1992/repos","events_url":"https://api.github.com/users/terryKing1992/events{/privacy}","received_events_url":"https://api.github.com/users/terryKing1992/received_events","type":"User","site_admin":false},"created_at":"2017-04-12T13:06:14Z","updated_at":"2017-04-12T13:06:14Z","body":"Thanks all the same"}}
{"actor":{"display_login":"terryKing1992"}}