{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14084","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14084/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14084/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14084/events","html_url":"https://github.com/tensorflow/tensorflow/issues/14084","id":269491721,"number":14084,"title":"Tensorflow hooks do not properly write events to HDFS","user":{"login":"wochinge","id":7667273,"avatar_url":"https://avatars1.githubusercontent.com/u/7667273?v=4","gravatar_id":"","url":"https://api.github.com/users/wochinge","html_url":"https://github.com/wochinge","followers_url":"https://api.github.com/users/wochinge/followers","following_url":"https://api.github.com/users/wochinge/following{/other_user}","gists_url":"https://api.github.com/users/wochinge/gists{/gist_id}","starred_url":"https://api.github.com/users/wochinge/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wochinge/subscriptions","organizations_url":"https://api.github.com/users/wochinge/orgs","repos_url":"https://api.github.com/users/wochinge/repos","events_url":"https://api.github.com/users/wochinge/events{/privacy}","received_events_url":"https://api.github.com/users/wochinge/received_events","type":"User","site_admin":false},"labels":[{"id":404586594,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20tensorflower","name":"stat:awaiting tensorflower","color":"b60205","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-10-30T06:14:43Z","updated_at":"2017-11-23T10:18:55Z","closed_at":"2017-11-02T16:51:31Z","author_association":"NONE","body":"### System information\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: Yes\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Alpine Linux 3.6.2 (running with Docker)\r\n- **TensorFlow installed from (source or binary)**: Binary (conda-forge)\r\n- **TensorFlow version (use command below)**: 1.3.0\r\n- **Python version**: 3.6.3\r\n- **Bazel version (if compiling from source)**: - \r\n- **CUDA/cuDNN version**: -\r\n- **GPU model and memory**: using CPU\r\n- **Exact command to reproduce**: `CLASSPATH=$(${HADOOP_HOME}/bin/hadoop classpath --glob) jupyter notebook`\r\n\r\n### Describe the problem\r\nI am reading files from hdfs and also want to use hdfs as `model_dir` to store the tensorflow output. Reading and writing of the checkpoints works fine. However, the the events file just gets created, but it does not get updated with the evaluation events. \r\n\r\nWhen I kill the notebook then the output is written, but not in between. Changing the model directory to a local one solves this. Also adding an extra `SummarySaverHook` which points to a local directory did not help.\r\n\r\nI am running my code using the instructions here: <https://www.tensorflow.org/deploy/hadoop>\r\n\r\n### Source code / logs\r\nI normally use an Estimator with a custom model function, but I reduced it to the following:\r\n\r\n```python\r\nimport tensorflow as tf\r\nimport numpy\r\n\r\nhdfs_namenode = 'hdfs://xx.xx.xx.xx:8020'\r\nfiles = ['{}/data/part-00000-2a4cdbb2-f152-4503-9da8-cba63c7648fc-c000.csv'.format(hdfs_namenode)]\r\n\r\noutput_directory = '{}/tresults/'.format(hdfs_namenode)\r\n\r\nfeature_columns = [tf.feature_column.numeric_column(\"x\", shape=[70])]\r\n\r\ndef input_function_training:\r\n  ...\r\ndef input_function_evaluation:\r\n  ...\r\n\r\ndef experiment_fn(run_config, hparams):\r\n    estimator = tf.estimator.DNNClassifier(hidden_units=[4], \r\n        feature_columns=feature_columns,\r\n        model_dir=output_directory, \r\n        n_classes=2,\r\n        config=run_config)\r\n    \r\n    return tf.contrib.learn.Experiment(estimator=estimator, \r\n        train_input_fn=input_function_training, \r\n        eval_input_fn=input_function_evaluation)\r\n\r\ntf.contrib.learn.learn_runner.run(experiment_fn,\r\n    run_config=tf.contrib.learn.RunConfig(model_dir=output_directory,\r\n        save_checkpoints_steps=10,\r\n        save_checkpoints_secs=None,\r\n        save_summary_steps=1))\r\n```\r\n\r\nThe logs do not show any problems:\r\n\r\n```\r\nWARNING:tensorflow:RunConfig.uid (from tensorflow.contrib.learn.python.learn.estimators.run_config) is experimental and may change or be removed at any time, and without warning.\r\nINFO:tensorflow:Using config: {'_task_type': None, '_task_id': 0, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7fb30e032780>, '_master': '', '_num_ps_replicas': 0, '_num_worker_replicas': 0, '_environment': 'local', '_is_chief': True, '_evaluation_master': '', '_tf_config': gpu_options {\r\n  per_process_gpu_memory_fraction: 1.0\r\n}\r\n, '_tf_random_seed': None, '_save_summary_steps': 1, '_save_checkpoints_secs': None, '_log_step_count_steps': 100, '_session_config': None, '_save_checkpoints_steps': 10, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_model_dir': 'hdfs://xx.xx.xx.xx:8020/tresults/'}\r\nWARNING:tensorflow:RunConfig.uid (from tensorflow.contrib.learn.python.learn.estimators.run_config) is experimental and may change or be removed at any time, and without warning.\r\nWARNING:tensorflow:From /opt/conda/lib/python3.6/site-packages/tensorflow/contrib/learn/python/learn/monitors.py:269: BaseMonitor.__init__ (from tensorflow.contrib.learn.python.learn.monitors) is deprecated and will be removed after 2016-12-05.\r\nInstructions for updating:\r\nMonitors are deprecated. Please use tf.train.SessionRunHook.\r\nINFO:tensorflow:Create CheckpointSaverHook.\r\nINFO:tensorflow:Saving checkpoints for 1 into hdfs://xx.xx.xx.xx:8020/tresults/model.ckpt.\r\nWARNING:tensorflow:Casting <dtype: 'float32'> labels to bool.\r\nWARNING:tensorflow:Casting <dtype: 'float32'> labels to bool.\r\nINFO:tensorflow:Starting evaluation at 2017-10-30-05:41:22\r\nINFO:tensorflow:Restoring parameters fromhdfs://xx.xx.xx.xx:8020/tresults/model.ckpt-1\r\nINFO:tensorflow:Evaluation [1/100]\r\n...\r\nINFO:tensorflow:Evaluation [100/100]\r\nINFO:tensorflow:Finished evaluation at 2017-10-30-05:42:24\r\nINFO:tensorflow:Saving dict for global step 1: accuracy = 0.46836, accuracy_baseline = 0.75205, auc = 0.697368, auc_precision_recall = 0.591545, average_loss = 0.692944, global_step = 1, label/mean = 0.24795, loss = 692.944, prediction/mean = 0.524391\r\nINFO:tensorflow:Validation (step 1): accuracy = 0.46836, accuracy_baseline = 0.75205, auc = 0.697368, auc_precision_recall = 0.591545, average_loss = 0.692944, label/mean = 0.24795, loss = 692.944, prediction/mean = 0.524391, global_step = 1\r\nINFO:tensorflow:loss = 42.1861, step = 1\r\nINFO:tensorflow:Saving checkpoints for 11 into hdfs://xx.xx.xx.xx:8020/tresults/model.ckpt.\r\n```"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/346578721","html_url":"https://github.com/tensorflow/tensorflow/issues/14084#issuecomment-346578721","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/14084","id":346578721,"user":{"login":"kelly-tlz","id":12132762,"avatar_url":"https://avatars3.githubusercontent.com/u/12132762?v=4","gravatar_id":"","url":"https://api.github.com/users/kelly-tlz","html_url":"https://github.com/kelly-tlz","followers_url":"https://api.github.com/users/kelly-tlz/followers","following_url":"https://api.github.com/users/kelly-tlz/following{/other_user}","gists_url":"https://api.github.com/users/kelly-tlz/gists{/gist_id}","starred_url":"https://api.github.com/users/kelly-tlz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kelly-tlz/subscriptions","organizations_url":"https://api.github.com/users/kelly-tlz/orgs","repos_url":"https://api.github.com/users/kelly-tlz/repos","events_url":"https://api.github.com/users/kelly-tlz/events{/privacy}","received_events_url":"https://api.github.com/users/kelly-tlz/received_events","type":"User","site_admin":false},"created_at":"2017-11-23T10:18:55Z","updated_at":"2017-11-23T10:18:55Z","author_association":"NONE","body":"After reading this, how can I write tf events to hdfs?"}}
{"actor":{"display_login":"kelly-tlz"}}