{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/models/issues/2359","repository_url":"https://api.github.com/repos/tensorflow/models","labels_url":"https://api.github.com/repos/tensorflow/models/issues/2359/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/models/issues/2359/comments","events_url":"https://api.github.com/repos/tensorflow/models/issues/2359/events","html_url":"https://github.com/tensorflow/models/issues/2359","id":256290634,"number":2359,"title":"Stuck training with increased input size","user":{"login":"UndeadBlow","id":2891003,"avatar_url":"https://avatars3.githubusercontent.com/u/2891003?v=4","gravatar_id":"","url":"https://api.github.com/users/UndeadBlow","html_url":"https://github.com/UndeadBlow","followers_url":"https://api.github.com/users/UndeadBlow/followers","following_url":"https://api.github.com/users/UndeadBlow/following{/other_user}","gists_url":"https://api.github.com/users/UndeadBlow/gists{/gist_id}","starred_url":"https://api.github.com/users/UndeadBlow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/UndeadBlow/subscriptions","organizations_url":"https://api.github.com/users/UndeadBlow/orgs","repos_url":"https://api.github.com/users/UndeadBlow/repos","events_url":"https://api.github.com/users/UndeadBlow/events{/privacy}","received_events_url":"https://api.github.com/users/UndeadBlow/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-09-08T15:29:39Z","updated_at":"2017-09-11T18:42:00Z","closed_at":null,"author_association":"NONE","body":"### System information\r\n- **What is the top-level directory of the model you are using**: models/object_detection\r\n- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**: No\r\n- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Linux Ubuntu 16.04\r\n- **TensorFlow installed from (source or binary)**: source\r\n- **TensorFlow version (use command below)**: 1.3.0-rc2\r\n- **Bazel version (if compiling from source)**: 0.5.4\r\n- **CUDA/cuDNN version**: 375.82\r\n- **GPU model and memory**: GeForce GTX 770 4GB\r\n- **Exact command to reproduce**:\r\nas default:\r\n```\r\n python3 object_detection/train.py \\\r\n    --logtostderr \\\r\n    --pipeline_config_path=ssd_mobilenet_v1_signs.config \\\r\n    --train_dir=object_detection/test\r\n\r\n```\r\n\r\n### Problem\r\nI'm trying to fine tune (to be more precise to overfit some little dataset) ssd_mobilenet_v1 from coco checkpoint. With default parameters all is ok, but when I try to change inpurt size from 300x300 to 512x512 or 600x600 or any other, training stucks. I've tried all possible combinations of hyperparameters, tried to train from scratch, tried to train on Pascal VOC, but every time training stucks. For example, with default parameters from ssd_mobilenet_v1_signs.config and default PascalVOC2012, with the only change:\r\n```\r\nimage_resizer {\r\n      fixed_shape_resizer {\r\n        height: 512\r\n        width: 512\r\n      }\r\n    }\r\n```\r\nTraining stucks on error with value ~10-13. On some little subsets (when I'm try to overfit for test) it gives little error (like ~0.2) but then it finds no objects on train frames (all scores very small like network didn't learn something at all).\r\n\r\nDoes the network recalculated when resizing? Is it supposed that a change in the input size should generally work?\r\n\r\n### Source code / logs\r\nSource code unchanged, except described. \r\nLogs is like endless:\r\n```\r\n....\r\nINFO:tensorflow:global step 20408: loss = 12.8249 (1.114 sec/step)\r\nINFO:tensorflow:global step 20409: loss = 11.7554 (0.877 sec/step)\r\nINFO:tensorflow:global step 20410: loss = 12.4740 (0.863 sec/step)\r\n....\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/models/issues/comments/328620915","html_url":"https://github.com/tensorflow/models/issues/2359#issuecomment-328620915","issue_url":"https://api.github.com/repos/tensorflow/models/issues/2359","id":328620915,"user":{"login":"reedwm","id":6510203,"avatar_url":"https://avatars2.githubusercontent.com/u/6510203?v=4","gravatar_id":"","url":"https://api.github.com/users/reedwm","html_url":"https://github.com/reedwm","followers_url":"https://api.github.com/users/reedwm/followers","following_url":"https://api.github.com/users/reedwm/following{/other_user}","gists_url":"https://api.github.com/users/reedwm/gists{/gist_id}","starred_url":"https://api.github.com/users/reedwm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/reedwm/subscriptions","organizations_url":"https://api.github.com/users/reedwm/orgs","repos_url":"https://api.github.com/users/reedwm/repos","events_url":"https://api.github.com/users/reedwm/events{/privacy}","received_events_url":"https://api.github.com/users/reedwm/received_events","type":"User","site_admin":false},"created_at":"2017-09-11T18:42:00Z","updated_at":"2017-09-11T18:42:00Z","author_association":"MEMBER","body":"@jch1 can you take a look?"}}
{"actor":{"display_login":"reedwm"}}