{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/1815","repository_url":"https://api.github.com/repos/dmlc/mxnet","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/1815/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/1815/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/1815/events","html_url":"https://github.com/dmlc/mxnet/issues/1815","id":147257611,"number":1815,"title":"fcnxs example validation accuracy does not change","user":{"login":"sinaj","id":2126250,"avatar_url":"https://avatars0.githubusercontent.com/u/2126250?v=3","gravatar_id":"","url":"https://api.github.com/users/sinaj","html_url":"https://github.com/sinaj","followers_url":"https://api.github.com/users/sinaj/followers","following_url":"https://api.github.com/users/sinaj/following{/other_user}","gists_url":"https://api.github.com/users/sinaj/gists{/gist_id}","starred_url":"https://api.github.com/users/sinaj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sinaj/subscriptions","organizations_url":"https://api.github.com/users/sinaj/orgs","repos_url":"https://api.github.com/users/sinaj/repos","events_url":"https://api.github.com/users/sinaj/events{/privacy}","received_events_url":"https://api.github.com/users/sinaj/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2016-04-10T19:52:49Z","updated_at":"2017-05-24T06:01:21Z","closed_at":"2017-05-23T23:54:46Z","body":"Hello,\nI am trying to use the example code for image segmentation. I simply changed the train.lst and val.lst to replace my images, but when I run it, although the train accuracy changes, the validation accuracy does not change at all (it is always 0.730370). Is there any potential situation that it could happen?\n\nHere is the log:\n\n> INFO:root:Start training with cpu(0)\n> INFO:root:Saved checkpoint to \"model_pascal/FCN32s_VGG16-0001.params\"\n> INFO:root:--->Epoch[0] Train-accuracy=0.500000\n> INFO:root: in eval process...\n> INFO:root:batch[4] Validation-accuracy=0.730370\n> INFO:root:Saved checkpoint to \"model_pascal/FCN32s_VGG16-0002.params\"\n> INFO:root:--->Epoch[1] Train-accuracy=0.500000\n> INFO:root: in eval process...\n> INFO:root:batch[4] Validation-accuracy=0.730370\n> INFO:root:Saved checkpoint to \"model_pascal/FCN32s_VGG16-0003.params\"\n> INFO:root:--->Epoch[2] Train-accuracy=0.750000\n> INFO:root: in eval process...\n> INFO:root:batch[4] Validation-accuracy=0.730370\n> INFO:root:Saved checkpoint to \"model_pascal/FCN32s_VGG16-0004.params\"\n> INFO:root:--->Epoch[3] Train-accuracy=1.000000\n> INFO:root: in eval process...\n> INFO:root:batch[4] Validation-accuracy=0.730370\n> INFO:root:Saved checkpoint to \"model_pascal/FCN32s_VGG16-0005.params\"\n"},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/303625920","html_url":"https://github.com/dmlc/mxnet/issues/1815#issuecomment-303625920","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/1815","id":303625920,"user":{"login":"liangfu","id":857821,"avatar_url":"https://avatars2.githubusercontent.com/u/857821?v=3","gravatar_id":"","url":"https://api.github.com/users/liangfu","html_url":"https://github.com/liangfu","followers_url":"https://api.github.com/users/liangfu/followers","following_url":"https://api.github.com/users/liangfu/following{/other_user}","gists_url":"https://api.github.com/users/liangfu/gists{/gist_id}","starred_url":"https://api.github.com/users/liangfu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/liangfu/subscriptions","organizations_url":"https://api.github.com/users/liangfu/orgs","repos_url":"https://api.github.com/users/liangfu/repos","events_url":"https://api.github.com/users/liangfu/events{/privacy}","received_events_url":"https://api.github.com/users/liangfu/received_events","type":"User","site_admin":false},"created_at":"2017-05-24T06:01:21Z","updated_at":"2017-05-24T06:01:21Z","body":"Try debug the output of each layer by adding following code before executing forward pass (in this case, the code exists in `solver.py` file)\r\n```\r\nimport time\r\ndef stat_helper(name, array):\r\n  \"\"\"wrapper for executor callback\"\"\"\r\n  import ctypes\r\n  from mxnet.ndarray import NDArray\r\n  from mxnet.base import NDArrayHandle, py_str\r\n  array = ctypes.cast(array, NDArrayHandle)\r\n  array = NDArray(array, writable=False).asnumpy()\r\n  # array.wait_to_read()\r\n  print (name, array.shape, np.mean(array), np.std(array), ('%.1fms' % (float(time.time()-stat_helper.start_time)*1000)))\r\n  # print (name, array.shape, ('%.1fms' % (float(time.time()-stat_helper.start_time)*1000)))\r\n  stat_helper.start_time=time.time()\r\n  stat_helper.start_time=float(time.time())\r\n  self.executor.set_monitor_callback(stat_helper)\r\n```\r\nAnd there should be no numerical overflow (displayed as `inf` for example) in any layer."}}
{"actor":{"display_login":"liangfu"}}