{"action":"created","issue":{"url":"https://api.github.com/repos/apache/incubator-mxnet/issues/6620","repository_url":"https://api.github.com/repos/apache/incubator-mxnet","labels_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/6620/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/6620/comments","events_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/6620/events","html_url":"https://github.com/apache/incubator-mxnet/issues/6620","id":234683710,"number":6620,"title":"Training_cifar10.py breaks when using USR_PROFILER","user":{"login":"GuanhuaWang","id":14852010,"avatar_url":"https://avatars2.githubusercontent.com/u/14852010?v=4","gravatar_id":"","url":"https://api.github.com/users/GuanhuaWang","html_url":"https://github.com/GuanhuaWang","followers_url":"https://api.github.com/users/GuanhuaWang/followers","following_url":"https://api.github.com/users/GuanhuaWang/following{/other_user}","gists_url":"https://api.github.com/users/GuanhuaWang/gists{/gist_id}","starred_url":"https://api.github.com/users/GuanhuaWang/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GuanhuaWang/subscriptions","organizations_url":"https://api.github.com/users/GuanhuaWang/orgs","repos_url":"https://api.github.com/users/GuanhuaWang/repos","events_url":"https://api.github.com/users/GuanhuaWang/events{/privacy}","received_events_url":"https://api.github.com/users/GuanhuaWang/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-06-08T23:29:26Z","updated_at":"2017-09-30T18:26:18Z","closed_at":null,"author_association":"NONE","body":"For bugs or installation issues, please provide the following information.\r\nThe more information you provide, the more likely people will be able to help you.\r\n\r\n## Environment info\r\nOperating System: Ubuntu 14.04\r\n\r\nCompiler: g++, nvcc\r\n\r\nPackage used (Python/R/Scala/Julia):\r\n\r\nMXNet version: latest\r\n\r\nOr if installed from source:\r\n\r\nMXNet commit hash (`git rev-parse HEAD`):\r\n\r\nIf you are using python package, please provide\r\n\r\nPython version and distribution: 2.7\r\n\r\nIf you are using R package, please provide\r\n\r\nR `sessionInfo()`:\r\n\r\n## Error Message:\r\nPlease paste the full error message, including stack trace.\r\n\r\nroot@d6dfc4c90761:/mxnet/example/image-classification# python train_cifar10.py --network resnet --num-layers 110 --batch-size 128 --gpus 0,1,2,3\r\nINFO:root:start with arguments Namespace(batch_size=128, benchmark=0, data_nthreads=4, data_train='data/cifar10_train.rec', data_val='data/cifar10_val.rec', disp_batches=20, dtype='float32', gpus='0,1,2,3', image_shape='3,28,28', kv_store='device', load_epoch=None, lr=0.05, lr_factor=0.1, lr_step_epochs='200,250', max_random_aspect_ratio=0, max_random_h=36, max_random_l=50, max_random_rotate_angle=0, max_random_s=50, max_random_scale=1, max_random_shear_ratio=0, min_random_scale=1, model_prefix=None, mom=0.9, monitor=0, network='resnet', num_classes=10, num_epochs=300, num_examples=50000, num_layers=110, optimizer='sgd', pad_size=4, random_crop=1, random_mirror=1, rgb_mean='123.68,116.779,103.939', test_io=0, top_k=0, wd=0.0001)\r\n[23:18:15] src/io/iter_image_recordio.cc:220: ImageRecordIOParser: data/cifar10_train.rec, use 4 threads for decoding..\r\n[23:18:15] src/io/iter_image_recordio.cc:220: ImageRecordIOParser: data/cifar10_val.rec, use 4 threads for decoding..\r\n[23:18:15] /mxnet/dmlc-core/include/dmlc/./logging.h:300: [23:18:15] src/c_api/c_api_ndarray.cc:381: Operator _zeros cannot be run; requires at least one of FCompute<xpu>, NDArrayFunction, FCreateOperator be registered\r\n\r\nStack trace returned 10 entries:\r\n[bt] (0) /mxnet/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7feadfb2b5cc]\r\n[bt] (1) /mxnet/python/mxnet/../../lib/libmxnet.so(MXImperativeInvoke+0x673) [0x7feae069e9a3]\r\n[bt] (2) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7feb828bce40]\r\n[bt] (3) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7feb828bc8ab]\r\n[bt] (4) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48f) [0x7feb82acc3df]\r\n[bt] (5) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0x11d82) [0x7feb82ad0d82]\r\n[bt] (6) python(PyObject_Call+0x43) [0x4b0cb3]\r\n[bt] (7) python(PyEval_EvalFrameEx+0x5faf) [0x4c9faf]\r\n[bt] (8) python(PyEval_EvalCodeEx+0x255) [0x4c2765]\r\n[bt] (9) python(PyEval_EvalFrameEx+0x68d1) [0x4ca8d1]\r\n\r\nTraceback (most recent call last):\r\n  File \"train_cifar10.py\", line 53, in <module>\r\n    fit.fit(args, sym, data.get_rec_iter)\r\n  File \"/mxnet/example/image-classification/common/fit.py\", line 187, in fit\r\n    monitor            = monitor)\r\n  File \"/mxnet/python/mxnet/module/base_module.py\", line 444, in fit\r\n    for_training=True, force_rebind=force_rebind)\r\n  File \"/mxnet/python/mxnet/module/module.py\", line 388, in bind\r\n    state_names=self._state_names)\r\n  File \"/mxnet/python/mxnet/module/executor_group.py\", line 205, in __init__\r\n    self.bind_exec(data_shapes, label_shapes, shared_group)\r\n  File \"/mxnet/python/mxnet/module/executor_group.py\", line 301, in bind_exec\r\n    shared_group))\r\n  File \"/mxnet/python/mxnet/module/executor_group.py\", line 605, in _bind_ith_exec\r\n    context, self.logger)\r\n  File \"/mxnet/python/mxnet/module/executor_group.py\", line 583, in _get_or_reshape\r\n    arg_arr = nd.zeros(arg_shape, context, dtype=arg_type)\r\n  File \"/mxnet/python/mxnet/ndarray.py\", line 900, in zeros\r\n    return _internal._zeros(shape=shape, ctx=ctx, dtype=dtype)\r\n  File \"/mxnet/python/mxnet/_ctypes/ndarray.py\", line 133, in generic_ndarray_function\r\n    c_array(ctypes.c_char_p, [c_str(str(i)) for i in kwargs.values()])))\r\n  File \"/mxnet/python/mxnet/base.py\", line 78, in check_call\r\n    raise MXNetError(py_str(_LIB.MXGetLastError()))\r\nmxnet.base.MXNetError: [23:18:15] src/c_api/c_api_ndarray.cc:381: Operator _zeros cannot be run; requires at least one of FCompute<xpu>, NDArrayFunction, FCreateOperator be registered\r\n\r\nStack trace returned 10 entries:\r\n[bt] (0) /mxnet/python/mxnet/../../lib/libmxnet.so(_ZN4dmlc15LogMessageFatalD1Ev+0x3c) [0x7feadfb2b5cc]\r\n[bt] (1) /mxnet/python/mxnet/../../lib/libmxnet.so(MXImperativeInvoke+0x673) [0x7feae069e9a3]\r\n[bt] (2) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7feb828bce40]\r\n[bt] (3) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7feb828bc8ab]\r\n[bt] (4) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48f) [0x7feb82acc3df]\r\n[bt] (5) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(+0x11d82) [0x7feb82ad0d82]\r\n[bt] (6) python(PyObject_Call+0x43) [0x4b0cb3]\r\n[bt] (7) python(PyEval_EvalFrameEx+0x5faf) [0x4c9faf]\r\n[bt] (8) python(PyEval_EvalCodeEx+0x255) [0x4c2765]\r\n[bt] (9) python(PyEval_EvalFrameEx+0x68d1) [0x4ca8d1]\r\n\r\n## Minimum reproducible example\r\nif you are using your own code, please provide a short script that reproduces the error.\r\n\r\n## Steps to reproduce\r\nor if you are running standard examples, please provide the commands you have run that lead to the error.\r\n\r\n1. I use mxnet docker for GPU. without re-build mxnet, the system works well with the command `python train_cifar10.py --network resnet --num-layers 110 --batch-size 128 --gpus 0,1,2,3`\r\n\r\n2. I  modify config.mk in /mxnet/make directory, and `set USR_PROFILER = 1`\r\n3. `make`\r\n\r\n4. after that, I can use profiler and I can run the sample py code in /mxnet/example/profiler/*\r\n\r\n5. However I cannot run `python train_cifar10.py --network resnet --num-layers 110 --batch-size 128 --gpus 0,1,2,3`. It shows the error message above.\r\n\r\n## What have you tried to solve it?\r\n\r\n1. No idea\r\n2.\r\n3.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/incubator-mxnet/issues/comments/333326930","html_url":"https://github.com/apache/incubator-mxnet/issues/6620#issuecomment-333326930","issue_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/6620","id":333326930,"user":{"login":"yajiedesign","id":771744,"avatar_url":"https://avatars1.githubusercontent.com/u/771744?v=4","gravatar_id":"","url":"https://api.github.com/users/yajiedesign","html_url":"https://github.com/yajiedesign","followers_url":"https://api.github.com/users/yajiedesign/followers","following_url":"https://api.github.com/users/yajiedesign/following{/other_user}","gists_url":"https://api.github.com/users/yajiedesign/gists{/gist_id}","starred_url":"https://api.github.com/users/yajiedesign/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yajiedesign/subscriptions","organizations_url":"https://api.github.com/users/yajiedesign/orgs","repos_url":"https://api.github.com/users/yajiedesign/repos","events_url":"https://api.github.com/users/yajiedesign/events{/privacy}","received_events_url":"https://api.github.com/users/yajiedesign/received_events","type":"User","site_admin":false},"created_at":"2017-09-30T18:26:18Z","updated_at":"2017-09-30T18:26:18Z","author_association":"CONTRIBUTOR","body":"This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!"}}
{"actor":{"display_login":"yajiedesign"}}