{"action":"opened","issue":{"url":"https://api.github.com/repos/apache/incubator-mxnet/issues/8799","repository_url":"https://api.github.com/repos/apache/incubator-mxnet","labels_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/8799/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/8799/comments","events_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/8799/events","html_url":"https://github.com/apache/incubator-mxnet/issues/8799","id":276374015,"number":8799,"title":"Dangling outputs and dtype != float32: Gradient computation fails","user":{"login":"mseeger","id":6508962,"avatar_url":"https://avatars3.githubusercontent.com/u/6508962?v=4","gravatar_id":"","url":"https://api.github.com/users/mseeger","html_url":"https://github.com/mseeger","followers_url":"https://api.github.com/users/mseeger/followers","following_url":"https://api.github.com/users/mseeger/following{/other_user}","gists_url":"https://api.github.com/users/mseeger/gists{/gist_id}","starred_url":"https://api.github.com/users/mseeger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mseeger/subscriptions","organizations_url":"https://api.github.com/users/mseeger/orgs","repos_url":"https://api.github.com/users/mseeger/repos","events_url":"https://api.github.com/users/mseeger/events{/privacy}","received_events_url":"https://api.github.com/users/mseeger/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-11-23T13:25:11Z","updated_at":"2017-11-23T13:25:11Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Hello,\r\n\r\nplease see the complete example below for full details.\r\nThe problem is that if in the course of a computation, certain outputs of operators are not used downstream (I call this \"dangling outputs\"), then gradient computation fails IFF dtype != float32.\r\n\r\nI suppose this is because somewhere deep inside, output gradients constant 0 are created for these outputs, but this is done in float32 instead of the type of the output. I think this should be really easy to fix for somebody who knows where this happens (I don't!).\r\n\r\nYou could argue that one should not ever produce dangling outputs. But at least for some operators, this is bound to happen. And in any case, things should not work for float32 and then fail for other dtype, and the user is left with a totally obscure error message (see below).\r\n\r\nThe way around this problem is to make sure there are no dangling outputs, see example below. I'd argue that this is highly obscure, and users should not have to do this.\r\n\r\nHere the complete example. Note that this is *not* specific to linalg.gelqf. I am just using it as an example of an operator with >1 outputs. You can use any other op with >1 output and will get the same error (you can also use an op with 1 output, and then not use it, even though this is arguably pretty dumb).\r\n\r\n--- Code ---\r\n\r\nimport mxnet as mx\r\nimport numpy as np\r\nfrom mxnet import autograd\r\n\r\ndef crit_func(a):\r\n    q, l = mx.nd.linalg.gelqf(a)\r\n    # Could as well write _, l = mx.nd.linalg.gelqf(a)\r\n    # This circumvents the issue, but is highly obscure:\r\n    #bogus = mx.nd.BlockGrad(q) * 0.0\r\n    #return mx.nd.sum(l) + bogus\r\n    return mx.nd.sum(l)\r\n\r\nctx = mx.cpu()\r\n\r\ndtype = np.float32\r\na32 = mx.nd.random.normal(shape=(2, 3), ctx=ctx, dtype=dtype)\r\na32.attach_grad()\r\nwith autograd.record():\r\n    crit32 = crit_func(a32)\r\nprint 'crit32 = %f' % crit32.asscalar()\r\ncrit32.backward()\r\nprint 'grad32\\n', a32.grad.asnumpy()\r\n\r\ndtype = np.float64\r\na64 = mx.nd.Cast(a32, dtype=dtype)\r\na64.attach_grad()\r\nwith autograd.record():\r\n    crit64 = crit_func(a64)\r\nprint 'crit64 = %f' % crit64.asscalar()\r\ncrit64.backward()\r\nprint 'grad64\\n', a64.grad.asnumpy()\r\n\r\n--- Output ---\r\n\r\ncrit32 = -0.633861\r\ngrad32\r\n[[-0.55487823  0.57657677  0.63104177]\r\n [-1.29575324  0.44739273  0.3476553 ]]\r\ncrit64 = -0.633862\r\n\r\n---------------------------------------------------------------------------\r\nMXNetError                                Traceback (most recent call last)\r\n<ipython-input-3-2e10f88a3247> in <module>()\r\n     24     crit64 = crit_func(a64)\r\n     25 print 'crit64 = %f' % crit64.asscalar()\r\n---> 26 crit64.backward()\r\n     27 print 'grad64\\n', a64.grad.asnumpy()\r\n\r\n/Users/matthis/MLBerlinHomes/matthis/virtenvs/linalg_ops/linalg_ops_env/lib/python2.7/site-packages/mxnet-0.12.1-py2.7.egg/mxnet/ndarray/ndarray.pyc in backward(self, out_grad, retain_graph, train_mode)\r\n   1763             ctypes.c_int(train_mode),\r\n   1764             ctypes.c_void_p(0),\r\n-> 1765             ctypes.c_void_p(0)))\r\n   1766 \r\n   1767     def tostype(self, stype):\r\n\r\n/Users/matthis/MLBerlinHomes/matthis/virtenvs/linalg_ops/linalg_ops_env/lib/python2.7/site-packages/mxnet-0.12.1-py2.7.egg/mxnet/base.pyc in check_call(ret)\r\n    144     \"\"\"\r\n    145     if ret != 0:\r\n--> 146         raise MXNetError(py_str(_LIB.MXGetLastError()))\r\n    147 \r\n    148 if sys.version_info[0] < 3:\r\n\r\nMXNetError: [14:09:12] include/mxnet/./tensor_blob.h:216: Check failed: mshadow::DataType<DType>::kFlag == type_flag_ TBlob.get_with_shape: data type do not match specified type.Expected: 0 v.s. given 1\r\n\r\nStack trace returned 10 entries:\r\n[bt] (0) 0   libmxnet.so                         0x000000010c8d8378 _ZN4dmlc15LogMessageFatalD2Ev + 40\r\n[bt] (1) 1   libmxnet.so                         0x000000010c9046d3 _ZNK5mxnet5TBlob4dptrIdEEPT_v + 211\r\n[bt] (2) 2   libmxnet.so                         0x000000010c904041 _ZNK5mxnet5TBlob14get_with_shapeIN7mshadow3cpuELi3EdEENS2_6TensorIT_XT0_ET1_EERKNS2_5ShapeIXT0_EEEPNS2_6StreamIS5_EE + 817\r\n[bt] (3) 3   libmxnet.so                         0x000000010d719ae7 _ZNK5mxnet5TBlob8FlatToKDIN7mshadow3cpuELi3EdEENS2_6TensorIT_XT0_ET1_EEPNS2_6StreamIS5_EE + 839\r\n[bt] (4) 4   libmxnet.so                         0x000000010d713248 _ZN5mxnet2op12LaOpBackwardIN7mshadow3cpuELi2ELi2ELi4ELi1ENS0_14gelqf_backwardEEEvRKN4nnvm9NodeAttrsERKNS_9OpContextERKNSt3__16vectorINS_5TBlobENSC_9allocatorISE_EEEERKNSD_INS_9OpReqTypeENSF_ISK_EEEESJ_ + 1464\r\n[bt] (5) 5   libmxnet.so                         0x000000010d8aed74 _ZZN5mxnet10imperative12PushFComputeERKNSt3__18functionIFvRKN4nnvm9NodeAttrsERKNS_9OpContextERKNS1_6vectorINS_5TBlobENS1_9allocatorISB_EEEERKNSA_INS_9OpReqTypeENSC_ISH_EEEESG_EEEPKNS3_2OpES6_RKNS_7ContextERKNSA_IPNS_6engine3VarENSC_ISY_EEEES12_RKNSA_INS_8ResourceENSC_IS13_EEEERKNSA_IPNS_7NDArrayENSC_IS19_EEEES1D_RKNSA_IjNSC_IjEEEESL_ENKUlNS_10RunContextENSW_18CallbackOnCompleteEE_clES1I_S1J_ + 532\r\n[bt] (6) 6   libmxnet.so                         0x000000010d8ae0f7 _ZNSt3__110__function6__funcIZN5mxnet10imperative12PushFComputeERKNS_8functionIFvRKN4nnvm9NodeAttrsERKNS2_9OpContextERKNS_6vectorINS2_5TBlobENS_9allocatorISD_EEEERKNSC_INS2_9OpReqTypeENSE_ISJ_EEEESI_EEEPKNS5_2OpES8_RKNS2_7ContextERKNSC_IPNS2_6engine3VarENSE_IS10_EEEES14_RKNSC_INS2_8ResourceENSE_IS15_EEEERKNSC_IPNS2_7NDArrayENSE_IS1B_EEEES1F_RKNSC_IjNSE_IjEEEESN_EUlNS2_10RunContextENSY_18CallbackOnCompleteEE_NSE_IS1M_EEFvS1K_S1L_EEclEOS1K_OS1L_ + 55\r\n[bt] (7) 7   libmxnet.so                         0x000000010d839cf2 _ZN5mxnet6engine11NaiveEngine9PushAsyncENSt3__18functionIFvNS_10RunContextENS0_18CallbackOnCompleteEEEENS_7ContextERKNS2_6vectorIPNS0_3VarENS2_9allocatorISB_EEEESG_NS_10FnPropertyEiPKc + 194\r\n[bt] (8) 8   libmxnet.so                         0x000000010d89d3c2 _ZN5mxnet10imperative12PushFComputeERKNSt3__18functionIFvRKN4nnvm9NodeAttrsERKNS_9OpContextERKNS1_6vectorINS_5TBlobENS1_9allocatorISB_EEEERKNSA_INS_9OpReqTypeENSC_ISH_EEEESG_EEEPKNS3_2OpES6_RKNS_7ContextERKNSA_IPNS_6engine3VarENSC_ISY_EEEES12_RKNSA_INS_8ResourceENSC_IS13_EEEERKNSA_IPNS_7NDArrayENSC_IS19_EEEES1D_RKNSA_IjNSC_IjEEEESL_ + 1186\r\n[bt] (9) 9   libmxnet.so                         0x000000010d89b501 _ZN5mxnet10Imperative8InvokeOpERKNS_7ContextERKN4nnvm9NodeAttrsERKNSt3__16vectorIPNS_7NDArrayENS8_9allocatorISB_EEEESG_RKNS9_INS_9OpReqTypeENSC_ISH_EEEENS_12DispatchModeENS_10OpStatePtrE + 1137\r\n\r\n\r\n"}}
{"actor":{"display_login":"mseeger"}}