{"action":"created","issue":{"url":"https://api.github.com/repos/Theano/Theano/issues/6543","repository_url":"https://api.github.com/repos/Theano/Theano","labels_url":"https://api.github.com/repos/Theano/Theano/issues/6543/labels{/name}","comments_url":"https://api.github.com/repos/Theano/Theano/issues/6543/comments","events_url":"https://api.github.com/repos/Theano/Theano/issues/6543/events","html_url":"https://github.com/Theano/Theano/issues/6543","id":284291577,"number":6543,"title":"CUDA_ERROR_LAUNCH_FAILED: unspecified launch failure","user":{"login":"guoxuesong","id":9998655,"avatar_url":"https://avatars3.githubusercontent.com/u/9998655?v=4","gravatar_id":"","url":"https://api.github.com/users/guoxuesong","html_url":"https://github.com/guoxuesong","followers_url":"https://api.github.com/users/guoxuesong/followers","following_url":"https://api.github.com/users/guoxuesong/following{/other_user}","gists_url":"https://api.github.com/users/guoxuesong/gists{/gist_id}","starred_url":"https://api.github.com/users/guoxuesong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guoxuesong/subscriptions","organizations_url":"https://api.github.com/users/guoxuesong/orgs","repos_url":"https://api.github.com/users/guoxuesong/repos","events_url":"https://api.github.com/users/guoxuesong/events{/privacy}","received_events_url":"https://api.github.com/users/guoxuesong/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-12-23T07:17:41Z","updated_at":"2017-12-24T07:19:11Z","closed_at":null,"author_association":"NONE","body":"Looks like something wrong inside gpuarray:\r\n\r\n```\r\nFile \"/mnt/work/deepstacks/framework/main.py\", line 2281, in main\r\nrun(args)\r\nFile \"/mnt/work/deepstacks/framework/main.py\", line 1905, in run\r\nres=train_fn(*sorted_values(batch))\r\nFile \"/opt/Theano/theano/compile/function_module.py\", line 917, in __call__\r\nstorage_map=getattr(self.fn, 'storage_map', None))\r\nFile \"/opt/Theano/theano/gof/link.py\", line 174, in raise_with_op\r\ndetailed_err_msg += \"\\nOutputs clients: %s\\n\" % clients\r\nFile \"/opt/Theano/theano/gof/graph.py\", line 187, in __repr__\r\nreturn str(self)\r\nFile \"/opt/Theano/theano/gof/graph.py\", line 184, in __str__\r\nreturn op_as_string(self.inputs, self)\r\nFile \"/opt/Theano/theano/gof/graph.py\", line 1304, in op_as_string\r\nstrs = as_string(i, op.inputs, leaf_formatter, node_formatter)\r\nFile \"/opt/Theano/theano/gof/graph.py\", line 1384, in as_string\r\nreturn [describe(output) for output in o]\r\nFile \"/opt/Theano/theano/gof/graph.py\", line 1382, in describe\r\nreturn leaf_formatter(r)\r\nFile \"/opt/Theano/theano/gpuarray/type.py\", line 607, in __str__\r\nreturn \"GpuArrayConstant{%s}\" % np_data\r\nFile \"pygpu/gpuarray.pyx\", line 2252, in pygpu.gpuarray.GpuArray.__str__ (pygpu/gpuarray.c:28692)\r\nFile \"/usr/lib/python2.7/dist-packages/numpy/core/numeric.py\", line 482, in asarray\r\nreturn array(a, dtype, copy=False, order=order)\r\nFile \"pygpu/gpuarray.pyx\", line 1728, in pygpu.gpuarray.GpuArray.__array__ (pygpu/gpuarray.c:21507)\r\nFile \"pygpu/gpuarray.pyx\", line 1398, in pygpu.gpuarray._pygpu_as_ndarray (pygpu/gpuarray.c:18117)\r\nFile \"pygpu/gpuarray.pyx\", line 394, in pygpu.gpuarray.array_read (pygpu/gpuarray.c:6885)\r\npygpu.gpuarray.GpuArrayException: cuMemcpyDtoHAsync(dst, src->ptr + srcoff, sz, ctx->mem_s): CUDA_ERROR_LAUNCH_FAILED: unspecified launch failure\r\n\r\n```\r\n\r\nTheano and libgpuarray pull from github at 20171222"},"comment":{"url":"https://api.github.com/repos/Theano/Theano/issues/comments/353769563","html_url":"https://github.com/Theano/Theano/issues/6543#issuecomment-353769563","issue_url":"https://api.github.com/repos/Theano/Theano/issues/6543","id":353769563,"user":{"login":"guoxuesong","id":9998655,"avatar_url":"https://avatars3.githubusercontent.com/u/9998655?v=4","gravatar_id":"","url":"https://api.github.com/users/guoxuesong","html_url":"https://github.com/guoxuesong","followers_url":"https://api.github.com/users/guoxuesong/followers","following_url":"https://api.github.com/users/guoxuesong/following{/other_user}","gists_url":"https://api.github.com/users/guoxuesong/gists{/gist_id}","starred_url":"https://api.github.com/users/guoxuesong/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guoxuesong/subscriptions","organizations_url":"https://api.github.com/users/guoxuesong/orgs","repos_url":"https://api.github.com/users/guoxuesong/repos","events_url":"https://api.github.com/users/guoxuesong/events{/privacy}","received_events_url":"https://api.github.com/users/guoxuesong/received_events","type":"User","site_admin":false},"created_at":"2017-12-24T07:19:10Z","updated_at":"2017-12-24T07:19:10Z","author_association":"NONE","body":"I modified Theano/theano/gpuarray/type.py like this:\r\n\r\n```\r\n    def __str__(self):\r\n        if self.name is not None:\r\n            return self.name\r\n        try:\r\n            np_data = np.asarray(self.data)\r\n        except gpuarray.GpuArrayException:\r\n            #np_data = self.data\r\n            np_data = 'Unknown'\r\n        return \"GpuArrayConstant{%s}\" % np_data\r\n```\r\n\r\nNow I get:\r\n\r\n```\r\nFile \"/mnt/work/deepstacks/framework/main.py\", line 2281, in main\r\nrun(args)\r\nFile \"/mnt/work/deepstacks/framework/main.py\", line 1905, in run\r\nres=train_fn(*sorted_values(batch))\r\nFile \"/opt/Theano/theano/compile/function_module.py\", line 917, in __call__\r\nstorage_map=getattr(self.fn, 'storage_map', None))\r\nFile \"/opt/Theano/theano/gof/link.py\", line 325, in raise_with_op\r\nreraise(exc_type, exc_value, exc_trace)\r\nFile \"/opt/Theano/theano/compile/function_module.py\", line 903, in __call__\r\nself.fn() if output_subset is None else\\\r\nRuntimeError: Could not read device data.\r\nApply node that caused the error: HostFromGpu(gpuarray)(GpuElemwise{add,no_inplace}.0)\r\nToposort index: 5608\r\nInputs types: [GpuArrayType<None>(float32, scalar)]\r\nInputs shapes: [()]\r\nInputs strides: [()]\r\nInputs values: [gpuarray.array(<content not available>)]\r\nOutputs clients: [['output']]\r\nHINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.\r\nHINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.\r\n```\r\n"}}
{"actor":{"display_login":"guoxuesong"}}