{"action":"created","issue":{"url":"https://api.github.com/repos/kratzert/finetune_alexnet_with_tensorflow/issues/28","repository_url":"https://api.github.com/repos/kratzert/finetune_alexnet_with_tensorflow","labels_url":"https://api.github.com/repos/kratzert/finetune_alexnet_with_tensorflow/issues/28/labels{/name}","comments_url":"https://api.github.com/repos/kratzert/finetune_alexnet_with_tensorflow/issues/28/comments","events_url":"https://api.github.com/repos/kratzert/finetune_alexnet_with_tensorflow/issues/28/events","html_url":"https://github.com/kratzert/finetune_alexnet_with_tensorflow/issues/28","id":273161754,"number":28,"title":"error occurs when running jupyter file on my linux server","user":{"login":"GraysonLiu","id":14071310,"avatar_url":"https://avatars2.githubusercontent.com/u/14071310?v=4","gravatar_id":"","url":"https://api.github.com/users/GraysonLiu","html_url":"https://github.com/GraysonLiu","followers_url":"https://api.github.com/users/GraysonLiu/followers","following_url":"https://api.github.com/users/GraysonLiu/following{/other_user}","gists_url":"https://api.github.com/users/GraysonLiu/gists{/gist_id}","starred_url":"https://api.github.com/users/GraysonLiu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GraysonLiu/subscriptions","organizations_url":"https://api.github.com/users/GraysonLiu/orgs","repos_url":"https://api.github.com/users/GraysonLiu/repos","events_url":"https://api.github.com/users/GraysonLiu/events{/privacy}","received_events_url":"https://api.github.com/users/GraysonLiu/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-11-11T17:34:40Z","updated_at":"2017-11-11T17:43:06Z","closed_at":null,"author_association":"NONE","body":"I want to run file `validate_alexnet_on_imagenet.ipynb` to test the implement of AlexNet. When I run it on my Windows PC, it works well. But when I run it on my Linux server, which is much more powerful than my PC, it's stuck for a while and output a `ResourceExhaustedError`, which I believe occurred in this line:\r\n```\r\nsess.run(tf.global_variables_initializer())\r\n```\r\nFull output is shown below:\r\n```\r\n---------------------------------------------------------------------------\r\nResourceExhaustedError                    Traceback (most recent call last)\r\n/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)\r\n   1322     try:\r\n-> 1323       return fn(*args)\r\n   1324     except errors.OpError as e:\r\n\r\n/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)\r\n   1301                                    feed_dict, fetch_list, target_list,\r\n-> 1302                                    status, run_metadata)\r\n   1303 \r\n\r\n/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py in __exit__(self, type_arg, value_arg, traceback_arg)\r\n    472             compat.as_text(c_api.TF_Message(self.status.status)),\r\n--> 473             c_api.TF_GetCode(self.status.status))\r\n    474     # Delete the underlying status object from memory otherwise it stays alive\r\n\r\nResourceExhaustedError: OOM when allocating tensor with shape[4096]\r\n\t [[Node: fc6/biases/Initializer/random_uniform/RandomUniform = RandomUniform[T=DT_INT32, _class=[\"loc:@fc6/biases\"], dtype=DT_FLOAT, seed=0, seed2=0, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](fc6/biases/Initializer/random_uniform/shape)]]\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nResourceExhaustedError                    Traceback (most recent call last)\r\n<ipython-input-4-eb8e619c57ae> in <module>()\r\n      2 \r\n      3     # Initialize all variables\r\n----> 4     sess.run(tf.global_variables_initializer())\r\n      5 \r\n      6     # Load the pretrained weights into the model\r\n\r\n/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)\r\n    887     try:\r\n    888       result = self._run(None, fetches, feed_dict, options_ptr,\r\n--> 889                          run_metadata_ptr)\r\n    890       if run_metadata:\r\n    891         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)\r\n\r\n/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)\r\n   1118     if final_fetches or final_targets or (handle and feed_dict_tensor):\r\n   1119       results = self._do_run(handle, final_targets, final_fetches,\r\n-> 1120                              feed_dict_tensor, options, run_metadata)\r\n   1121     else:\r\n   1122       results = []\r\n\r\n/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)\r\n   1315     if handle is None:\r\n   1316       return self._do_call(_run_fn, self._session, feeds, fetches, targets,\r\n-> 1317                            options, run_metadata)\r\n   1318     else:\r\n   1319       return self._do_call(_prun_fn, self._session, handle, feeds, fetches)\r\n\r\n/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)\r\n   1334         except KeyError:\r\n   1335           pass\r\n-> 1336       raise type(e)(node_def, op, message)\r\n   1337 \r\n   1338   def _extend_graph(self):\r\n\r\nResourceExhaustedError: OOM when allocating tensor with shape[4096]\r\n\t [[Node: fc6/biases/Initializer/random_uniform/RandomUniform = RandomUniform[T=DT_INT32, _class=[\"loc:@fc6/biases\"], dtype=DT_FLOAT, seed=0, seed2=0, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](fc6/biases/Initializer/random_uniform/shape)]]\r\n\r\nCaused by op 'fc6/biases/Initializer/random_uniform/RandomUniform', defined at:\r\n  File \"/usr/lib/python3.5/runpy.py\", line 193, in _run_module_as_main\r\n    \"__main__\", mod_spec)\r\n  File \"/usr/lib/python3.5/runpy.py\", line 85, in _run_code\r\n    exec(code, run_globals)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/ipykernel_launcher.py\", line 16, in <module>\r\n    app.launch_new_instance()\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/traitlets/config/application.py\", line 658, in launch_instance\r\n    app.start()\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/ipykernel/kernelapp.py\", line 477, in start\r\n    ioloop.IOLoop.instance().start()\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/zmq/eventloop/ioloop.py\", line 177, in start\r\n    super(ZMQIOLoop, self).start()\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/tornado/ioloop.py\", line 888, in start\r\n    handler_func(fd_obj, events)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/tornado/stack_context.py\", line 277, in null_wrapper\r\n    return fn(*args, **kwargs)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py\", line 440, in _handle_events\r\n    self._handle_recv()\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py\", line 472, in _handle_recv\r\n    self._run_callback(callback, msg)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/zmq/eventloop/zmqstream.py\", line 414, in _run_callback\r\n    callback(*args, **kwargs)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/tornado/stack_context.py\", line 277, in null_wrapper\r\n    return fn(*args, **kwargs)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/ipykernel/kernelbase.py\", line 283, in dispatcher\r\n    return self.dispatch_shell(stream, msg)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/ipykernel/kernelbase.py\", line 235, in dispatch_shell\r\n    handler(stream, idents, msg)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/ipykernel/kernelbase.py\", line 399, in execute_request\r\n    user_expressions, allow_stdin)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/ipykernel/ipkernel.py\", line 196, in do_execute\r\n    res = shell.run_cell(code, store_history=store_history, silent=silent)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/ipykernel/zmqshell.py\", line 533, in run_cell\r\n    return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/IPython/core/interactiveshell.py\", line 2728, in run_cell\r\n    interactivity=interactivity, compiler=compiler, result=result)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/IPython/core/interactiveshell.py\", line 2850, in run_ast_nodes\r\n    if self.run_code(code, result):\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/IPython/core/interactiveshell.py\", line 2910, in run_code\r\n    exec(code_obj, self.user_global_ns, self.user_ns)\r\n  File \"<ipython-input-3-5d95ec9c311f>\", line 9, in <module>\r\n    model = AlexNet(x, keep_prob, 1000, [])\r\n  File \"/home1/grayson/finetune_alexnet_with_tensorflow/alexnet.py\", line 56, in __init__\r\n    self.create()\r\n  File \"/home1/grayson/finetune_alexnet_with_tensorflow/alexnet.py\", line 82, in create\r\n    fc6 = fc(flattened, 6*6*256, 4096, name='fc6')\r\n  File \"/home1/grayson/finetune_alexnet_with_tensorflow/alexnet.py\", line 177, in fc\r\n    biases = tf.get_variable('biases', [num_out], trainable=True)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/variable_scope.py\", line 1203, in get_variable\r\n    constraint=constraint)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/variable_scope.py\", line 1092, in get_variable\r\n    constraint=constraint)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/variable_scope.py\", line 425, in get_variable\r\n    constraint=constraint)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/variable_scope.py\", line 394, in _true_getter\r\n    use_resource=use_resource, constraint=constraint)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/variable_scope.py\", line 805, in _get_single_variable\r\n    constraint=constraint)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/variables.py\", line 213, in __init__\r\n    constraint=constraint)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/variables.py\", line 303, in _init_from_args\r\n    initial_value(), name=\"initial_value\", dtype=dtype)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/variable_scope.py\", line 779, in <lambda>\r\n    shape.as_list(), dtype=dtype, partition_info=partition_info)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/init_ops.py\", line 445, in __call__\r\n    shape, -limit, limit, dtype, seed=self.seed)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/random_ops.py\", line 240, in random_uniform\r\n    shape, dtype, seed=seed1, seed2=seed2)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/ops/gen_random_ops.py\", line 473, in _random_uniform\r\n    name=name)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py\", line 787, in _apply_op_helper\r\n    op_def=op_def)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/ops.py\", line 2956, in create_op\r\n    op_def=op_def)\r\n  File \"/home1/grayson/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/ops.py\", line 1470, in __init__\r\n    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access\r\n\r\nResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[4096]\r\n\t [[Node: fc6/biases/Initializer/random_uniform/RandomUniform = RandomUniform[T=DT_INT32, _class=[\"loc:@fc6/biases\"], dtype=DT_FLOAT, seed=0, seed2=0, _device=\"/job:localhost/replica:0/task:0/device:GPU:0\"](fc6/biases/Initializer/random_uniform/shape)]]\r\n```\r\nAnyone has an idea about it?"},"comment":{"url":"https://api.github.com/repos/kratzert/finetune_alexnet_with_tensorflow/issues/comments/343681330","html_url":"https://github.com/kratzert/finetune_alexnet_with_tensorflow/issues/28#issuecomment-343681330","issue_url":"https://api.github.com/repos/kratzert/finetune_alexnet_with_tensorflow/issues/28","id":343681330,"user":{"login":"GraysonLiu","id":14071310,"avatar_url":"https://avatars2.githubusercontent.com/u/14071310?v=4","gravatar_id":"","url":"https://api.github.com/users/GraysonLiu","html_url":"https://github.com/GraysonLiu","followers_url":"https://api.github.com/users/GraysonLiu/followers","following_url":"https://api.github.com/users/GraysonLiu/following{/other_user}","gists_url":"https://api.github.com/users/GraysonLiu/gists{/gist_id}","starred_url":"https://api.github.com/users/GraysonLiu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GraysonLiu/subscriptions","organizations_url":"https://api.github.com/users/GraysonLiu/orgs","repos_url":"https://api.github.com/users/GraysonLiu/repos","events_url":"https://api.github.com/users/GraysonLiu/events{/privacy}","received_events_url":"https://api.github.com/users/GraysonLiu/received_events","type":"User","site_admin":false},"created_at":"2017-11-11T17:43:06Z","updated_at":"2017-11-11T17:43:06Z","author_association":"NONE","body":"I don't think that's the problem. The GPU on my PC is just GTX 950M. But it's Titan X on my server. Besides, \r\nI don't even train the network, only testing the implement of the model, which I think doesn't need much computation power."}}
{"actor":{"display_login":"GraysonLiu"}}