{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/9091","repository_url":"https://api.github.com/repos/tensorflow/tensorflow","labels_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/9091/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/9091/comments","events_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/9091/events","html_url":"https://github.com/tensorflow/tensorflow/issues/9091","id":220500375,"number":9091,"title":"Memory Leak Running simple feed_dict graph","user":{"login":"JonathanRaiman","id":1411079,"avatar_url":"https://avatars1.githubusercontent.com/u/1411079?v=4","gravatar_id":"","url":"https://api.github.com/users/JonathanRaiman","html_url":"https://github.com/JonathanRaiman","followers_url":"https://api.github.com/users/JonathanRaiman/followers","following_url":"https://api.github.com/users/JonathanRaiman/following{/other_user}","gists_url":"https://api.github.com/users/JonathanRaiman/gists{/gist_id}","starred_url":"https://api.github.com/users/JonathanRaiman/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JonathanRaiman/subscriptions","organizations_url":"https://api.github.com/users/JonathanRaiman/orgs","repos_url":"https://api.github.com/users/JonathanRaiman/repos","events_url":"https://api.github.com/users/JonathanRaiman/events{/privacy}","received_events_url":"https://api.github.com/users/JonathanRaiman/received_events","type":"User","site_admin":false},"labels":[{"id":404586594,"url":"https://api.github.com/repos/tensorflow/tensorflow/labels/stat:awaiting%20tensorflower","name":"stat:awaiting tensorflower","color":"b60205","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-04-09T21:26:46Z","updated_at":"2017-08-01T20:30:28Z","closed_at":null,"body":"In a series simple tensorflow programs I obtain memory leaks (unbounded growth of CPU memory).\r\nOn original program on a computer with 64GB of RAM this leak is about 640 megabytes per hour (1% of total memory).\r\n\r\n### Plots of computer's memory over time:\r\n#### Long time scale picture:\r\n![unknown-1](https://cloud.githubusercontent.com/assets/1411079/24841025/1064f380-1d2f-11e7-8738-943be211bb1b.png)\r\n####  short time scale picture:\r\n![unknown](https://cloud.githubusercontent.com/assets/1411079/24841026/12d78984-1d2f-11e7-8efd-62bd8e46b46c.png)\r\n\r\n\r\n## Problem description\r\n\r\nThe original program was more advanced and included RNNs/Saving/Loading etc.. but I \"narrowed it down\" to a simple for loop with no gradient descent where memory grows over time without bound. \r\nTested on Fedora 25 and Mac OSX 10.11.5. Issue occurs when running on single GPU (Titan X Pascal) or on CPU. Varying the sizes of the variables in the graph only changes the degree of growth, but does not prevent the effect from occurring. This issue occurs on tensorflow 0.12 and on current tensorflow 1.0.1. No custom code was used. Tensorflow was installed using pip in both cases (pre-compiled binary. Each time this was `pip3 install tensorflow-gpu`). Using CUDA 8.0,  CuDNN v5 [though this should not impact the use-case, since no cudnn kernels are being used]. GPU is a Titan X Pascal 12GB of VRAM (not Titan Xp).\r\n\r\n## To reproduce:\r\n\r\n```\r\nimport argparse\r\nimport psutil\r\n\r\nfrom os import getpid\r\nimport tensorflow as tf\r\nimport numpy as np\r\n\r\ndef fc(inputs, output_size):\r\n    with tf.variable_scope(\"FC\"):\r\n        input_size = inputs.get_shape()[-1].value\r\n        W = tf.get_variable(\"W\", shape=[input_size, output_size])\r\n        b = tf.get_variable(\"b\", shape=[output_size], initializer=tf.constant_initializer(0))\r\n        out = tf.nn.xw_plus_b(inputs, W, b)\r\n    return out\r\n\r\ndef create_model(input_size, output_size):\r\n    # model placeholders:\r\n    with tf.variable_scope(\"Inputs\"):\r\n        input_placeholder = tf.placeholder(\r\n            tf.float32, [None, input_size], name=\"input_placeholder\"\r\n        )\r\n    # meaningless function of inputs\r\n    op = tf.reduce_mean(tf.reduce_sum(fc(input_placeholder, output_size), 1))\r\n    return input_placeholder, op\r\n\r\ndef parse_args(args=None):\r\n    parser = argparse.ArgumentParser()\r\n    parser.add_argument('--max_epochs', type=int, default=1000)\r\n    parser.add_argument('--batch_size', type=int, default=7000)\r\n    parser.add_argument('--input_size', type=int, default=100)\r\n    parser.add_argument('--output_size', type=int, default=100)\r\n    parser.add_argument('--device', type=str, default=\"gpu:0\")\r\n    return parser.parse_args(args=args)\r\n\r\ndef create_batches(inputs, input_size, batch_size, n):\r\n    batches = []\r\n    for i in range(n):\r\n        X = np.random.uniform(-1.0, 1.0, size=(batch_size, input_size))\r\n        batches.append({inputs: X})\r\n    return batches\r\n\r\ndef main():\r\n    args = parse_args()\r\n    session_conf = tf.ConfigProto(allow_soft_placement=True)\r\n    np.random.seed(1234)\r\n    process = psutil.Process(getpid())\r\n\r\n    with tf.Session(config=session_conf) as session, tf.device(args.device):\r\n        inputs, op = create_model(args.input_size, args.output_size)\r\n        session.run(tf.global_variables_initializer())\r\n        batches = create_batches(inputs, args.input_size, args.batch_size, 20)\r\n\r\n        for epoch in range(args.max_epochs):\r\n            before = process.memory_percent()\r\n            for feed_dict in batches:\r\n                session.run(op, feed_dict)\r\n            after = process.memory_percent()\r\n            print(\"MEMORY CHANGE %.4f -> %.4f\" % (before, after))\r\n\r\nif __name__ == \"__main__\":\r\n    main()\r\n```\r\n\r\nOutput will be (exact numbers are percentages of computer's ram, so should change based on hardware, but main point is that memory continues to grow when the program has no variation between graph runs, batches are all the same size, no randomness is left in the program, etc.):\r\n\r\n```\r\nMEMORY CHANGE 1.2427 -> 1.3101\r\nMEMORY CHANGE 1.3101 -> 1.3103\r\nMEMORY CHANGE 1.3103 -> 1.3104\r\nMEMORY CHANGE 1.3104 -> 1.3106\r\nMEMORY CHANGE 1.3106 -> 1.3108\r\nMEMORY CHANGE 1.3108 -> 1.3108\r\nMEMORY CHANGE 1.3108 -> 1.3108\r\n...\r\nMEMORY CHANGE 1.3108 -> 1.3109\r\n...\r\nMEMORY CHANGE 1.3109 -> 1.3110\r\n...\r\n```\r\n\r\nHow can I fix this? I currently suspect a CPU memory pool issue inside tensorflow since the problem is fairly generic, and does not depend on the ops inside the graph (much). From what I've gathered most likely candidate is the `tf.asarray`/copying of numpy arrays in `feed_dict`, leading to memory fragmentation etc. Supposing this were the case, I've heard that `tcmalloc` should alleviate this, but no dice (note: I've also checked that `objgraph` shows no growth in program over time).\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/tensorflow/tensorflow/issues/comments/319487420","html_url":"https://github.com/tensorflow/tensorflow/issues/9091#issuecomment-319487420","issue_url":"https://api.github.com/repos/tensorflow/tensorflow/issues/9091","id":319487420,"user":{"login":"jstaker7","id":1364252,"avatar_url":"https://avatars2.githubusercontent.com/u/1364252?v=4","gravatar_id":"","url":"https://api.github.com/users/jstaker7","html_url":"https://github.com/jstaker7","followers_url":"https://api.github.com/users/jstaker7/followers","following_url":"https://api.github.com/users/jstaker7/following{/other_user}","gists_url":"https://api.github.com/users/jstaker7/gists{/gist_id}","starred_url":"https://api.github.com/users/jstaker7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jstaker7/subscriptions","organizations_url":"https://api.github.com/users/jstaker7/orgs","repos_url":"https://api.github.com/users/jstaker7/repos","events_url":"https://api.github.com/users/jstaker7/events{/privacy}","received_events_url":"https://api.github.com/users/jstaker7/received_events","type":"User","site_admin":false},"created_at":"2017-08-01T20:30:28Z","updated_at":"2017-08-01T20:30:28Z","body":"This is becoming problematic for me as well. I also use non-fixed input dimensions.\r\n\r\nRelated: \r\nhttps://github.com/tensorflow/tensorflow/issues/8560\r\nhttps://stackoverflow.com/questions/42861956/gpu-poolallocator-explodes-the-cpu-memory"}}
{"actor":{"display_login":"jstaker7"}}