{"action":"created","issue":{"url":"https://api.github.com/repos/aerospike/aerospike-client-python/issues/142","repository_url":"https://api.github.com/repos/aerospike/aerospike-client-python","labels_url":"https://api.github.com/repos/aerospike/aerospike-client-python/issues/142/labels{/name}","comments_url":"https://api.github.com/repos/aerospike/aerospike-client-python/issues/142/comments","events_url":"https://api.github.com/repos/aerospike/aerospike-client-python/issues/142/events","html_url":"https://github.com/aerospike/aerospike-client-python/issues/142","id":204901411,"number":142,"title":"Possible memory leak in Python driver ordered map insert","user":{"login":"vegabook","id":3780883,"avatar_url":"https://avatars.githubusercontent.com/u/3780883?v=3","gravatar_id":"","url":"https://api.github.com/users/vegabook","html_url":"https://github.com/vegabook","followers_url":"https://api.github.com/users/vegabook/followers","following_url":"https://api.github.com/users/vegabook/following{/other_user}","gists_url":"https://api.github.com/users/vegabook/gists{/gist_id}","starred_url":"https://api.github.com/users/vegabook/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vegabook/subscriptions","organizations_url":"https://api.github.com/users/vegabook/orgs","repos_url":"https://api.github.com/users/vegabook/repos","events_url":"https://api.github.com/users/vegabook/events{/privacy}","received_events_url":"https://api.github.com/users/vegabook/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-02-02T15:12:43Z","updated_at":"2017-02-03T05:12:04Z","closed_at":null,"body":"Refers to this Community board entry: \r\nhttps://discuss.aerospike.com/t/large-memory-usage-by-python-driver-on-big-batch-insert/3819\r\n\r\nThere appears to be growing memory usage, an order of magnitude greater than the space occupied by the data itself, by the Python driver when inserting into an ordered map. Here is a simplified version of the code from the Community forum link above. This will insert 100 million floats, batched by a ticker code and an integer, into ordered maps. The memory used by this piece of code grows enormously as the inserts progress. After 10 million inserts, AMC shows Aerospike test db using only 208MB, whereas this piece of code's memory footprint has grown to 1.5GB. And it continues upwards from there. \r\n\r\n### UPDATE: This appears only to affect the python 3 driver. Tested with version 2.0.6. \r\nThis code will work with both python 2.7 and 3.x. Just need to install numpy and psutil. \r\n```\r\n\r\n\r\nfrom __future__ import print_function\r\nimport aerospike\r\nimport numpy as np\r\nfrom datetime import datetime\r\nimport os\r\nimport psutil   \r\n\r\nif __name__ == \"__main__\":\r\n    thisprocess = psutil.Process(os.getpid())\r\n    config = {\r\n        'hosts': [ ('localhost', 3000) ]\r\n    }\r\n    client = aerospike.client(config).connect()\r\n    map_policy={'map_order':aerospike.MAP_KEY_VALUE_ORDERED}\r\n    num_tickers = 1000 # number of individual securities\r\n    num_periods = 24*30 # \r\n    total_entries = int(100e6)\r\n    entries_per_period = int(total_entries / num_periods)\r\n    this_period = 0\r\n    timer = datetime.utcnow()\r\n    counter = 0\r\n\r\n    # set the map policies\r\n    for period in range(num_periods):\r\n        for ticker in range(num_tickers):\r\n            key = (\"test\", \"tickermap\", str(ticker) + \":\" + str(period))\r\n            client.map_set_policy(key, \"ticks\", map_policy)\r\n\r\n    # insert\r\n    for period in range(num_periods):\r\n        for entry in range(entries_per_period):\r\n            ticker = str(np.random.choice(num_tickers)) + \":\" + str(period)\r\n            key = (\"test\", \"tickermap\", ticker)\r\n            client.map_put(key, \"ticks\", this_period, np.random.uniform())\r\n            this_period += 1\r\n            counter += 1\r\n            if(datetime.utcnow() - timer).seconds > 0:\r\n                print(counter, \"per second, total insertions:\", this_period, \" GB used:\",\\\r\n                        thisprocess.memory_info().rss / 1e9)\r\n                counter = 0\r\n                timer = datetime.utcnow()\r\n"},"comment":{"url":"https://api.github.com/repos/aerospike/aerospike-client-python/issues/comments/277166280","html_url":"https://github.com/aerospike/aerospike-client-python/issues/142#issuecomment-277166280","issue_url":"https://api.github.com/repos/aerospike/aerospike-client-python/issues/142","id":277166280,"user":{"login":"pygupta","id":4228346,"avatar_url":"https://avatars.githubusercontent.com/u/4228346?v=3","gravatar_id":"","url":"https://api.github.com/users/pygupta","html_url":"https://github.com/pygupta","followers_url":"https://api.github.com/users/pygupta/followers","following_url":"https://api.github.com/users/pygupta/following{/other_user}","gists_url":"https://api.github.com/users/pygupta/gists{/gist_id}","starred_url":"https://api.github.com/users/pygupta/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pygupta/subscriptions","organizations_url":"https://api.github.com/users/pygupta/orgs","repos_url":"https://api.github.com/users/pygupta/repos","events_url":"https://api.github.com/users/pygupta/events{/privacy}","received_events_url":"https://api.github.com/users/pygupta/received_events","type":"User","site_admin":false},"created_at":"2017-02-03T05:12:04Z","updated_at":"2017-02-03T05:12:04Z","body":"I tested the above code up to 25 million inserts. I did not see the 10x increase in total memory. The memory consumed by the application code stays stable, while memory consumed by the server grows with the data, almost one to one.\r\n\r\nSummary of test on Ubuntu 16.04 VM, running CE 3.11.0.1\r\n1 - Boot up VM from shutdown, open terminal, start htop: Tot mem: 663M\r\n2 - Start second terminal, launch aerospike single node server: 811M\r\n3 - Start AMC: 838M\r\n4 - Start firefox, launch AMC dashboard: 1.08G **total**, 0 Mb **server**\r\n5 - Start above code (memleaktest.py) : 1.09G / 0Mb\r\n6 - All 720K keys inserted - first code loop: 1.16G/62.9M\r\n7 - 5 mill insert updates to maps: 1.36G/142M\r\n8 - 10 mill insert updates to maps: 1.45G/224M\r\n(Steps 1 -8 were repeated twice - shutdown, reboot VM, similar data)\r\n9 - 20 mill insert updates to maps: 1.63G/385M\r\nmemleaktest.py mem usage remains at 1.2% to total memory throughout.\r\n\r\n10 - 25 million insert updates to maps: 1.71G/466M\r\n\r\nI have screen shots for all steps 1-9. \r\n\r\nStep#5: \r\n![screen shot 2017-02-02 at 8 39 43 pm](https://cloud.githubusercontent.com/assets/4228346/22580225/6ab4831e-e98b-11e6-9f02-58fdd552fec0.png)\r\n\r\nStep#6:\r\n![screen shot 2017-02-02 at 8 40 45 pm](https://cloud.githubusercontent.com/assets/4228346/22580232/755b8e7a-e98b-11e6-93cd-81992f1ed236.png)\r\n\r\nStep#7: (5 mill)\r\n\r\n![screen shot 2017-02-02 at 8 46 32 pm](https://cloud.githubusercontent.com/assets/4228346/22580235/83cd628a-e98b-11e6-9632-a39abe1a8008.png)\r\n\r\nStep#8: (10 mill)\r\n![screen shot 2017-02-02 at 8 52 09 pm](https://cloud.githubusercontent.com/assets/4228346/22580250/9d137e14-e98b-11e6-8e59-d304373b0f9f.png)\r\n\r\nStep #9: (20 mill)\r\n![screen shot 2017-02-02 at 9 03 36 pm](https://cloud.githubusercontent.com/assets/4228346/22580255/a6f21346-e98b-11e6-8f19-59f55f24a0c2.png)\r\n\r\nStep#10 (25 million)\r\n![screen shot 2017-02-02 at 9 08 48 pm](https://cloud.githubusercontent.com/assets/4228346/22580293/061f4ed8-e98c-11e6-8fc0-36d20784d628.png)\r\n\r\n\r\n\r\n\r\n"}}
{"actor":{"display_login":"pygupta"}}