{"action":"created","issue":{"url":"https://api.github.com/repos/bartTC/dpaste/issues/85","repository_url":"https://api.github.com/repos/bartTC/dpaste","labels_url":"https://api.github.com/repos/bartTC/dpaste/issues/85/labels{/name}","comments_url":"https://api.github.com/repos/bartTC/dpaste/issues/85/comments","events_url":"https://api.github.com/repos/bartTC/dpaste/issues/85/events","html_url":"https://github.com/bartTC/dpaste/issues/85","id":201663336,"number":85,"title":"dpaste performance slows when >60k snippets are in the DB","user":{"login":"ferringb","id":1531180,"avatar_url":"https://avatars.githubusercontent.com/u/1531180?v=3","gravatar_id":"","url":"https://api.github.com/users/ferringb","html_url":"https://github.com/ferringb","followers_url":"https://api.github.com/users/ferringb/followers","following_url":"https://api.github.com/users/ferringb/following{/other_user}","gists_url":"https://api.github.com/users/ferringb/gists{/gist_id}","starred_url":"https://api.github.com/users/ferringb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ferringb/subscriptions","organizations_url":"https://api.github.com/users/ferringb/orgs","repos_url":"https://api.github.com/users/ferringb/repos","events_url":"https://api.github.com/users/ferringb/events{/privacy}","received_events_url":"https://api.github.com/users/ferringb/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-01-18T19:04:45Z","updated_at":"2017-01-18T21:05:02Z","closed_at":null,"body":"In my setup, I've got over 60k snippets in the DB, and dpaste's speed goes from a previous 'fast' to a few  seconds for new insertion.\r\n\r\nDo not run the following code on a live dpaste instance; stand up a test one, it wipes records to control the test.  The following if pasted into a ```django-admin shell``` instance should replicate it- pardon if it's postgres specific:\r\n\r\n```\r\nfrom dpaste.models import Snippet\r\nfrom django.db import connection\r\nimport time\r\nstart = time.time()\r\n_ = Snippet(content=\"asdf\").save()\r\nprint \"empty insert took %2.2f\" % (time.time() - start,)\r\n\r\n# bulk load; note that Snippet.objects.bulk_create fails w/ a lft exception, likely a bug in mptt.\r\n# hence this approach.\r\nwith connection.cursor() as cursor:\r\n  # Wipe the previous record since we have assumptions about secret_id below\r\n  cursor.execute(\"delete from dpaste_snippet\");\r\n  cursor.executemany(\"\"\"\r\ninsert into dpaste_snippet(secret_id, content, lexer, published, expire_type, view_count, lft, rght, tree_id, level)\r\n            values(%s, 'my-content', 'text', CURRENT_TIMESTAMP, 0, 1, 1, 1,\r\n                (SELECT coalesce(max(tree_id), 1) +1 from dpaste_snippet), 0);\r\n  \"\"\", [(x,) for x in xrange(60000)])\r\n\r\n# Now we time the API call now that we have 60k in there.\r\nfrom dpaste.models import Snippet\r\nimport time\r\nstart = time.time()\r\n_ = Snippet(content=\"asdf\").save()\r\nprint \"took %2.2fs\" % (time.time() - start,)\r\nprint \"finished\"\r\n```\r\n\r\nIn some basic playing in django-admin, I've found that if I disable mptt updates, then performance is normal- I'm still investigating however."},"comment":{"url":"https://api.github.com/repos/bartTC/dpaste/issues/comments/273601413","html_url":"https://github.com/bartTC/dpaste/issues/85#issuecomment-273601413","issue_url":"https://api.github.com/repos/bartTC/dpaste/issues/85","id":273601413,"user":{"login":"bartTC","id":1896,"avatar_url":"https://avatars.githubusercontent.com/u/1896?v=3","gravatar_id":"","url":"https://api.github.com/users/bartTC","html_url":"https://github.com/bartTC","followers_url":"https://api.github.com/users/bartTC/followers","following_url":"https://api.github.com/users/bartTC/following{/other_user}","gists_url":"https://api.github.com/users/bartTC/gists{/gist_id}","starred_url":"https://api.github.com/users/bartTC/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bartTC/subscriptions","organizations_url":"https://api.github.com/users/bartTC/orgs","repos_url":"https://api.github.com/users/bartTC/repos","events_url":"https://api.github.com/users/bartTC/events{/privacy}","received_events_url":"https://api.github.com/users/bartTC/received_events","type":"User","site_admin":false},"created_at":"2017-01-18T21:05:02Z","updated_at":"2017-01-18T21:05:02Z","body":"Interesting. I've never stumbled over this problem because the snippets on dpaste.de don't have an infinite life time, I think at max there were about 5k in the db. dpaste.de is using Postgres as well. \r\n\r\nWithout further testing I think your benchmarks make sense. dpaste is using Nested Sets which are read-optimized, not for writing. Inserting an item in a Nested set tree requires the entire tree to be rebuild. Your observation on removing mptt makes therefor sense. \r\n\r\nI don't think we can do much there shortnterm, however I do think that I could simply ditch mptt as a requirement and use a simpler, more low level approach to display a tree relation of snippets. They are rarely used so I dont mind it. "}}
{"actor":{"display_login":"bartTC"}}