{"action":"closed","issue":{"url":"https://api.github.com/repos/faraday/wikiprep-esa/issues/6","repository_url":"https://api.github.com/repos/faraday/wikiprep-esa","labels_url":"https://api.github.com/repos/faraday/wikiprep-esa/issues/6/labels{/name}","comments_url":"https://api.github.com/repos/faraday/wikiprep-esa/issues/6/comments","events_url":"https://api.github.com/repos/faraday/wikiprep-esa/issues/6/events","html_url":"https://github.com/faraday/wikiprep-esa/issues/6","id":975294,"number":6,"title":"Use cElementTree to make everything faster and simpler","user":{"login":"turian","id":65918,"avatar_url":"https://avatars3.githubusercontent.com/u/65918?v=4","gravatar_id":"","url":"https://api.github.com/users/turian","html_url":"https://github.com/turian","followers_url":"https://api.github.com/users/turian/followers","following_url":"https://api.github.com/users/turian/following{/other_user}","gists_url":"https://api.github.com/users/turian/gists{/gist_id}","starred_url":"https://api.github.com/users/turian/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/turian/subscriptions","organizations_url":"https://api.github.com/users/turian/orgs","repos_url":"https://api.github.com/users/turian/repos","events_url":"https://api.github.com/users/turian/events{/privacy}","received_events_url":"https://api.github.com/users/turian/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2011-05-30T18:00:09Z","updated_at":"2017-10-12T19:57:35Z","closed_at":"2017-10-12T19:57:35Z","author_association":"NONE","body":"I would like to try wikiprep-esa, but it is currently waaay too slow. I was struggling with wikiprep-esa, but it seemed like it was going to take 10 days or more to run scan\\* on a recent wikidump.\nThis was the case even when I used pyre2 (a FAST regex library).\n\nI wrote the following code to read a wikiprep XML using cElementTree (which is a core library in Python 2.5):\n\n```\nhttps://github.com/turian/wikiprep-postprocess/blob/master/xmlwikiprep.py\n```\n\nThis code reads the XML in a _streaming_ fashion (i.e. doesn't read everything into memory). I just ran over an entire wikiprep dump overnight.\nSo the code is much faster than the regex parsing you do. Also, cElementTree does buffering automatically, so the code is much cleaner and shorter too, and simpler to understand.\n\nIt would be great if you could use my code as inspiration for refactoring wikiprep-esa to use cElementTree. It would work VERY fast then.\n"}}
{"actor":{"display_login":"faraday"}}