{"action":"created","issue":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/382","repository_url":"https://api.github.com/repos/cgiffard/node-simplecrawler","labels_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/382/labels{/name}","comments_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/382/comments","events_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/382/events","html_url":"https://github.com/cgiffard/node-simplecrawler/pull/382","id":244332300,"number":382,"title":"Performance optimization for very large sitemaps.","user":{"login":"elxa","id":5871169,"avatar_url":"https://avatars2.githubusercontent.com/u/5871169?v=4","gravatar_id":"","url":"https://api.github.com/users/elxa","html_url":"https://github.com/elxa","followers_url":"https://api.github.com/users/elxa/followers","following_url":"https://api.github.com/users/elxa/following{/other_user}","gists_url":"https://api.github.com/users/elxa/gists{/gist_id}","starred_url":"https://api.github.com/users/elxa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elxa/subscriptions","organizations_url":"https://api.github.com/users/elxa/orgs","repos_url":"https://api.github.com/users/elxa/repos","events_url":"https://api.github.com/users/elxa/events{/privacy}","received_events_url":"https://api.github.com/users/elxa/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-07-20T11:39:13Z","updated_at":"2017-07-21T07:50:33Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/pulls/382","html_url":"https://github.com/cgiffard/node-simplecrawler/pull/382","diff_url":"https://github.com/cgiffard/node-simplecrawler/pull/382.diff","patch_url":"https://github.com/cgiffard/node-simplecrawler/pull/382.patch"},"body":"Fixes #364\r\n\r\n## What this PR changes\r\nThis refactors the cleanExpandResources method in crawler.js to use a single for loop instead of Array.{filter,map,reduce}. The speedup observed was 20x for a sitemap with 20k urls.\r\nTo allow usage of const, let and sets Eslint was bumped to ES6 and Node.js < v4 was dropped from the travis/appveyor configuration.\r\n\r\n## Rationale\r\nAfter creating and analyzing several CPU profiles in Webstorm I decided to get rid of Javascripts native array methods filter, map and reduce, because they are very slow for large arrays. Additionally using a set instead of an array for the result of the reduce part makes a lot more sense because we want to keep unique URLs only."},"comment":{"url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/comments/316931001","html_url":"https://github.com/cgiffard/node-simplecrawler/pull/382#issuecomment-316931001","issue_url":"https://api.github.com/repos/cgiffard/node-simplecrawler/issues/382","id":316931001,"user":{"login":"elxa","id":5871169,"avatar_url":"https://avatars2.githubusercontent.com/u/5871169?v=4","gravatar_id":"","url":"https://api.github.com/users/elxa","html_url":"https://github.com/elxa","followers_url":"https://api.github.com/users/elxa/followers","following_url":"https://api.github.com/users/elxa/following{/other_user}","gists_url":"https://api.github.com/users/elxa/gists{/gist_id}","starred_url":"https://api.github.com/users/elxa/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elxa/subscriptions","organizations_url":"https://api.github.com/users/elxa/orgs","repos_url":"https://api.github.com/users/elxa/repos","events_url":"https://api.github.com/users/elxa/events{/privacy}","received_events_url":"https://api.github.com/users/elxa/received_events","type":"User","site_admin":false},"created_at":"2017-07-21T07:50:33Z","updated_at":"2017-07-21T07:50:33Z","body":"Dropping Node.js 0.X not only makes sense technically, but also because it is EOL since the end of 2016 https://github.com/nodejs/LTS#lts-schedule1"}}
{"actor":{"display_login":"elxa"}}