{"action":"created","issue":{"url":"https://api.github.com/repos/tensorflow/models/issues/1108","repository_url":"https://api.github.com/repos/tensorflow/models","labels_url":"https://api.github.com/repos/tensorflow/models/issues/1108/labels{/name}","comments_url":"https://api.github.com/repos/tensorflow/models/issues/1108/comments","events_url":"https://api.github.com/repos/tensorflow/models/issues/1108/events","html_url":"https://github.com/tensorflow/models/pull/1108","id":211752374,"number":1108,"title":"Swivel: fastprep: use mmap()-ed IO for vocabulary parsing","user":{"login":"vmarkovtsev","id":2793551,"avatar_url":"https://avatars0.githubusercontent.com/u/2793551?v=3","gravatar_id":"","url":"https://api.github.com/users/vmarkovtsev","html_url":"https://github.com/vmarkovtsev","followers_url":"https://api.github.com/users/vmarkovtsev/followers","following_url":"https://api.github.com/users/vmarkovtsev/following{/other_user}","gists_url":"https://api.github.com/users/vmarkovtsev/gists{/gist_id}","starred_url":"https://api.github.com/users/vmarkovtsev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vmarkovtsev/subscriptions","organizations_url":"https://api.github.com/users/vmarkovtsev/orgs","repos_url":"https://api.github.com/users/vmarkovtsev/repos","events_url":"https://api.github.com/users/vmarkovtsev/events{/privacy}","received_events_url":"https://api.github.com/users/vmarkovtsev/received_events","type":"User","site_admin":false},"labels":[{"id":337864321,"url":"https://api.github.com/repos/tensorflow/models/labels/cla:%20yes","name":"cla: yes","color":"009800","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2017-03-03T17:30:31Z","updated_at":"2017-03-04T18:55:44Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/tensorflow/models/pulls/1108","html_url":"https://github.com/tensorflow/models/pull/1108","diff_url":"https://github.com/tensorflow/models/pull/1108.diff","patch_url":"https://github.com/tensorflow/models/pull/1108.patch"},"body":"This speeds up vocabulary parsing 2x for me.\r\n\r\nI mmap 8x PAGE_SIZE. Some care was required on buffer boundaries and in the file end."},"comment":{"url":"https://api.github.com/repos/tensorflow/models/issues/comments/284172767","html_url":"https://github.com/tensorflow/models/pull/1108#issuecomment-284172767","issue_url":"https://api.github.com/repos/tensorflow/models/issues/1108","id":284172767,"user":{"login":"waterson","id":93609,"avatar_url":"https://avatars2.githubusercontent.com/u/93609?v=3","gravatar_id":"","url":"https://api.github.com/users/waterson","html_url":"https://github.com/waterson","followers_url":"https://api.github.com/users/waterson/followers","following_url":"https://api.github.com/users/waterson/following{/other_user}","gists_url":"https://api.github.com/users/waterson/gists{/gist_id}","starred_url":"https://api.github.com/users/waterson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/waterson/subscriptions","organizations_url":"https://api.github.com/users/waterson/orgs","repos_url":"https://api.github.com/users/waterson/repos","events_url":"https://api.github.com/users/waterson/events{/privacy}","received_events_url":"https://api.github.com/users/waterson/received_events","type":"User","site_admin":false},"created_at":"2017-03-04T18:55:44Z","updated_at":"2017-03-04T18:55:44Z","body":"Gotcha, glad to see that you tried the simple thing first.\r\n\r\nWere you able to profile to see what the problem was in that case? For example, was something like tellg or eof bottoming out in a system call? In the interest of keeping things simple (and consistent between reading vocab and counting co-occurrences), it would be great to understand why ifstream is performing badly.\r\n\r\nIf blockwise mmap is really the best fix, implementing the tokenization consistently (i.e., using the same \"GetWord\" implementation) between the two phases would be my preference.\r\n\r\n"}}
{"actor":{"display_login":"waterson"}}