{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/2518","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/2518/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/2518/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/2518/events","html_url":"https://github.com/pytorch/pytorch/issues/2518","id":252300674,"number":2518,"title":"investigate performance regressions in v0.2.0 + DyNet benchmarks","user":{"login":"soumith","id":1310570,"avatar_url":"https://avatars0.githubusercontent.com/u/1310570?v=4","gravatar_id":"","url":"https://api.github.com/users/soumith","html_url":"https://github.com/soumith","followers_url":"https://api.github.com/users/soumith/followers","following_url":"https://api.github.com/users/soumith/following{/other_user}","gists_url":"https://api.github.com/users/soumith/gists{/gist_id}","starred_url":"https://api.github.com/users/soumith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumith/subscriptions","organizations_url":"https://api.github.com/users/soumith/orgs","repos_url":"https://api.github.com/users/soumith/repos","events_url":"https://api.github.com/users/soumith/events{/privacy}","received_events_url":"https://api.github.com/users/soumith/received_events","type":"User","site_admin":false},"labels":[{"id":424131847,"url":"https://api.github.com/repos/pytorch/pytorch/labels/bug","name":"bug","color":"b60205","default":true},{"id":443484135,"url":"https://api.github.com/repos/pytorch/pytorch/labels/high%20priority","name":"high priority","color":"F22613","default":false}],"state":"open","locked":false,"assignee":{"login":"soumith","id":1310570,"avatar_url":"https://avatars0.githubusercontent.com/u/1310570?v=4","gravatar_id":"","url":"https://api.github.com/users/soumith","html_url":"https://github.com/soumith","followers_url":"https://api.github.com/users/soumith/followers","following_url":"https://api.github.com/users/soumith/following{/other_user}","gists_url":"https://api.github.com/users/soumith/gists{/gist_id}","starred_url":"https://api.github.com/users/soumith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumith/subscriptions","organizations_url":"https://api.github.com/users/soumith/orgs","repos_url":"https://api.github.com/users/soumith/repos","events_url":"https://api.github.com/users/soumith/events{/privacy}","received_events_url":"https://api.github.com/users/soumith/received_events","type":"User","site_admin":false},"assignees":[{"login":"soumith","id":1310570,"avatar_url":"https://avatars0.githubusercontent.com/u/1310570?v=4","gravatar_id":"","url":"https://api.github.com/users/soumith","html_url":"https://github.com/soumith","followers_url":"https://api.github.com/users/soumith/followers","following_url":"https://api.github.com/users/soumith/following{/other_user}","gists_url":"https://api.github.com/users/soumith/gists{/gist_id}","starred_url":"https://api.github.com/users/soumith/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/soumith/subscriptions","organizations_url":"https://api.github.com/users/soumith/orgs","repos_url":"https://api.github.com/users/soumith/repos","events_url":"https://api.github.com/users/soumith/events{/privacy}","received_events_url":"https://api.github.com/users/soumith/received_events","type":"User","site_admin":false}],"milestone":null,"comments":5,"created_at":"2017-08-23T14:20:22Z","updated_at":"2017-09-11T01:57:59Z","closed_at":null,"author_association":"OWNER","body":"According to the scripts given in https://twitter.com/kchonyc/status/899832376955744257 there are two issues:\r\n1. 0.2.0 has a 2x perf regression over 0.1.2 on small workloads\r\n2. pytorch is ~10x slower than DyNet on this workload.\r\n\r\nAn issue to track this.\r\n\r\nHere's the google doc with links to the code snippets, current numbers (with and without threading) and will update further as I investigate: https://docs.google.com/spreadsheets/d/1_k_mcxgR94OXH4qD6YYZNiS25QnKbQqAyB72KVD-Z9M/edit?usp=sharing\r\n\r\n#### On PyTorch vs DyNet\r\n\r\nSo far, I've done stack sampling using `perf top` and it looks like MKL is spending significant time in barriers / threading overhead. DyNet on the other hand seems to be using Eigen's gemm for the workload.\r\n\r\n#### On 0.2.0 vs 0.1.2\r\ninvestigating, will edit comment once I know more.\r\n\r\n\r\nThanks to @kyunghyuncho @hal3 and @neubig for bringing this up. I will only close this issue once we're within reasonable ballpark of the DyNet numbers."},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/328396062","html_url":"https://github.com/pytorch/pytorch/issues/2518#issuecomment-328396062","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/2518","id":328396062,"user":{"login":"negrinho","id":9198933,"avatar_url":"https://avatars3.githubusercontent.com/u/9198933?v=4","gravatar_id":"","url":"https://api.github.com/users/negrinho","html_url":"https://github.com/negrinho","followers_url":"https://api.github.com/users/negrinho/followers","following_url":"https://api.github.com/users/negrinho/following{/other_user}","gists_url":"https://api.github.com/users/negrinho/gists{/gist_id}","starred_url":"https://api.github.com/users/negrinho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/negrinho/subscriptions","organizations_url":"https://api.github.com/users/negrinho/orgs","repos_url":"https://api.github.com/users/negrinho/repos","events_url":"https://api.github.com/users/negrinho/events{/privacy}","received_events_url":"https://api.github.com/users/negrinho/received_events","type":"User","site_admin":false},"created_at":"2017-09-11T01:57:58Z","updated_at":"2017-09-11T01:57:58Z","author_association":"NONE","body":"@honnibal, we are talking about 10x slower when compared to DyNet for the same model. This is prohibitively slow. There are cases in research where batching is difficult or you would rather bother with it later. In these cases, it is important to be able to get a working model that runs reasonably fast. This is is especially important in a dynamic frameworks such as PyTorch, as flexibility is one its main selling points. A factor of ten is the difference between getting results in an hour or getting results in ten hours. This definitely makes a difference in research projects where you want to iterate fast. \r\n\r\nThere is always this idea that we are able to spend most of the time in fast C/C++ code, but I wonder if it really happens that frequently. I got uniformly slow Python code on more than one occasion. I'm definitely looking forward for the coming performance fix."}}
{"actor":{"display_login":"negrinho"}}