{"action":"created","issue":{"url":"https://api.github.com/repos/Microsoft/LightGBM/issues/542","repository_url":"https://api.github.com/repos/Microsoft/LightGBM","labels_url":"https://api.github.com/repos/Microsoft/LightGBM/issues/542/labels{/name}","comments_url":"https://api.github.com/repos/Microsoft/LightGBM/issues/542/comments","events_url":"https://api.github.com/repos/Microsoft/LightGBM/issues/542/events","html_url":"https://github.com/Microsoft/LightGBM/issues/542","id":230456805,"number":542,"title":"Visual Studio reports higher CPU usage than MinGW","user":{"login":"Laurae2","id":9083669,"avatar_url":"https://avatars0.githubusercontent.com/u/9083669?v=3","gravatar_id":"","url":"https://api.github.com/users/Laurae2","html_url":"https://github.com/Laurae2","followers_url":"https://api.github.com/users/Laurae2/followers","following_url":"https://api.github.com/users/Laurae2/following{/other_user}","gists_url":"https://api.github.com/users/Laurae2/gists{/gist_id}","starred_url":"https://api.github.com/users/Laurae2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Laurae2/subscriptions","organizations_url":"https://api.github.com/users/Laurae2/orgs","repos_url":"https://api.github.com/users/Laurae2/repos","events_url":"https://api.github.com/users/Laurae2/events{/privacy}","received_events_url":"https://api.github.com/users/Laurae2/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2017-05-22T16:52:17Z","updated_at":"2017-06-03T11:31:20Z","closed_at":null,"body":"(Requires investigation)\r\n\r\nThis issue reflects an issue found on the following issues:\r\n\r\n* https://github.com/Microsoft/LightGBM/issues/446: Installation of CMake&MinGW is slow than Visual Studio\r\n\r\n> I tried the two method of installaion on my computer (6core,3.6Ghz). trainning and predicting with the same datasets(one million samples and 150 features). two method have different CPU occupancy. the detail info as follow:\r\n> \r\n> CMake+MinGW:\r\n> trainning cost 560 seconds (CPU occupancy is 70%)\r\n> testing cost 44 seconds (CPU occupancy is 50%)\r\n>\r\n> Visual Studio:\r\n> trainning cost 547 seconds (CPU occupancy is 100%)\r\n> testing cost 23 seconds (CPU occupancy is 100%)\r\n\r\n* https://github.com/Microsoft/LightGBM/issues/512: lightgbm not using all the CPU\r\n\r\n![pic vs](https://cloud.githubusercontent.com/assets/16040950/26032207/4e2b6bc0-38c0-11e7-9754-4fadb27a3b57.png)\r\n\r\n![pic mingw](https://cloud.githubusercontent.com/assets/9083669/26032190/6b444f48-388d-11e7-8f3d-2fd75a2351a8.png)\r\n\r\n* (probably missed more issues)\r\n\r\nWe have to find out why there is such discrepancy between Visual Studio and MinGW.\r\n\r\nEven worse is MinGW compilation leading to a way less CPU usage during training, but ending only slightly slower than Visual Studio which is taking 100% CPU.\r\n\r\nMinGW vs Visual Studio is a known issue with OpenMP especially for xgboost in Windows. One can look there: https://github.com/dmlc/xgboost/pull/2243\r\n\r\n> Reported in issue dmlc/xgboost#2165. Dynamic scheduling of OpenMP loops involve\r\nimplicit synchronization. To implement synchronization, libgomp uses futex\r\n(fast userspace mutex), whereas MinGW uses kernel-space mutex, which is more\r\ncostly. With chunk size of 1, synchronization overhead may become prohibitive\r\non Windows machines.\r\n> \r\n> Solution: use 'guided' schedule to minimize the number of syncs\r\n\r\nPossible theory: perhaps threads are locked in Visual Studio (so they are hot), while MinGW is freeing them when possible with OpenMP (cold cores). This would lead to MinGW compiled LightGBM having less CPU usage due to higher synchronization/overhead costs (explaining why it is slower) while Visual Studio compiled LightGBM would keep the cores \"ready\" for training (but both behaviors are unexplained so far)."},"comment":{"url":"https://api.github.com/repos/Microsoft/LightGBM/issues/comments/305969246","html_url":"https://github.com/Microsoft/LightGBM/issues/542#issuecomment-305969246","issue_url":"https://api.github.com/repos/Microsoft/LightGBM/issues/542","id":305969246,"user":{"login":"guolinke","id":16040950,"avatar_url":"https://avatars3.githubusercontent.com/u/16040950?v=3","gravatar_id":"","url":"https://api.github.com/users/guolinke","html_url":"https://github.com/guolinke","followers_url":"https://api.github.com/users/guolinke/followers","following_url":"https://api.github.com/users/guolinke/following{/other_user}","gists_url":"https://api.github.com/users/guolinke/gists{/gist_id}","starred_url":"https://api.github.com/users/guolinke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guolinke/subscriptions","organizations_url":"https://api.github.com/users/guolinke/orgs","repos_url":"https://api.github.com/users/guolinke/repos","events_url":"https://api.github.com/users/guolinke/events{/privacy}","received_events_url":"https://api.github.com/users/guolinke/received_events","type":"User","site_admin":false},"created_at":"2017-06-03T11:31:20Z","updated_at":"2017-06-03T11:31:20Z","body":"@Laurae2 I think all expose apis are c api. So it can be used.\r\n\r\nFor the DLL in R packages, I think the VS version can be normally used. The only problem is how to put it into R package correctly, without the error of \"cannot  find lightgbm.dll\" . \r\n\r\nAnother possible solution is solving the multi-threading problem in MinGW. "}}
{"actor":{"display_login":"guolinke"}}