{"action":"created","issue":{"url":"https://api.github.com/repos/facebook/hhvm/issues/8043","repository_url":"https://api.github.com/repos/facebook/hhvm","labels_url":"https://api.github.com/repos/facebook/hhvm/issues/8043/labels{/name}","comments_url":"https://api.github.com/repos/facebook/hhvm/issues/8043/comments","events_url":"https://api.github.com/repos/facebook/hhvm/issues/8043/events","html_url":"https://github.com/facebook/hhvm/issues/8043","id":272539613,"number":8043,"title":"HHVM crashes or becomes unresponsive -- likely due to the use of \\HH\\Asio\\curl*","user":{"login":"pbl64k","id":814655,"avatar_url":"https://avatars3.githubusercontent.com/u/814655?v=4","gravatar_id":"","url":"https://api.github.com/users/pbl64k","html_url":"https://github.com/pbl64k","followers_url":"https://api.github.com/users/pbl64k/followers","following_url":"https://api.github.com/users/pbl64k/following{/other_user}","gists_url":"https://api.github.com/users/pbl64k/gists{/gist_id}","starred_url":"https://api.github.com/users/pbl64k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pbl64k/subscriptions","organizations_url":"https://api.github.com/users/pbl64k/orgs","repos_url":"https://api.github.com/users/pbl64k/repos","events_url":"https://api.github.com/users/pbl64k/events{/privacy}","received_events_url":"https://api.github.com/users/pbl64k/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":12,"created_at":"2017-11-09T12:01:31Z","updated_at":"2017-12-14T13:07:26Z","closed_at":null,"author_association":"NONE","body":"### HHVM Version\r\n\r\nIn particular, 3.21.3-lts, 3.22.1. Multiple OSes, running on hardware, virtualized, containerized etc. At this point we can crash it pretty much anywhere, although the severity and particulars needed to cause the crash may vary. We're currently using proxygen across the board, but we've first run into this problem while using nginx+fastcgi.\r\n\r\n```\r\nHipHop VM 3.21.3 (rel)\r\nCompiler: tags/HHVM-3.21.3-0-gce34e5458e6a75931438a2d8fc41b91454a2f2b2\r\nRepo schema: 90ed4216b98563d0585cf5c87959dc9ff7c2e1a1\r\n```\r\n\r\n```\r\nHipHop VM 3.22.1 (rel)\r\nCompiler: tags/HHVM-3.22.1-0-ge8ce62608d2d14e6418772b07ebaddcbdb37598f\r\nRepo schema: 05fc45ee8e70c9f18dfb1e867219c8528be87996\r\n```\r\n\r\nI would hazard a guess that the problem is long-standing, and all reasonably recent versions are affected.\r\n\r\n### Standalone code, or other way to reproduce the problem\r\n\r\nhttps://github.com/genych/burn\r\n\r\nSee instructions in the `README.md`. This can be deployed and run on an existing HHVM node, but also includes a docker-compose recipe to run the three components in isolated containers.\r\n\r\nThe repro includes three components:\r\n\r\n1. CLI script that initiates calls to the \"service\".\r\n2. Remote service retrieving another remote resource using `\\HH\\Asio\\curl_exec` -- this is the part that causes the crashes.\r\n3. \"Static\" resource retrieved by part 2.\r\n\r\nNote that this is not guaranteed to produce a crash in a single run. Increasing the number of attempts or simultaneous connections increases the likelihood of success.\r\n\r\nResponse time for part 3 also seems to affect reproducibility. Very short response times are much more likely to result in a crash, so it's best to use a locally deployed static resource.\r\n\r\n### Expected result\r\n\r\nHHVM continues serving requests or at least crashes gracefully.\r\n\r\n### Actual result\r\n\r\nSeveral outcomes possible.\r\n\r\n1. HHVM crashes with:\r\n```\r\nFinished singleJitRequest 1\r\nFinished singleJitRequest 2\r\nFinished singleJitRequest 4\r\nFinished singleJitRequest 3\r\nFinished singleJitRequest 5\r\nFinished singleJitRequest 6\r\nFinished singleJitRequest 7\r\nFinished singleJitRequest 8\r\nFinished singleJitRequest 9\r\nFinished singleJitRequest 10\r\nFinished singleJitRequest 11\r\nFinished singleJitRequest 12\r\nFinished singleJitRequest 13\r\nFinished singleJitRequest 14\r\nFinished singleJitRequest 15\r\nFinished singleJitRequest 16\r\nFinished singleJitRequest 17\r\nFinished singleJitRequest 18\r\nFinished singleJitRequest 19\r\nFinished singleJitRequest 20\r\nFinished singleJitRequest 21\r\nFinished singleJitRequest 22\r\npure virtual method called\r\nterminate called without an active exception\r\nCore dumped: Aborted\r\nStack trace in /tmp/stacktrace.29196.log\r\nAborted (core dumped)\r\n```\r\nStack trace contains:\r\n```\r\nHost: xub\r\nProcessID: 29196\r\nThreadID: 140377659008768\r\nThreadPID: 29223\r\nName: /usr/bin/hhvm\r\nType: Aborted\r\nRuntime: hhvm\r\nVersion: tags/HHVM-3.22.1-0-ge8ce62608d2d14e6418772b07ebaddcbdb37598f\r\nDebuggerCount: 0\r\n\r\n\r\n# 0  0000000000dd20d6\r\n# 1  00000000011cb452\r\n# 2  00007fac76c31390\r\n# 3  00007fac701a0428\r\n# 4  00007fac701a202a\r\n# 5  00007fac70ae384d\r\n# 6  00007fac70ae16b6\r\n# 7  00007fac70ae1701\r\n# 8  00007fac70ae223f\r\n# 9  00007fac777094c9\r\n# 10 00000000036d2ee6\r\n# 11 0000000003135c19\r\n# 12 00007fac70b0cc80\r\n# 13 00007fac76c276ba\r\n# 14 00007fac702723dd\r\n```\r\nThis is the preferrable outcome from our standpoint as we can at least detect the problem and revive the service. It's still quite unsatisfactory of course.\r\n2. HHVM becomes unresponsive and will not respond to any further HTTP requests. This *may* be accompanied by CPU usage for the process stuck at 100%. Naturally, this is highly undesirable, as everything deployed on the affected node just goes belly up until someone intervenes.\r\n\r\n### Possible causes/related issues\r\n\r\n#8029, #8037 - early reports on -- probably -- the same problem. Since @genych managed to put together an isolated repro without any external dependencies (the previous reports refer to the proprietary software we're working on), I decided to file this separately.\r\n\r\n#7972, #6500, #6518 look closely related, assuming our hypothesis about asio being the culprit is correct. #7972 includes an isolated, purely local repro. #6518 is marked as closed, but there's a later comment https://github.com/facebook/hhvm/pull/6518#issuecomment-174083842 claiming that the fix is incorrect. This comment may also explain the fact that the repro is more likely to cause crashes when \"static remote resource\" in part 3 responds very quickly.\r\n\r\nIf this is indeed caused by asio, that would at least be manageable from our POV. We've already started weeding out all uses of async I/O across the project. It's a letdown, as async I/O was one of the three major reasons for choosing Hack for the project, and it's gonna be a major undertaking, but at least it's something we can \"fix\" on our side, and unstable web nodes showing the symptoms described above would be a complete deal-breaker.\r\n\r\nWe'll be trying to narrow down the problem further. I'll report any new findings here."},"comment":{"url":"https://api.github.com/repos/facebook/hhvm/issues/comments/351705861","html_url":"https://github.com/facebook/hhvm/issues/8043#issuecomment-351705861","issue_url":"https://api.github.com/repos/facebook/hhvm/issues/8043","id":351705861,"user":{"login":"pbl64k","id":814655,"avatar_url":"https://avatars3.githubusercontent.com/u/814655?v=4","gravatar_id":"","url":"https://api.github.com/users/pbl64k","html_url":"https://github.com/pbl64k","followers_url":"https://api.github.com/users/pbl64k/followers","following_url":"https://api.github.com/users/pbl64k/following{/other_user}","gists_url":"https://api.github.com/users/pbl64k/gists{/gist_id}","starred_url":"https://api.github.com/users/pbl64k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pbl64k/subscriptions","organizations_url":"https://api.github.com/users/pbl64k/orgs","repos_url":"https://api.github.com/users/pbl64k/repos","events_url":"https://api.github.com/users/pbl64k/events{/privacy}","received_events_url":"https://api.github.com/users/pbl64k/received_events","type":"User","site_admin":false},"created_at":"2017-12-14T13:07:26Z","updated_at":"2017-12-14T13:07:26Z","author_association":"NONE","body":"Oops, my bad, #8037 probably is unrelated. We spent ~four weeks trying to narrow down the problem before we got to the stage of repro I originally posted here, and we've run into a few other issues along the way. #8037 is one of them, unfortunately, I don't think we can repro it.\r\n\r\n> ASIO in general is solid\r\n\r\nI understand that `folly::EventHandler` is not used for MySQL and memcached? I can't help but suspect it must be `EventHandler` itself.\r\n\r\n> I'll take a quick look at this + stream_await when I've finished my current work\r\n\r\nI'm really glad to hear that.\r\n\r\n> issues that don't include a small, isolated, inline example are at the bottom of the pile, as the time-to-investigate vs benefit-of-fixing is in general much lower.\r\n\r\nIt's obviously a value judgement, but the `stream_await` repro in #7972 is probably as good as it could get for an issue like this, and HHVM just ceasing to serve requests without even terminating, normally or abnormally, is about as nasty an effect as I can imagine."}}
{"actor":{"display_login":"pbl64k"}}