{"action":"closed","issue":{"url":"https://api.github.com/repos/chriskohlhoff/asio/issues/180","repository_url":"https://api.github.com/repos/chriskohlhoff/asio","labels_url":"https://api.github.com/repos/chriskohlhoff/asio/issues/180/labels{/name}","comments_url":"https://api.github.com/repos/chriskohlhoff/asio/issues/180/comments","events_url":"https://api.github.com/repos/chriskohlhoff/asio/issues/180/events","html_url":"https://github.com/chriskohlhoff/asio/issues/180","id":202537175,"number":180,"title":"Asio on Linux stalls in epoll()","user":{"login":"jeremyd-gl","id":25299868,"avatar_url":"https://avatars3.githubusercontent.com/u/25299868?v=4","gravatar_id":"","url":"https://api.github.com/users/jeremyd-gl","html_url":"https://github.com/jeremyd-gl","followers_url":"https://api.github.com/users/jeremyd-gl/followers","following_url":"https://api.github.com/users/jeremyd-gl/following{/other_user}","gists_url":"https://api.github.com/users/jeremyd-gl/gists{/gist_id}","starred_url":"https://api.github.com/users/jeremyd-gl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeremyd-gl/subscriptions","organizations_url":"https://api.github.com/users/jeremyd-gl/orgs","repos_url":"https://api.github.com/users/jeremyd-gl/repos","events_url":"https://api.github.com/users/jeremyd-gl/events{/privacy}","received_events_url":"https://api.github.com/users/jeremyd-gl/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2017-01-23T13:44:20Z","updated_at":"2017-07-31T21:53:08Z","closed_at":"2017-07-31T21:53:08Z","body":"[ As posted on StackOverflow - [http://stackoverflow.com/questions/41804866/asio-on-linux-stalls-in-epoll](http://stackoverflow.com/questions/41804866/asio-on-linux-stalls-in-epoll) ]\r\n\r\nWe're experiencing a problem with asynchronous operation of standalone (non-Boost) Asio 1.10.6 on Linux, which is demonstrated using the following test app:\r\n\r\n    #define ASIO_STANDALONE\r\n    #define ASIO_HEADER_ONLY\r\n    #define ASIO_NO_EXCEPTIONS\r\n    #define ASIO_NO_TYPEID\r\n    #include \"asio.hpp\"\r\n    \r\n    #include <chrono>\r\n    #include <iostream>\r\n    #include <list>\r\n    #include <map>\r\n    #include <thread>\r\n    \r\n    static bool s_freeInboundSocket = false;\r\n    static bool s_freeOutboundSocket = false;\r\n    \r\n    class Tester\r\n    {\r\n    public:\r\n    \tTester(asio::io_service& i_ioService, unsigned i_n)\r\n    \t\t: m_inboundStrand(i_ioService)\r\n    \t\t, m_listener(i_ioService)\r\n    \t\t, m_outboundStrand(i_ioService)\r\n    \t\t, m_resolver(i_ioService)\r\n    \t\t, m_n(i_n)\r\n    \t\t, m_traceStart(std::chrono::high_resolution_clock::now())\r\n    \t{}\r\n    \r\n    \t~Tester()\r\n    \t{}\r\n    \r\n    \tvoid TraceIn(unsigned i_line)\r\n    \t{\r\n    \t\tm_inboundTrace.emplace_back(i_line, std::chrono::duration_cast<std::chrono::nanoseconds>(std::chrono::high_resolution_clock::now() - m_traceStart));\r\n    \t}\r\n    \r\n    \tvoid AbortIn(unsigned i_line)\r\n    \t{\r\n    \t\tTraceIn(i_line);\r\n    \t\tabort();\r\n    \t}\r\n    \r\n    \tvoid TraceOut(unsigned i_line)\r\n    \t{\r\n    \t\tm_outboundTrace.emplace_back(i_line, std::chrono::duration_cast<std::chrono::nanoseconds>(std::chrono::high_resolution_clock::now() - m_traceStart));\r\n    \t}\r\n    \r\n    \tvoid AbortOut(unsigned i_line)\r\n    \t{\r\n    \t\tTraceOut(i_line);\r\n    \t\tabort();\r\n    \t}\r\n    \r\n    \tvoid DumpTrace(std::map<unsigned, unsigned>& o_counts)\r\n    \t{\r\n    \t\tstd::cout << \"## \" << m_n << \" ##\\n\";\r\n    \t\tstd::cout << \"-- \" << m_traceStart.time_since_epoch().count() << \"\\n\";\r\n    \t\tstd::cout << \"- in -             - out -\\n\";\r\n    \r\n    \t\tauto in = m_inboundTrace.begin();\r\n    \t\tauto out = m_outboundTrace.begin();\r\n    \r\n    \t\twhile ((in != m_inboundTrace.end()) || (out != m_outboundTrace.end()))\r\n    \t\t{\r\n    \t\t\tif (in == m_inboundTrace.end())\t\r\n    \t\t\t{\r\n    \t\t\t\t++o_counts[out->first];\r\n    \r\n    \t\t\t\tstd::cout << \"                  \" << out->first << \" : \" << out->second.count() << \"\\n\";\r\n    \t\t\t\t++out;\r\n    \t\t\t}\r\n    \t\t\telse if (out == m_outboundTrace.end())\t\r\n    \t\t\t{\r\n    \t\t\t\t++o_counts[in->first];\r\n    \r\n    \t\t\t\tstd::cout << in->first << \" : \" << in->second.count() << \"\\n\";\r\n    \t\t\t\t++in;\r\n    \t\t\t}\r\n    \t\t\telse if (out->second < in->second)\r\n    \t\t\t{\r\n    \t\t\t\t++o_counts[out->first];\r\n    \r\n    \t\t\t\tstd::cout << \"                  \" << out->first << \" : \" << out->second.count() << \"\\n\";\r\n    \t\t\t\t++out;\r\n    \t\t\t}\r\n    \t\t\telse\r\n    \t\t\t{\r\n    \t\t\t\t++o_counts[in->first];\r\n    \r\n    \t\t\t\tstd::cout << in->first << \" : \" << in->second.count() << \"\\n\";\r\n    \t\t\t\t++in;\r\n    \t\t\t}\r\n    \t\t}\r\n    \t\tstd::cout << std::endl;\r\n    \t}\r\n    \r\n    \t//////////////\r\n    \t// Inbound\r\n    \r\n    \tvoid Listen(uint16_t i_portBase)\r\n    \t{\r\n    \t\tm_inboundSocket.reset(new asio::ip::tcp::socket(m_inboundStrand.get_io_service()));\r\n     \r\n    \t\tasio::error_code ec;\r\n    \t\tif (m_listener.open(asio::ip::tcp::v4(), ec)\r\n    \t\t || m_listener.bind(asio::ip::tcp::endpoint(asio::ip::tcp::v4(), i_portBase+m_n), ec)\r\n    \t\t || m_listener.listen(-1, ec))\r\n    \t\t{\r\n    \t\t\tAbortIn(__LINE__); return;\r\n    \t\t}\r\n    \r\n    \t\tTraceIn(__LINE__);\r\n    \r\n    \t\tm_listener.async_accept(*m_inboundSocket,\r\n    \t\t\tm_inboundStrand.wrap([this](const asio::error_code& i_error)\r\n    \t\t{\r\n    \t\t\tOnInboundAccepted(i_error);\r\n    \t\t}));\r\n    \t}\r\n    \r\n    \tvoid OnInboundAccepted(const asio::error_code& i_error)\r\n    \t{\r\n    \t\tTraceIn(__LINE__);\r\n    \r\n    \t\tif (i_error) { AbortIn(__LINE__); return; }\r\n    \r\n    \t\tasio::async_read_until(*m_inboundSocket, m_inboundRxBuf, '\\n',\r\n    \t\t\tm_inboundStrand.wrap([this](const asio::error_code& i_err, size_t i_nRd)\r\n    \t\t{\r\n    \t\t\tOnInboundReadCompleted(i_err, i_nRd);\r\n    \t\t}));\r\n    \t}\r\n    \r\n    \r\n    \tvoid OnInboundReadCompleted(const asio::error_code& i_error, size_t i_nRead)\r\n    \t{\r\n    \t\tTraceIn(__LINE__);\r\n    \r\n    \t\tif (i_error.value() != 0) { AbortIn(__LINE__); return; }\r\n    \t\tif (bool(i_error)) { AbortIn(__LINE__); return; }\r\n    \t\tif (i_nRead != 4) { AbortIn(__LINE__); return; }  // \"msg\\n\"\r\n    \r\n    \t\tstd::istream is(&m_inboundRxBuf);\r\n    \t\tstd::string s;\r\n    \t\tif (!std::getline(is, s)) { AbortIn(__LINE__); return; }\r\n    \t\tif (s != \"msg\") { AbortIn(__LINE__); return; }\r\n    \t\tif (m_inboundRxBuf.in_avail() != 0) { AbortIn(__LINE__); return; }\r\n     \r\n    \t\tasio::async_read_until(*m_inboundSocket, m_inboundRxBuf, '\\n',\r\n    \t\t\tm_inboundStrand.wrap([this](const asio::error_code& i_err, size_t i_nRd)\r\n    \t\t{\r\n    \t\t\tOnInboundWaitCompleted(i_err, i_nRd);\r\n    \t\t}));\r\n    \r\n    \t}\r\n    \r\n    \tvoid OnInboundWaitCompleted(const asio::error_code& i_error, size_t i_nRead)\r\n    \t{\r\n    \t\tTraceIn(__LINE__);\r\n    \r\n    \t\tif (i_error != asio::error::eof) { AbortIn(__LINE__); return; }\r\n    \t\tif (i_nRead != 0) { AbortIn(__LINE__); return; }\r\n     \r\n    \t\tif (s_freeInboundSocket)\r\n    \t\t{\r\n    \t\t\tm_inboundSocket.reset();\r\n    \t\t}\r\n    \t}\r\n    \r\n    \t//////////////\r\n    \t// Outbound\r\n    \r\n    \tvoid Connect(std::string i_host, uint16_t i_portBase)\r\n    \t{\r\n    \t\tasio::error_code ec;\r\n    \t\tauto endpoint = m_resolver.resolve(asio::ip::tcp::resolver::query(i_host, std::to_string(i_portBase+m_n)), ec);\r\n    \t\tif (ec) { AbortOut(__LINE__); return; }\r\n    \r\n    \t\tm_outboundSocket.reset(new asio::ip::tcp::socket(m_outboundStrand.get_io_service()));\r\n    \r\n    \t\tTraceOut(__LINE__);\r\n    \r\n    \t\tasio::async_connect(*m_outboundSocket, endpoint,\r\n    \t\t\tm_outboundStrand.wrap([this](const std::error_code& i_error, const asio::ip::tcp::resolver::iterator& i_ep)\r\n    \t\t{\r\n    \t\t\tOnOutboundConnected(i_error, i_ep);\r\n    \t\t}));\r\n    \t}\r\n    \r\n    \tvoid OnOutboundConnected(const asio::error_code& i_error, const asio::ip::tcp::resolver::iterator& i_endpoint)\r\n    \t{\r\n    \t\tTraceOut(__LINE__);\r\n    \r\n    \t\tif (i_error) { AbortOut(__LINE__); return; }\r\n    \r\n    \t\tstd::ostream(&m_outboundTxBuf) << \"msg\" << '\\n';\r\n    \r\n    \t\tasio::async_write(*m_outboundSocket, m_outboundTxBuf.data(),\r\n    \t\t\tm_outboundStrand.wrap([this](const asio::error_code& i_error, size_t i_nWritten)\r\n    \t\t{\r\n    \t\t\tOnOutboundWriteCompleted(i_error, i_nWritten);\r\n    \t\t}));\r\n    \t}\r\n    \r\n    \tvoid OnOutboundWriteCompleted(const asio::error_code& i_error, size_t i_nWritten)\r\n    \t{\r\n    \t\tTraceOut(__LINE__);\r\n    \r\n    \t\tif (i_error) { AbortOut(__LINE__); return; }\r\n    \t\tif (i_nWritten != 4) { AbortOut(__LINE__); return; } // \"msg\\n\"\r\n    \r\n    \t\tTraceOut(__LINE__);\r\n    \t\tm_outboundSocket->shutdown(asio::socket_base::shutdown_both);\r\n    \r\n    \t\tasio::async_read_until(*m_outboundSocket, m_outboundRxBuf, '\\n',\r\n    \t\t\tm_outboundStrand.wrap([this](const asio::error_code& i_error, size_t i_nRead)\r\n    \t\t{\r\n    \t\t\tOnOutboundWaitCompleted(i_error, i_nRead);\r\n    \t\t}));\r\n    \t}\r\n    \r\n    \tvoid OnOutboundWaitCompleted(const asio::error_code& i_error, size_t i_nRead)\r\n    \t{\r\n    \t\tTraceOut(__LINE__);\r\n    \r\n    \t\tif (i_error != asio::error::eof) { AbortOut(__LINE__); return; }\r\n    \t\tif (i_nRead != 0) { AbortOut(__LINE__); return; }\r\n    \r\n    \t\tif (s_freeOutboundSocket)\r\n    \t\t{\r\n    \t\t\tm_outboundSocket.reset();\r\n    \t\t}\r\n    \t}\r\n    \r\n    private:\r\n    \t//////////////\r\n    \t// Inbound\r\n    \r\n    \tasio::io_service::strand m_inboundStrand;\r\n    \r\n    \tasio::ip::tcp::acceptor m_listener;\r\n    \tstd::unique_ptr<asio::ip::tcp::socket> m_inboundSocket;\r\n    \r\n    \tasio::streambuf m_inboundRxBuf;\r\n    \tasio::streambuf m_inboundTxBuf;\r\n    \r\n    \t//////////////\r\n    \t// Outbound\r\n    \r\n    \tasio::io_service::strand m_outboundStrand;\r\n    \r\n    \tasio::ip::tcp::resolver m_resolver;\r\n    \tstd::unique_ptr<asio::ip::tcp::socket> m_outboundSocket;\r\n    \r\n    \tasio::streambuf m_outboundRxBuf;\r\n    \tasio::streambuf m_outboundTxBuf;\r\n    \r\n    \t//////////////\r\n    \t// Common\r\n    \r\n    \tunsigned m_n;\r\n    \r\n    \tconst std::chrono::high_resolution_clock::time_point m_traceStart;\r\n    \tstd::vector<std::pair<unsigned, std::chrono::nanoseconds>> m_inboundTrace;\r\n    \tstd::vector<std::pair<unsigned, std::chrono::nanoseconds>> m_outboundTrace;\r\n    };\r\n    \r\n    static int Usage(int i_ret)\r\n    {\r\n    \tstd::cout << \"[\" << i_ret << \"]\" << \"Usage: example <nThreads> <nConnections> <inboundFree> <outboundFree>\" << std::endl;\r\n    \treturn i_ret;\r\n    }\r\n    \r\n    int main(int argc, char* argv[])\r\n    {\r\n    \tif (argc < 5)\r\n    \t\treturn Usage(__LINE__);\r\n    \r\n    \tconst unsigned nThreads = unsigned(std::stoul(argv[1]));\r\n    \tif (nThreads == 0)\r\n    \t\treturn Usage(__LINE__);\r\n    \tconst unsigned nConnections = unsigned(std::stoul(argv[2]));\r\n    \tif (nConnections == 0)\r\n    \t\treturn Usage(__LINE__);\r\n    \r\n    \ts_freeInboundSocket = (*argv[3] == 'y');\r\n    \ts_freeOutboundSocket = (*argv[4] == 'y');\r\n    \t\r\n    \tconst uint16_t listenPortBase = 25000;\r\n    \tconst uint16_t connectPortBase = 25000;\r\n    \tconst std::string connectHost = \"127.0.0.1\";\r\n    \r\n    \tasio::io_service ioService;\r\n    \r\n    \tstd::cout << \"Creating.\" << std::endl;\r\n    \t\r\n    \tstd::list<Tester> testers;\r\n    \r\n    \tfor (unsigned i = 0; i < nConnections; ++i)\r\n    \t{\r\n    \t\ttesters.emplace_back(ioService, i);\r\n    \t\ttesters.back().Listen(listenPortBase);\r\n    \t\ttesters.back().Connect(connectHost, connectPortBase);\r\n    \t}\r\n    \t\r\n    \tstd::cout << \"Starting.\" << std::endl;\r\n    \r\n    \tstd::vector<std::thread> threads;\r\n    \t\r\n    \tfor (unsigned i = 0; i < nThreads; ++i)\r\n    \t{\t\r\n    \t\tthreads.emplace_back([&]()\r\n    \t\t{\r\n    \t\t\tioService.run();\r\n    \t\t});\r\n    \t}\r\n    \r\n    \tstd::cout << \"Waiting.\" << std::endl;\r\n    \r\n    \tfor (auto& thread : threads)\r\n    \t{\r\n    \t\tthread.join();\r\n    \t}\r\n    \r\n    \tstd::cout << \"Stopped.\" << std::endl;\r\n    \r\n    \treturn 0;\r\n    }\r\n    \r\n    void DumpAllTraces(std::list<Tester>& i_testers)\r\n    {\r\n    \tstd::map<unsigned, unsigned> counts;\r\n    \r\n    \tfor (auto& tester : i_testers)\r\n    \t{\r\n    \t\ttester.DumpTrace(counts);\r\n    \t}\t\r\n    \r\n    \tstd::cout << \"##############################\\n\";\r\n    \tfor (const auto& count : counts)\r\n    \t{\r\n    \t\tstd::cout << count.first << \" : \" << count.second << \"\\n\";\r\n    \t}\r\n    \tstd::cout << std::endl;\r\n    }\r\n    \r\n    #if defined(ASIO_NO_EXCEPTIONS)\r\n    namespace asio\r\n    {\r\n            namespace detail\r\n            {\r\n    \r\n                    template <typename Exception>\r\n                    void throw_exception(const Exception& e)\r\n                    {\r\n    \t\t\tabort();\r\n                    }\r\n    \r\n            } // namespace detail\r\n    } // namespace asio\r\n    #endif\r\n\r\nWe compile as follows (the problem only occurs in optimised builds):\r\n\r\n    g++ -o example -m64 -g -O3 --no-exceptions --no-rtti --std=c++11 -I asio-1.10.6/include -lpthread example.cpp\r\n\r\nWe're running on Debian Jessie.  `uname -a` reports `(Linux <hostname> 3.16.0-4-amd64 #1 SMP Debian 3.16.36-1+deb8u2 (2016-10-19) x86_64 GNU/Linux`.\r\nThe problem appears under both GCC (`g++ (Debian 4.9.2-10) 4.9.2`) and Clang (`Debian clang version 3.5.0-10 (tags/RELEASE_350/final) (based on LLVM 3.5.0)`).\r\n\r\nIn summary, the test app does the following:\r\n\r\n - We create N connections, each consisting of an inbound (listening)\r\n   end and an outbound (connecting) end.  Each inbound listener is bound\r\n   to a unique port (starting at 25000), and each outbound connector\r\n   uses a system-selected originating port.\r\n   \r\n - The inbound end performs an `async_accept`, and on\r\n   completion issues an `async_read`.  When the read completes it issues\r\n   another async_read that we expect to return `eof`.  When that\r\n   completes, we either free the socket immediately, or leave it as-is\r\n   (with no pending async operations) to be cleaned up by the relevant\r\n   destructors at program exit.  (Note that the listener socket is\r\n   always left as-is, with no pending accept, until exit.)\r\n   \r\n - The outbound end performs an `async_connect`, and on completion issues\r\n   an `async_write`.  When the write completes it issues a `shutdown`\r\n   (specifically, `shutdown(both)`) followed by an `async_read` that we\r\n   expect to return `eof`.  On completion, we once again either leave the\r\n   socket as-is, with no pending operations, or we free it immediately.\r\n   \r\n - Any error or unexpected receive data results in an immediate `abort()`\r\n   call.\r\n   \r\n - The test app lets us specify the number of worker threads for the\r\n   `io_service`, as well as the total number of connections to create, as\r\n   well as flags controlling whether inbound and outbound sockets\r\n   respectively are freed or left as-is.\r\n   \r\n - We run the test app repeatedly, specifying 50 threads and 1000\r\n   connections.\r\n\r\n   i.e. `while ./example 50 1000 n y >out.txt ; do echo -n . ; done`\r\n\r\nIf we specify that all sockets are left as-is, the test loop runs indefinitely.  To avoid muddying the waters with `SO_REUSEADDR` considerations, we take care that no sockets are in `TIME_WAIT` state from a previous test run before we start the test, otherwise the listens can fail.  But with this caveat satisfied, the test app runs literally hundreds, even thousands of times with no error.  Similarly, if we specify that inbound sockets (but NOT outbound sockets) should be explicitly freed, all runs fine.\r\n\r\nHowever, if we specify that outbound sockets should be freed, the app stalls after a variable number of executions - sometimes ten or fewer, sometimes a hundred or more, usually somewhere in between.\r\n\r\nConnecting to the stalled process with GDB, we see that the main thread is waiting to join the worker threads, all but one of the worker threads are idle (waiting on an Asio internal condition variable), and that one worker thread is waiting in Asio's call to `epoll()`.  The internal trace instrumentation verifies that some of the sockets are waiting on async operations to complete - sometimes the initial (inbound) accept, sometimes the (inbound) data read, and sometimes the final inbound or outbound reads that normally complete with `eof`.\r\n\r\nIn all cases, the other end of the connection has successfully done its bit: if the inbound accept is still pending, we see that the corresponding outbound connect has successfully completed, along with the outbound write;  likewise if the inbound data read is pending, the corresponding outbound connect and write have completed; if the inbound EOF read is pending, the outbound shutdown has been performed, and likewise if the outbound EOF read is pending, the inbound EOF read has completed due to the outbound shutdown.\r\n\r\nExamining the process's /proc/N/fdinfo shows that the epoll file descriptor is indeed waiting on the file descriptors indicated by the instrumentation.\r\n\r\nMost puzzlingly, `netstat` shows nonzero RecvQ sizes for the waiting sockets - that is, sockets for which there is a read operation pending are shown to have receive data or close events ready to read.  This is consistent with our instrumentation, in that it shows that write data has been delivered to the inbound socket, but has not yet been read (or alternatively that the outbound shutdown has issued a FIN to the inbound side, but that the EOF has not yet been 'read').\r\n\r\nThis leads me to suspect that Asio's `epoll` bookkeeping - in particular its edge-triggered event management - is getting out of sync somewhere due to a race condition.  Clearly this is more than likely due to incorrect operations on my part, but I can't see where the problem would be.\r\n\r\nAll insights, suggestions, known issues, and pointing-out-glaring-screwups would be greatly appreciated.\r\n\r\n"}}
{"actor":{"display_login":"chriskohlhoff"}}