{"action":"reopened","issue":{"url":"https://api.github.com/repos/edenhill/librdkafka/issues/1569","repository_url":"https://api.github.com/repos/edenhill/librdkafka","labels_url":"https://api.github.com/repos/edenhill/librdkafka/issues/1569/labels{/name}","comments_url":"https://api.github.com/repos/edenhill/librdkafka/issues/1569/comments","events_url":"https://api.github.com/repos/edenhill/librdkafka/issues/1569/events","html_url":"https://github.com/edenhill/librdkafka/issues/1569","id":277918481,"number":1569,"title":"CPU utilization increases to 100% when broker is down/unavailable","user":{"login":"suprajhaS","id":10994114,"avatar_url":"https://avatars1.githubusercontent.com/u/10994114?v=4","gravatar_id":"","url":"https://api.github.com/users/suprajhaS","html_url":"https://github.com/suprajhaS","followers_url":"https://api.github.com/users/suprajhaS/followers","following_url":"https://api.github.com/users/suprajhaS/following{/other_user}","gists_url":"https://api.github.com/users/suprajhaS/gists{/gist_id}","starred_url":"https://api.github.com/users/suprajhaS/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/suprajhaS/subscriptions","organizations_url":"https://api.github.com/users/suprajhaS/orgs","repos_url":"https://api.github.com/users/suprajhaS/repos","events_url":"https://api.github.com/users/suprajhaS/events{/privacy}","received_events_url":"https://api.github.com/users/suprajhaS/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-11-29T21:45:22Z","updated_at":"2017-11-29T22:47:36Z","closed_at":null,"author_association":"NONE","body":"Description\r\n===========\r\n\r\n\r\nThe CPU usage increases to 100% while trying to send messages to a kafka cluster when brokers are unavailable.\r\nWe recently had some issues with our kafka brokers going down temporarily and the CPU usage on the machines producing messages to the brokers spiked up to 100% (the top process was nginx which writes to Kafka) .\r\n\r\nThe CPU usage was still at a 100% even after the brokers were up and running. \r\n```\r\n 2285 ubuntu    20   0 2064976 471128 132608 S 139.9  6.2   9659:41 nginx\r\n 2287 ubuntu    20   0 1998352 388580 132916 S  57.1  5.1   1416:16 nginx\r\n 2286 ubuntu    20   0 1998304 390072 132896 S  48.5  5.1   1417:28 nginx\r\n```\r\n\r\nOn debugging, we discovered that the thread which was utilizing 100% of the CPU was a `librdkafka` thread which kept spinning while trying to connect to a broker. \r\nThis is the stack trace of the thread -\r\n\r\n```\r\n#0  0x00007f98db1124ee in __pthread_mutex_unlock_usercnt (decr=1, mutex=mutex@entry=0x7f98b401af68)\r\n    at pthread_mutex_unlock.c:55\r\n#1  __GI___pthread_mutex_unlock (mutex=mutex@entry=0x7f98b401af68) at pthread_mutex_unlock.c:314\r\n#2  0x00007f98d72bd5e9 in mtx_unlock (mtx=mtx@entry=0x7f98b401af68) at tinycthread.c:284\r\n#3  0x00007f98d72744a7 in rd_refcnt_get (R=0x7f98b401af68) at rd.h:307\r\n#4  rd_kafka_broker_thread_main (arg=arg@entry=0x7f98b401adc0) at rdkafka_broker.c:3070\r\n#5  0x00007f98d72bd4d7 in _thrd_wrapper_function (aArg=<optimized out>) at tinycthread.c:624\r\n#6  0x00007f98db10e6ba in start_thread (arg=0x7f98aeffd700) at pthread_create.c:333\r\n#7  0x00007f98d9f1d3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109\r\n```\r\n\r\nHow to reproduce\r\n================\r\n Produce messages to a cluster when brokers are down and pbserve CPU after broker is back up.\r\n\r\n\r\nChecklist\r\n=========\r\n\r\n\r\nPlease provide the following information:\r\n\r\n - [x] librdkafka version (release number or git tag): v0.10.1.1\r\n - [x] Apache Kafka version:  0.10.1.1\r\n - [x] librdkafka client configuration: `batch.num.messages=10000;queue.buffering.max.messages=1000000;log.connection.close=false`\r\n - [x] Operating system: Ubuntu 16.04\r\n - [ ] Provide logs (with `debug=..` as necessary) from librdkafka\r\n - [ ] Provide broker log excerpts\r\n - [x] Critical issue: Yes\r\n\r\n"}}
{"actor":{"display_login":"suprajhaS"}}