{"action":"opened","issue":{"url":"https://api.github.com/repos/scrapinghub/frontera/issues/266","repository_url":"https://api.github.com/repos/scrapinghub/frontera","labels_url":"https://api.github.com/repos/scrapinghub/frontera/issues/266/labels{/name}","comments_url":"https://api.github.com/repos/scrapinghub/frontera/issues/266/comments","events_url":"https://api.github.com/repos/scrapinghub/frontera/issues/266/events","html_url":"https://github.com/scrapinghub/frontera/issues/266","id":213145465,"number":266,"title":"Strategy worker error","user":{"login":"kazuar","id":422739,"avatar_url":"https://avatars0.githubusercontent.com/u/422739?v=3","gravatar_id":"","url":"https://api.github.com/users/kazuar","html_url":"https://github.com/kazuar","followers_url":"https://api.github.com/users/kazuar/followers","following_url":"https://api.github.com/users/kazuar/following{/other_user}","gists_url":"https://api.github.com/users/kazuar/gists{/gist_id}","starred_url":"https://api.github.com/users/kazuar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kazuar/subscriptions","organizations_url":"https://api.github.com/users/kazuar/orgs","repos_url":"https://api.github.com/users/kazuar/repos","events_url":"https://api.github.com/users/kazuar/events{/privacy}","received_events_url":"https://api.github.com/users/kazuar/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-03-09T19:50:28Z","updated_at":"2017-03-09T19:50:28Z","closed_at":null,"body":"Hello,\r\n\r\nI have frontera setup with Kafka and Hbase.\r\nEvery once in a while I get the following errors in the strategy worker (BCPerHostLimit):\r\n\r\n```\r\n[strategy-worker] last_consumed=0\r\nINFO:strategy-worker:last_consumed=0\r\n[strategy-worker] consumed_since_start=6313\r\nINFO:strategy-worker:consumed_since_start=6313\r\n[strategy-worker] last_consumption_run=Thu Mar  9 18:40:10 2017\r\nINFO:strategy-worker:last_consumption_run=Thu Mar  9 18:40:10 2017\r\n[strategy-worker] Flushing states\r\nINFO:strategy-worker:Flushing states\r\n[strategy-worker] [Errno 32] Broken pipe\r\nNone\r\nERROR:strategy-worker:[Errno 32] Broken pipe\r\nNone\r\n[strategy-worker]   File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/twisted/internet/defer.py\", line 150, in maybeDeferred\r\n    result = f(*args, **kw)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/frontera/worker/strategy.py\", line 247, in flush_states\r\n    self.states_context.flush()\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/frontera/worker/strategy.py\", line 78, in flush\r\n    self._states.flush(force_clear=False)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/frontera/contrib/backends/hbase.py\", line 303, in flush\r\n    b.put(unhexlify(fprint), hb_obj)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/happybase/batch.py\", line 137, in __exit__\r\n    self.send()\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/happybase/batch.py\", line 60, in send\r\n    self._table.connection.client.mutateRows(self._table.name, bms, {})\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/thriftpy/thrift.py\", line 195, in _req\r\n    self._send(_api, **kwargs)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/thriftpy/thrift.py\", line 205, in _send\r\n    args.write(self._oprot)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/thriftpy/thrift.py\", line 153, in write\r\n    oprot.write_struct(self)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 477, in cybin.TCyBinaryProtocol.write_struct (thriftpy/protocol/cybin/cybin.c:7205)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 474, in cybin.TCyBinaryProtocol.write_struct (thriftpy/protocol/cybin/cybin.c:7127)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 212, in cybin.write_struct (thriftpy/protocol/cybin/cybin.c:3342)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 356, in cybin.c_write_val (thriftpy/protocol/cybin/cybin.c:5142)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 115, in cybin.write_list (thriftpy/protocol/cybin/cybin.c:2115)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 362, in cybin.c_write_val (thriftpy/protocol/cybin/cybin.c:5200)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 212, in cybin.write_struct (thriftpy/protocol/cybin/cybin.c:3342)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 353, in cybin.c_write_val (thriftpy/protocol/cybin/cybin.c:5112)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 122, in cybin.write_string (thriftpy/protocol/cybin/cybin.c:2202)\r\n  File \"thriftpy/transport/buffered/cybuffered.pyx\", line 55, in thriftpy.transport.buffered.cybuffered.TCyBufferedTransport.c_write (thriftpy/transport/buffered/cybuffered.c:1753)\r\n  File \"thriftpy/transport/buffered/cybuffered.pyx\", line 80, in thriftpy.transport.buffered.cybuffered.TCyBufferedTransport.c_flush (thriftpy/transport/buffered/cybuffered.c:2147)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/thriftpy/transport/socket.py\", line 129, in write\r\n    self.sock.sendall(buff)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/socket.py\", line 228, in meth\r\n    return getattr(self._sock,name)(*args)\r\n\r\nCRITICAL:strategy-worker:  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/twisted/internet/defer.py\", line 150, in maybeDeferred\r\n    result = f(*args, **kw)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/frontera/worker/strategy.py\", line 247, in flush_states\r\n    self.states_context.flush()\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/frontera/worker/strategy.py\", line 78, in flush\r\n    self._states.flush(force_clear=False)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/frontera/contrib/backends/hbase.py\", line 303, in flush\r\n    b.put(unhexlify(fprint), hb_obj)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/happybase/batch.py\", line 137, in __exit__\r\n    self.send()\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/happybase/batch.py\", line 60, in send\r\n    self._table.connection.client.mutateRows(self._table.name, bms, {})\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/thriftpy/thrift.py\", line 195, in _req\r\n    self._send(_api, **kwargs)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/thriftpy/thrift.py\", line 205, in _send\r\n    args.write(self._oprot)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/thriftpy/thrift.py\", line 153, in write\r\n    oprot.write_struct(self)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 477, in cybin.TCyBinaryProtocol.write_struct (thriftpy/protocol/cybin/cybin.c:7205)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 474, in cybin.TCyBinaryProtocol.write_struct (thriftpy/protocol/cybin/cybin.c:7127)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 212, in cybin.write_struct (thriftpy/protocol/cybin/cybin.c:3342)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 356, in cybin.c_write_val (thriftpy/protocol/cybin/cybin.c:5142)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 115, in cybin.write_list (thriftpy/protocol/cybin/cybin.c:2115)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 362, in cybin.c_write_val (thriftpy/protocol/cybin/cybin.c:5200)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 212, in cybin.write_struct (thriftpy/protocol/cybin/cybin.c:3342)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 353, in cybin.c_write_val (thriftpy/protocol/cybin/cybin.c:5112)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 122, in cybin.write_string (thriftpy/protocol/cybin/cybin.c:2202)\r\n  File \"thriftpy/transport/buffered/cybuffered.pyx\", line 55, in thriftpy.transport.buffered.cybuffered.TCyBufferedTransport.c_write (thriftpy/transport/buffered/cybuffered.c:1753)\r\n  File \"thriftpy/transport/buffered/cybuffered.pyx\", line 80, in thriftpy.transport.buffered.cybuffered.TCyBufferedTransport.c_flush (thriftpy/transport/buffered/cybuffered.c:2147)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/thriftpy/transport/socket.py\", line 129, in write\r\n    self.sock.sendall(buff)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/socket.py\", line 228, in meth\r\n    return getattr(self._sock,name)(*args)\r\n\r\n[strategy-worker] Flushing states\r\nINFO:strategy-worker:Flushing states\r\n[strategy-worker] [Errno 32] Broken pipe\r\nNone\r\nERROR:strategy-worker:[Errno 32] Broken pipe\r\nNone\r\n```\r\n\r\nAnd when I close the strategy worker, I get the following error:\r\n\r\n```\r\nUnhandled Error\r\nTraceback (most recent call last):\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/twisted/internet/base.py\", line 1199, in run\r\n    self.mainLoop()\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/twisted/internet/base.py\", line 1208, in mainLoop\r\n    self.runUntilCurrent()\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/twisted/internet/base.py\", line 850, in runUntilCurrent\r\n    self.fireSystemEvent(\"shutdown\")\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/twisted/internet/base.py\", line 643, in fireSystemEvent\r\n    event.fireEvent()\r\n--- <exception caught here> ---\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/twisted/internet/base.py\", line 414, in fireEvent\r\n    result = callable(*args, **kwargs)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/frontera/worker/strategy.py\", line 251, in stop\r\n    self.strategy.close()\r\n  File \"general_scraper/broadcrawl/__init__.py\", line 117, in close\r\n    self.domain_cache.flush()\r\n  File \"general_scraper/broadcrawl/__init__.py\", line 46, in flush\r\n    self._store_item(b, k, v)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/happybase/batch.py\", line 137, in __exit__\r\n    self.send()\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/happybase/batch.py\", line 60, in send\r\n    self._table.connection.client.mutateRows(self._table.name, bms, {})\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/thriftpy/thrift.py\", line 195, in _req\r\n    self._send(_api, **kwargs)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/thriftpy/thrift.py\", line 205, in _send\r\n    args.write(self._oprot)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/thriftpy/thrift.py\", line 153, in write\r\n    oprot.write_struct(self)\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 477, in cybin.TCyBinaryProtocol.write_struct (thriftpy/protocol/cybin/cybin.c:7205)\r\n\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 474, in cybin.TCyBinaryProtocol.write_struct (thriftpy/protocol/cybin/cybin.c:7127)\r\n\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 212, in cybin.write_struct (thriftpy/protocol/cybin/cybin.c:3342)\r\n\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 356, in cybin.c_write_val (thriftpy/protocol/cybin/cybin.c:5142)\r\n\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 115, in cybin.write_list (thriftpy/protocol/cybin/cybin.c:2115)\r\n\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 362, in cybin.c_write_val (thriftpy/protocol/cybin/cybin.c:5200)\r\n\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 212, in cybin.write_struct (thriftpy/protocol/cybin/cybin.c:3342)\r\n\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 356, in cybin.c_write_val (thriftpy/protocol/cybin/cybin.c:5142)\r\n\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 115, in cybin.write_list (thriftpy/protocol/cybin/cybin.c:2115)\r\n\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 362, in cybin.c_write_val (thriftpy/protocol/cybin/cybin.c:5200)\r\n\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 212, in cybin.write_struct (thriftpy/protocol/cybin/cybin.c:3342)\r\n\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 353, in cybin.c_write_val (thriftpy/protocol/cybin/cybin.c:5112)\r\n\r\n  File \"thriftpy/protocol/cybin/cybin.pyx\", line 122, in cybin.write_string (thriftpy/protocol/cybin/cybin.c:2202)\r\n\r\n  File \"thriftpy/transport/buffered/cybuffered.pyx\", line 55, in thriftpy.transport.buffered.cybuffered.TCyBufferedTransport.c_write (thriftpy/transport/buffered/cybuffered.c:1753)\r\n\r\n  File \"thriftpy/transport/buffered/cybuffered.pyx\", line 80, in thriftpy.transport.buffered.cybuffered.TCyBufferedTransport.c_flush (thriftpy/transport/buffered/cybuffered.c:2147)\r\n\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/site-packages/thriftpy/transport/socket.py\", line 129, in write\r\n    self.sock.sendall(buff)\r\n  File \"/home/ubuntu/miniconda2/lib/python2.7/socket.py\", line 228, in meth\r\n    return getattr(self._sock,name)(*args)\r\nsocket.error: [Errno 32] Broken pipe\r\n\r\nINFO:kafka.producer.kafka:Closing the Kafka producer with 999999999 secs timeout.\r\n```\r\n\r\nI'm trying to understand what the actual error is, where it's coming from and how to debug this.\r\n\r\nA few notes:\r\n1) I'm running Hbase, Kafka and Frontera on a single machine (ubuntu).\r\n2) The implemented SW (strategy worker) is [BCPerHostLimit](https://github.com/scrapinghub/frontera/blob/master/examples/cluster/bc/broadcrawl/__init__.py#L48)\r\n2) It seems to be happening at random times but once it starts it will keep throwing errors until there's nothing left to crawl.\r\n3) There aren't seem to be errors or warnings in the db worker, crawler, kafka logs or hbase thrift logs.\r\n4) I run the SW (strategy worker) with the following command:\r\n```\r\npython -m frontera.worker.strategy --config general_scraper.config.sw --partition-id 0\r\n```\r\n\r\nAny advice or help on how to approach this would be much appreciated.\r\nThanks!\r\n"}}
{"actor":{"display_login":"kazuar"}}