{"action":"created","issue":{"url":"https://api.github.com/repos/thelastpickle/cassandra-reaper/issues/285","repository_url":"https://api.github.com/repos/thelastpickle/cassandra-reaper","labels_url":"https://api.github.com/repos/thelastpickle/cassandra-reaper/issues/285/labels{/name}","comments_url":"https://api.github.com/repos/thelastpickle/cassandra-reaper/issues/285/comments","events_url":"https://api.github.com/repos/thelastpickle/cassandra-reaper/issues/285/events","html_url":"https://github.com/thelastpickle/cassandra-reaper/issues/285","id":277206574,"number":285,"title":"Reaper triggers then immediately aborts repair","user":{"login":"eriksw","id":24530198,"avatar_url":"https://avatars1.githubusercontent.com/u/24530198?v=4","gravatar_id":"","url":"https://api.github.com/users/eriksw","html_url":"https://github.com/eriksw","followers_url":"https://api.github.com/users/eriksw/followers","following_url":"https://api.github.com/users/eriksw/following{/other_user}","gists_url":"https://api.github.com/users/eriksw/gists{/gist_id}","starred_url":"https://api.github.com/users/eriksw/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eriksw/subscriptions","organizations_url":"https://api.github.com/users/eriksw/orgs","repos_url":"https://api.github.com/users/eriksw/repos","events_url":"https://api.github.com/users/eriksw/events{/privacy}","received_events_url":"https://api.github.com/users/eriksw/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-11-27T23:29:54Z","updated_at":"2017-11-30T15:27:10Z","closed_at":null,"author_association":"NONE","body":"When creating a repair run manually using the UI, reaper starts a repair command but then immediately aborts it gets stuck on `Failed to renew lead on segment`.\r\n\r\nI dropped all reaper_db tables before starting cassandra-reaper (v1.0.1) for the first time, and am running only one reaper instance which has jmx access to all nodes.\r\n\r\n    Nov 27 23:09:39 presence-cassandra-1-reaper-1 java[9873]: INFO   [dw-28 - POST /repair_run?clusterName=presence&keyspace=presence&owner=sre&tables=client_macs&repairParallelism=PARALLEL&incrementalRepair=true] i.c.s.RepairUnitService - create new repair unit for cluster 'presence', keyspace 'presence', column families: [client_macs], nodes: [] and datacenters: []\r\n    Nov 27 23:09:40 presence-cassandra-1-reaper-1 java[9873]: INFO   [dw-28 - POST /repair_run?clusterName=presence&keyspace=presence&owner=sre&tables=client_macs&repairParallelism=PARALLEL&incrementalRepair=true] c.d.d.c.u.UUIDs - PID obtained through native call to getpid(): 9873\r\n    Nov 27 23:09:48 presence-cassandra-1-reaper-1 java[9873]: INFO   [dw-25 - PUT /repair_run/0bbd3a40-d3c8-11e7-9220-e77c0745b02b?state=RUNNING] i.c.r.RepairRunResource - modify repair run state called with: id = 0bbd3a40-d3c8-11e7-9220-e77c0745b02b, state = Optional.of(RUNNING)\r\n    Nov 27 23:09:48 presence-cassandra-1-reaper-1 java[9873]: INFO   [dw-25 - PUT /repair_run/0bbd3a40-d3c8-11e7-9220-e77c0745b02b?state=RUNNING] i.c.r.RepairRunResource - Starting run 0bbd3a40-d3c8-11e7-9220-e77c0745b02b\r\n    Nov 27 23:09:48 presence-cassandra-1-reaper-1 java[9873]: INFO   [dw-25 - PUT /repair_run/0bbd3a40-d3c8-11e7-9220-e77c0745b02b?state=RUNNING] i.c.s.RepairManager - Starting a run with id #0bbd3a40-d3c8-11e7-9220-e77c0745b02b with current state 'NOT_STARTED'\r\n    Nov 27 23:09:48 presence-cassandra-1-reaper-1 java[9873]: INFO   [dw-25 - PUT /repair_run/0bbd3a40-d3c8-11e7-9220-e77c0745b02b?state=RUNNING] i.c.s.RepairManager - scheduling repair for repair run #0bbd3a40-d3c8-11e7-9220-e77c0745b02b\r\n    Nov 27 23:09:48 presence-cassandra-1-reaper-1 java[9873]: INFO   [presence:0bbd3a40-d3c8-11e7-9220-e77c0745b02b] i.c.s.RepairRunner - Running segment for range (-6148914691236517202,-6148914691236517202]\r\n    Nov 27 23:09:48 presence-cassandra-1-reaper-1 java[9873]: INFO   [presence:0bbd3a40-d3c8-11e7-9220-e77c0745b02b] i.c.s.RepairRunner - Next segment to run : 0bbd6151-d3c8-11e7-9220-e77c0745b02b\r\n    Nov 27 23:09:48 presence-cassandra-1-reaper-1 java[9873]: INFO   [presence:0bbd3a40-d3c8-11e7-9220-e77c0745b02b:0bbd6151-d3c8-11e7-9220-e77c0745b02b] i.c.j.JmxProxy - Triggering repair of range (-6148914691236517202,2] for keyspace \"presence\" on host 10.7.32.110, with repair parallelism parallel, in cluster with Cassandra version '2.1.17' (can use DATACENTER_AWARE 'true'), for column families: [client_macs]\r\n    Nov 27 23:09:48 presence-cassandra-1-reaper-1 java[9873]: INFO   [presence:0bbd3a40-d3c8-11e7-9220-e77c0745b02b:0bbd6151-d3c8-11e7-9220-e77c0745b02b] i.c.s.SegmentRunner - Repair for segment 0bbd6151-d3c8-11e7-9220-e77c0745b02b started, status wait will timeout in 36000000 millis\r\n    Nov 27 23:09:51 presence-cassandra-1-reaper-1 java[9873]: INFO   [pool-1-thread-1] i.c.s.SegmentRunner - Postponing segment 0bbd6151-d3c8-11e7-9220-e77c0745b02b\r\n    Nov 27 23:09:51 presence-cassandra-1-reaper-1 java[9873]: INFO   [pool-1-thread-1] i.c.s.SegmentRunner - Aborting repair on segment with id 0bbd6151-d3c8-11e7-9220-e77c0745b02b on coordinator 10.7.32.110\r\n    Nov 27 23:10:48 presence-cassandra-1-reaper-1 java[9873]: ERROR  [presence:0bbd3a40-d3c8-11e7-9220-e77c0745b02b:0bbd6151-d3c8-11e7-9220-e77c0745b02b] i.c.s.CassandraStorage - Failed to renew lead on segment 0bbd3a40-d3c8-11e7-9220-e77c0745b02b\r\n    Nov 27 23:11:48 presence-cassandra-1-reaper-1 java[9873]: ERROR  [presence:0bbd3a40-d3c8-11e7-9220-e77c0745b02b:0bbd6151-d3c8-11e7-9220-e77c0745b02b] i.c.s.CassandraStorage - Failed to renew lead on segment 0bbd3a40-d3c8-11e7-9220-e77c0745b02b\r\n    Nov 27 23:12:48 presence-cassandra-1-reaper-1 java[9873]: ERROR  [presence:0bbd3a40-d3c8-11e7-9220-e77c0745b02b:0bbd6151-d3c8-11e7-9220-e77c0745b02b] i.c.s.CassandraStorage - Failed to renew lead on segment 0bbd3a40-d3c8-11e7-9220-e77c0745b02b\r\n    Nov 27 23:13:48 presence-cassandra-1-reaper-1 java[9873]: ERROR  [presence:0bbd3a40-d3c8-11e7-9220-e77c0745b02b:0bbd6151-d3c8-11e7-9220-e77c0745b02b] i.c.s.CassandraStorage - Failed to renew lead on segment 0bbd3a40-d3c8-11e7-9220-e77c0745b02b\r\n\r\nLooking at metrics for the cassandra cluster, it appears that the \"aborted\" repair is in fact still going on.\r\n\r\nAddendum:\r\n\r\nReaper is still stuck, logging `Failed to renew lead on segment` repeatedly, even after the \"aborted\" repair completed and all compaction died down."},"comment":{"url":"https://api.github.com/repos/thelastpickle/cassandra-reaper/issues/comments/348221704","html_url":"https://github.com/thelastpickle/cassandra-reaper/issues/285#issuecomment-348221704","issue_url":"https://api.github.com/repos/thelastpickle/cassandra-reaper/issues/285","id":348221704,"user":{"login":"adejanovski","id":5096002,"avatar_url":"https://avatars1.githubusercontent.com/u/5096002?v=4","gravatar_id":"","url":"https://api.github.com/users/adejanovski","html_url":"https://github.com/adejanovski","followers_url":"https://api.github.com/users/adejanovski/followers","following_url":"https://api.github.com/users/adejanovski/following{/other_user}","gists_url":"https://api.github.com/users/adejanovski/gists{/gist_id}","starred_url":"https://api.github.com/users/adejanovski/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/adejanovski/subscriptions","organizations_url":"https://api.github.com/users/adejanovski/orgs","repos_url":"https://api.github.com/users/adejanovski/repos","events_url":"https://api.github.com/users/adejanovski/events{/privacy}","received_events_url":"https://api.github.com/users/adejanovski/received_events","type":"User","site_admin":false},"created_at":"2017-11-30T15:27:10Z","updated_at":"2017-11-30T15:27:10Z","author_association":"CONTRIBUTOR","body":"Hi @eriksw,\r\n\r\ncould you check the content of the `reaper_db.leader` table when you get such errors ?\r\nFailure to renew the lead means that the lead wasn't \"taken\" or isn't considered as such. \r\nTaking the lead should happen when the instance picks the segment to start working on it. There a lead will be created using an LWT with a TTL of 3 minutes.\r\nThe lead is renewed regularly during the segment processing, using another LWT that expects the row to be present in the `leader` table. \r\n\r\nWhat is odd is that a failure to take the lead would prevent the segment from being picked...\r\n\r\nDo you have a single datacenter or several ? \r\nIf you have several, is the reaper_db keyspace replicated on all datacenters ?\r\nDo you run an incremental repair or a full one ?\r\n\r\nI would gladly take the full log files by email : reaper[at]thelastpickle.com\r\n\r\nThanks !"}}
{"actor":{"display_login":"adejanovski"}}