{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/19699","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/19699/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/19699/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/19699/events","html_url":"https://github.com/cockroachdb/cockroach/issues/19699","id":270152464,"number":19699,"title":"storage: Improve reliability of node liveness","user":{"login":"a-robinson","id":7085343,"avatar_url":"https://avatars3.githubusercontent.com/u/7085343?v=4","gravatar_id":"","url":"https://api.github.com/users/a-robinson","html_url":"https://github.com/a-robinson","followers_url":"https://api.github.com/users/a-robinson/followers","following_url":"https://api.github.com/users/a-robinson/following{/other_user}","gists_url":"https://api.github.com/users/a-robinson/gists{/gist_id}","starred_url":"https://api.github.com/users/a-robinson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a-robinson/subscriptions","organizations_url":"https://api.github.com/users/a-robinson/orgs","repos_url":"https://api.github.com/users/a-robinson/repos","events_url":"https://api.github.com/users/a-robinson/events{/privacy}","received_events_url":"https://api.github.com/users/a-robinson/received_events","type":"User","site_admin":false},"labels":[{"id":705715344,"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/bulkio","name":"bulkio","color":"d93f0b","default":false},{"id":596458097,"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/in%20progress","name":"in progress","color":"ededed","default":false},{"id":406290612,"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/performance","name":"performance","color":"fef2c0","default":false},{"id":371606733,"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/stability","name":"stability","color":"b60205","default":false}],"state":"open","locked":false,"assignee":{"login":"a-robinson","id":7085343,"avatar_url":"https://avatars3.githubusercontent.com/u/7085343?v=4","gravatar_id":"","url":"https://api.github.com/users/a-robinson","html_url":"https://github.com/a-robinson","followers_url":"https://api.github.com/users/a-robinson/followers","following_url":"https://api.github.com/users/a-robinson/following{/other_user}","gists_url":"https://api.github.com/users/a-robinson/gists{/gist_id}","starred_url":"https://api.github.com/users/a-robinson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a-robinson/subscriptions","organizations_url":"https://api.github.com/users/a-robinson/orgs","repos_url":"https://api.github.com/users/a-robinson/repos","events_url":"https://api.github.com/users/a-robinson/events{/privacy}","received_events_url":"https://api.github.com/users/a-robinson/received_events","type":"User","site_admin":false},"assignees":[{"login":"a-robinson","id":7085343,"avatar_url":"https://avatars3.githubusercontent.com/u/7085343?v=4","gravatar_id":"","url":"https://api.github.com/users/a-robinson","html_url":"https://github.com/a-robinson","followers_url":"https://api.github.com/users/a-robinson/followers","following_url":"https://api.github.com/users/a-robinson/following{/other_user}","gists_url":"https://api.github.com/users/a-robinson/gists{/gist_id}","starred_url":"https://api.github.com/users/a-robinson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a-robinson/subscriptions","organizations_url":"https://api.github.com/users/a-robinson/orgs","repos_url":"https://api.github.com/users/a-robinson/repos","events_url":"https://api.github.com/users/a-robinson/events{/privacy}","received_events_url":"https://api.github.com/users/a-robinson/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/11","html_url":"https://github.com/cockroachdb/cockroach/milestone/11","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/milestones/11/labels","id":2543483,"number":11,"title":"1.2","description":"","creator":{"login":"dianasaur323","id":5768024,"avatar_url":"https://avatars0.githubusercontent.com/u/5768024?v=4","gravatar_id":"","url":"https://api.github.com/users/dianasaur323","html_url":"https://github.com/dianasaur323","followers_url":"https://api.github.com/users/dianasaur323/followers","following_url":"https://api.github.com/users/dianasaur323/following{/other_user}","gists_url":"https://api.github.com/users/dianasaur323/gists{/gist_id}","starred_url":"https://api.github.com/users/dianasaur323/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dianasaur323/subscriptions","organizations_url":"https://api.github.com/users/dianasaur323/orgs","repos_url":"https://api.github.com/users/dianasaur323/repos","events_url":"https://api.github.com/users/dianasaur323/events{/privacy}","received_events_url":"https://api.github.com/users/dianasaur323/received_events","type":"User","site_admin":false},"open_issues":474,"closed_issues":146,"state":"open","created_at":"2017-05-25T18:08:15Z","updated_at":"2017-11-28T16:04:37Z","due_on":null,"closed_at":null},"comments":31,"created_at":"2017-10-31T23:42:49Z","updated_at":"2017-11-28T18:47:27Z","closed_at":null,"author_association":"OWNER","body":"Opening a tracking/organizational issue for the work behind trying to make node liveness more reliable in clusters with very heavy workloads (e.g. #15332). More thoughts/ideas very welcome.\r\n\r\n**Problem definition**\r\n\r\nNode liveness heartbeats time out when a cluster is overloaded. This typically makes things even worse in the cluster, since nodes losing their liveness prevents pretty much all other work from completing. Slow node liveness heartbeats are particularly common/problematic during bulk I/O jobs like imports or non-rate-limited restores. Slow heartbeats have also become a problem due to GC queue badness in at least one case\r\n\r\n**Options**\r\n\r\n* Detect bad disks on startup and just refuse to run if they’re bad\r\n   * This wouldn’t fix the IMPORT issues we’ve been seeing on GCE - GCE’s disks aren’t that bad\r\n* Implement prioritization of different kinds of work (e.g. liveness > normal KV operations > bulk I/O)\r\n   * Would just doing this for disk I/O be enough? Or would we need network prioritization as well?\r\n   * Likely a pretty large project to propagate priority info around everywhere, and it looks like the only idea of priority that RocksDB has is a low-priority option\r\n* Move node liveness in-memory\r\n   * Would have the broadest benefits, but we’d need to be very careful, since node liveness is intimately tied to correctness. I’ve spent some time thinking about this but haven’t been able to convince myself of a way to make it safe. We may be able to do something that allows heartbeats to work in memory and only put epoch increments to disk, but the liveness range would need a lot more special handling. Details TBD.\r\n   * Even if we did this, bulk I/O still might slow down OLTP workloads more than is acceptable, necessitating other work to keep it under control.\r\n* Reduce GC threshold for node liveness range\r\n   * Seems like an easy, but very small, win. Unlikely to fix much by itself.\r\n* Learn how to detect and avoid I/O throttling / slowdown\r\n   * Something along the lines of TCP’s flow control,  e.g. start slow, measure latency, speed up writes until latency starts increasing beyond acceptable levels, then back off to stabler I/O rates.\r\n* Tweak RocksDB and/or filesystem settings to reduce impact of bulk I/O\r\n   * Peter has already found a couple sysctl settings that help, and there may be others. This wouldn’t really improve node liveness in general, but might be able to avoid the worst issues during imports/restores.\r\n\r\n**Experiments/TODOs**\r\n\r\n- [x] Don’t increment another node’s epoch if the new proposed leaseholder is also not live - insufficient (https://github.com/cockroachdb/cockroach/pull/19436#issuecomment-340057144)\r\n- [x] Cancel node liveness heartbeat contexts less aggressively - insufficient (https://github.com/cockroachdb/cockroach/pull/19436#issuecomment-340057144)\r\n- [ ] Optimistically gossip node liveness in advance of heartbeats succeeding - initially unsuccessful (https://github.com/cockroachdb/cockroach/pull/19436#issuecomment-339206066), but may merit additional consideration\r\n- [ ] Default the node liveness range’s GC threshold much lower\r\n- [ ] Use [low_pri RocksDB WriteOption](https://github.com/facebook/rocksdb/blob/addbd279c271e987778d20e9a6191fa19023be6d/docs/_posts/2017-07-25-rocksdb-5-6-1-released.markdown)? Would need to bump our RocksDB version\r\n- [ ] Disable RocksDB WAL on the temp store? It [supposedly hurt performance](https://reviewable.io/reviews/cockroachdb/cockroach/16716#-KoN20frn39L-cBsVTol:-Koh5qeHan-HW56v4x1a:b-uv3bzp), but might reduce I/O\r\n- [ ] Double check that [the “blocking” of writes caused by IngestExternalFile](https://github.com/facebook/rocksdb/wiki/Creating-and-Ingesting-SST-files#what-happens-when-you-ingest-a-file) isn’t causing problems\r\n- [ ] Additional miscellaneous RocksDB/sysctl setting tweaks\r\n- [ ] Prototype a throttler that adjusts bulk I/O throughput in reaction to the latency of disk syncs\r\n- [ ] Work out details and write RFC for doing node liveness via an in-memory mechanism\r\n- [ ] Work out details and write RFC for some sort of priority mechanism\r\n- [ ] Make command queue work LIFO instead of FIFO for liveness records and intents (https://github.com/cockroachdb/cockroach/issues/19699#issuecomment-340946319)"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/347624304","html_url":"https://github.com/cockroachdb/cockroach/issues/19699#issuecomment-347624304","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/19699","id":347624304,"user":{"login":"a-robinson","id":7085343,"avatar_url":"https://avatars3.githubusercontent.com/u/7085343?v=4","gravatar_id":"","url":"https://api.github.com/users/a-robinson","html_url":"https://github.com/a-robinson","followers_url":"https://api.github.com/users/a-robinson/followers","following_url":"https://api.github.com/users/a-robinson/following{/other_user}","gists_url":"https://api.github.com/users/a-robinson/gists{/gist_id}","starred_url":"https://api.github.com/users/a-robinson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a-robinson/subscriptions","organizations_url":"https://api.github.com/users/a-robinson/orgs","repos_url":"https://api.github.com/users/a-robinson/repos","events_url":"https://api.github.com/users/a-robinson/events{/privacy}","received_events_url":"https://api.github.com/users/a-robinson/received_events","type":"User","site_admin":false},"created_at":"2017-11-28T18:47:27Z","updated_at":"2017-11-28T18:47:27Z","author_association":"OWNER","body":"Two pieces of bad news:\r\n\r\n1. Adding calls to `r->file->Flush()` in `BlockBasedTableWriter::WriteRawBlock()` does not prevent the growth of dirty bytes into the hundreds of megabytes.\r\n2. I set `sysctl vm.dirty_bytes=$[4 * 1024 * 1024]` and `sysctl vm.dirty_background_bytes=$[1 * 1024 * 1024]` after the above experiment had been running for a while and clearly wasn't working out great. It did create the desired dirty bytes behavior and flattened out the tail latencies of small syncs to disk. However, after a few minutes, the small syncs to disk got blocked for **40 seconds**. Throughout this time period, writes to disk were still happening -- the number of dirty bytes was changing every time I checked. This probably means that L0 was full and thus incoming writes were blocked on compactions, which were slow, but that's way longer than I'd ever expect to be blocked."}}
{"actor":{"display_login":"a-robinson"}}