{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/17826","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/17826/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/17826/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/17826/events","html_url":"https://github.com/cockroachdb/cockroach/issues/17826","id":252078585,"number":17826,"title":"storage: more quota pool deadlocks","user":{"login":"benesch","id":882976,"avatar_url":"https://avatars1.githubusercontent.com/u/882976?v=4","gravatar_id":"","url":"https://api.github.com/users/benesch","html_url":"https://github.com/benesch","followers_url":"https://api.github.com/users/benesch/followers","following_url":"https://api.github.com/users/benesch/following{/other_user}","gists_url":"https://api.github.com/users/benesch/gists{/gist_id}","starred_url":"https://api.github.com/users/benesch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benesch/subscriptions","organizations_url":"https://api.github.com/users/benesch/orgs","repos_url":"https://api.github.com/users/benesch/repos","events_url":"https://api.github.com/users/benesch/events{/privacy}","received_events_url":"https://api.github.com/users/benesch/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-08-22T20:25:15Z","updated_at":"2017-08-22T20:36:26Z","closed_at":null,"body":"I'm able to fairly reliably reproduce a deadlock in the quota pool while performing a ten-node, 2TB restore. Looks very much like #17524, except that #17796 doesn't seem to fix the problem. On the node coordinating the restore, we see a hung goroutine:\r\n\r\n```\r\ngoroutine 3312 [select, 22 minutes]:\r\ngithub.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc4204c0000, 0x7fb93f5f7438, 0xc4242309c0, 0xc4204c0048, 0x3f1, 0xc426483f80, 0x4, 0x4, 0x0, 0x0, ...)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1153 +0x1417\r\ngithub.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc4204c0000, 0x7fb93f5f7438, 0xc4242309c0, 0x3f1, 0xc426483f80, 0x4, 0x4, 0x0, 0x0, 0x0, ...)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:391 +0x2db\r\ngithub.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc4204c0000, 0x7fb93f6504d0, 0xc45309e7b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:455 +0x17b\r\ngithub.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc4204c0000, 0x7fb93f6504d0, 0xc45309e7b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:939 +0x447\r\ngithub.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc4204c0000, 0x7fb93f6504d0, 0xc45309e7b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:804 +0xb46\r\ngithub.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc4204c0000, 0x7fb93f6504d0, 0xc45309e7b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:606 +0x344\r\ngithub.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc420630000, 0x7fb93f6504d0, 0xc45309e780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:435 +0x1f1\r\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(0xc420415b40, 0x7fb93f5f7438, 0xc421e2bbc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:554 +0x1ff\r\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*DB).(github.com/cockroachdb/cockroach/pkg/internal/client.send)-fm(0x7fb93f5f7438, 0xc421e2bbc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:491 +0x83\r\ngithub.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x7fb93f5f7438, 0xc421e2bbc0, 0xc422d217d0, 0xc428eb7800, 0xc424230300, 0xc428ed8320)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:463 +0x103\r\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc420415b40, 0x7fb93f5f7438, 0xc421e2bbc0, 0xc428eb7800, 0xc428ebc900, 0xc422d219e8)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:491 +0x9d\r\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*DB).AdminSplit(0xc420415b40, 0x7fb93f5f7438, 0xc421e2bbc0, 0x1bbbca0, 0xc428ebc8e0, 0x1bbbca0, 0xc428ebc900, 0x2, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:383 +0x98\r\ngithub.com/cockroachdb/cockroach/pkg/ccl/sqlccl.restore.func2(0x0, 0x0)\r\n\t/go/src/github.com/cockroachdb/cockroach/pkg/ccl/sqlccl/restore.go:735 +0x56b\r\ngithub.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc421e2bc00, 0xc421e31730)\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:58 +0x57\r\ncreated by github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup.(*Group).Go\r\n\t/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/errgroup/errgroup.go:66 +0x66\r\n```\r\n\r\nOn the node processing that split, there's a whole bunch of hung goroutines, plus a whole lot of goroutines in `quotaPool.acquire`: https://gist.github.com/benesch/c747301192eb984686fb60fccf24c506\r\n\r\nI'm running with a relatively recent version of #17796, though not exactly the version that merged. https://github.com/benesch/cockroach/tree/quota-pool-deadlock"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/324144351","html_url":"https://github.com/cockroachdb/cockroach/issues/17826#issuecomment-324144351","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/17826","id":324144351,"user":{"login":"nvanbenschoten","id":5438456,"avatar_url":"https://avatars3.githubusercontent.com/u/5438456?v=4","gravatar_id":"","url":"https://api.github.com/users/nvanbenschoten","html_url":"https://github.com/nvanbenschoten","followers_url":"https://api.github.com/users/nvanbenschoten/followers","following_url":"https://api.github.com/users/nvanbenschoten/following{/other_user}","gists_url":"https://api.github.com/users/nvanbenschoten/gists{/gist_id}","starred_url":"https://api.github.com/users/nvanbenschoten/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nvanbenschoten/subscriptions","organizations_url":"https://api.github.com/users/nvanbenschoten/orgs","repos_url":"https://api.github.com/users/nvanbenschoten/repos","events_url":"https://api.github.com/users/nvanbenschoten/events{/privacy}","received_events_url":"https://api.github.com/users/nvanbenschoten/received_events","type":"User","site_admin":false},"created_at":"2017-08-22T20:36:26Z","updated_at":"2017-08-22T20:36:26Z","body":"Unfortunately, this doesn't look like it's the same issue that #17796 was addressing. There are certainly goroutines hung in `quotaPool.acquire`, but here they're properly waiting to acquire a quota. Did the `have been waiting %s attempting to acquire quota` log fire?\r\n\r\n/cc @irfansharif "}}
{"actor":{"display_login":"nvanbenschoten"}}