{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/14835","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/14835/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/14835/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/14835/events","html_url":"https://github.com/cockroachdb/cockroach/issues/14835","id":221165332,"number":14835,"title":"storage: TestTxnPutOutOfOrder failed under stress","user":{"login":"cockroach-teamcity","id":20559710,"avatar_url":"https://avatars3.githubusercontent.com/u/20559710?v=3","gravatar_id":"","url":"https://api.github.com/users/cockroach-teamcity","html_url":"https://github.com/cockroach-teamcity","followers_url":"https://api.github.com/users/cockroach-teamcity/followers","following_url":"https://api.github.com/users/cockroach-teamcity/following{/other_user}","gists_url":"https://api.github.com/users/cockroach-teamcity/gists{/gist_id}","starred_url":"https://api.github.com/users/cockroach-teamcity/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cockroach-teamcity/subscriptions","organizations_url":"https://api.github.com/users/cockroach-teamcity/orgs","repos_url":"https://api.github.com/users/cockroach-teamcity/repos","events_url":"https://api.github.com/users/cockroach-teamcity/events{/privacy}","received_events_url":"https://api.github.com/users/cockroach-teamcity/received_events","type":"User","site_admin":false},"labels":[{"id":335217860,"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/Robot","name":"Robot","color":"5319e7","default":false},{"id":191587409,"url":"https://api.github.com/repos/cockroachdb/cockroach/labels/test-failure","name":"test-failure","color":"e11d21","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-04-12T06:39:02Z","updated_at":"2017-04-12T14:07:31Z","closed_at":null,"body":"SHA: https://github.com/cockroachdb/cockroach/commits/ba49c4f0b28468a436e8c41873c94992d9e6b71b\n\nParameters:\n```\nCOCKROACH_PROPOSER_EVALUATED_KV=true\nTAGS=\nGOFLAGS=\n```\n\nStress build found a failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=217591&tab=buildLog\n\n```\nI170412 06:39:01.487523 33114 gossip/gossip.go:293  [n1] NodeDescriptor set to node_id:1 address:<network_field:\"\" address_field:\"\" > attrs:<> locality:<> \nI170412 06:39:01.491597 33031 storage/split_queue.go:92  [split,s1,r1/1:/M{in-ax},@c4200a6000] splitting at key /System/\"\"\nI170412 06:39:01.491824 33114 storage/store.go:1256  [n1,s1]: failed initial metrics computation: [n1,s1]: system config not yet available\nI170412 06:39:01.492477 33031 storage/replica_command.go:2602  [split,s1,r1/1:/M{in-ax},@c4200a6000] initiating a split of this range at key /System/\"\" [r2]\nI170412 06:39:01.495604 33031 storage/split_queue.go:92  [split,s1,r2/1:/{System/\"\"-Max},@c42045a900] splitting at key /System/tsd\nI170412 06:39:01.495635 33031 storage/replica_command.go:2602  [split,s1,r2/1:/{System/\"\"-Max},@c42045a900] initiating a split of this range at key /System/tsd [r3]\nI170412 06:39:01.497234 33114 storage/replica_command.go:175  [s1,r1/1:/{Min-System/\"\"},@c4200a6000] test injecting error: Test\nW170412 06:39:01.498085 33031 storage/stores.go:218  range not contained in one range: [/Meta2/System/tsd,/System/tsd/NULL), but have [/Min,/System/\"\")\nI170412 06:39:01.498965 33114 util/stop/stopper.go:494  quiescing; tasks left:\n2      kv/txn_coord_sender.go:903\n1      storage/queue.go:523\nI170412 06:39:01.498996 33114 util/stop/stopper.go:494  quiescing; tasks left:\n1      storage/queue.go:523\nW170412 06:39:01.499165 33175 storage/intent_resolver.go:266  failed to resolve intents: node unavailable; try another peer\npanic: llrb: inverted range\n\ngoroutine 33175 [running]:\ngithub.com/cockroachdb/cockroach/vendor/github.com/biogo/store/llrb.(*Tree).DoRange(0xc42005e758, 0xc421081a18, 0x29b4000, 0xc4202d4870, 0x29b4000, 0xc4202d48d0, 0x69a5ef)\n\t/go/src/github.com/cockroachdb/cockroach/vendor/github.com/biogo/store/llrb/llrb.go:505 +0x118\ngithub.com/cockroachdb/cockroach/pkg/util/cache.(*OrderedCache).DoRange(0xc42005e700, 0xc421081bb0, 0x1b96b20, 0xc42016b0c0, 0x1b96b20, 0xc42016b0e0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/cache/cache.go:428 +0xfd\ngithub.com/cockroachdb/cockroach/pkg/kv.(*rangeDescriptorCache).clearOverlappingCachedRangeDescriptors(0xc420ec0b90, 0x7f4638804b90, 0xc420945440, 0xc42159b4a0, 0x0, 0x11)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/range_cache.go:609 +0x3e1\ngithub.com/cockroachdb/cockroach/pkg/kv.(*rangeDescriptorCache).insertRangeDescriptorsLocked(0xc420ec0b90, 0x7f4638804b90, 0xc420945440, 0xc42159b4a0, 0x1, 0x1, 0x7f4638804b00, 0xc420083970)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/range_cache.go:547 +0x91\ngithub.com/cockroachdb/cockroach/pkg/kv.(*rangeDescriptorCache).lookupRangeDescriptorInternal.func3(0x0, 0x0, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/range_cache.go:329 +0x31f\ngithub.com/cockroachdb/cockroach/vendor/golang.org/x/sync/singleflight.(*Group).doCall(0xc420ec0bc8, 0xc420a62460, 0xc420f45af0, 0x6, 0xc42103db80)\n\t/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/singleflight/singleflight.go:93 +0x3c\ncreated by github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/singleflight.(*Group).DoChan\n\t/go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/sync/singleflight/singleflight.go:86 +0x323\n\ngoroutine 1 [chan receive]:\ntesting.(*T).Run(0xc4201785b0, 0x1d42cbd, 0x14, 0x1daad38, 0xc42052fa01)\n\t/usr/local/go/src/testing/testing.go:698 +0x2f4\ntesting.runTests.func1(0xc4201785b0)\n\t/usr/local/go/src/testing/testing.go:882 +0x67\ntesting.tRunner(0xc4201785b0, 0xc42052fb78)\n\t/usr/local/go/src/testing/testing.go:657 +0x96\ntesting.runTests(0xc4200ef220, 0x29aada0, 0x19c, 0x19c, 0x6952c5)\n\t/usr/local/go/src/testing/testing.go:888 +0x2c1\ntesting.(*M).Run(0xc4201f7f20, 0xc42052fd70)\n\t/usr/local/go/src/testing/testing.go:822 +0xfc\ngithub.com/cockroachdb/cockroach/pkg/storage_test.TestMain(0xc42052ff20)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/main_test.go:59 +0x20a\nmain.main()\n\tgithub.com/cockroachdb/cockroach/pkg/storage/_test/_testmain.go:874 +0xf7\n\ngoroutine 17 [syscall, locked to thread]:\nruntime.goexit()\n\t/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1\n\ngoroutine 5 [chan receive]:\ngithub.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).flushDaemon(0x2dffa00)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1020 +0x77\ncreated by github.com/cockroachdb/cockroach/pkg/util/log.init.1\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:583 +0xd2\n\ngoroutine 6 [chan receive]:\ngithub.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0x2dffa00)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1043 +0x5b\ncreated by github.com/cockroachdb/cockroach/pkg/util/log.init.1\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:584 +0xf6\n\ngoroutine 33034 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:490 +0x39e\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc421122080)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33031 [chan receive]:\ngithub.com/cockroachdb/cockroach/pkg/kv.(*rangeDescriptorCache).lookupRangeDescriptorInternal(0xc420ec0b90, 0x7f4638804b90, 0xc420945440, 0xc420f45523, 0x4, 0xd, 0xc42103da80, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/range_cache.go:345 +0x696\ngithub.com/cockroachdb/cockroach/pkg/kv.(*rangeDescriptorCache).LookupRangeDescriptor(0xc420ec0b90, 0x7f4638804b90, 0xc420945440, 0xc420f45523, 0x4, 0xd, 0xc42103da80, 0x3fc3333333333300, 0xc42130a240, 0xc4209c4300, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/range_cache.go:243 +0x92\ngithub.com/cockroachdb/cockroach/pkg/kv.(*DistSender).getDescriptor(0xc42000a800, 0x7f4638804b90, 0xc420945440, 0xc420f45523, 0x4, 0xd, 0xc42103da80, 0xc42130a200, 0x2faf080, 0x12a05f200, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:444 +0x8d\ngithub.com/cockroachdb/cockroach/pkg/kv.(*RangeIterator).Seek(0xc421076048, 0x7f4638804b90, 0xc420945440, 0xc420f45523, 0x4, 0xd, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/range_iter.go:159 +0x1c9\ngithub.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc42000a800, 0x7f4638804b90, 0xc420945440, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42004e9c0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:730 +0x574\ngithub.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc42000a800, 0x7f4638804b90, 0xc420945440, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42004e5b0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:630 +0x31f\ngithub.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc42125c0b0, 0x7f4638804b90, 0xc420945440, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42004e4e0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:426 +0x382\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(0xc4202cc7c0, 0x7f4638804b90, 0xc420945440, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42004e4e0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:528 +0x1f2\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).send(0xc420438140, 0x7f4638804b90, 0xc420945440, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:807 +0x307\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).(github.com/cockroachdb/cockroach/pkg/internal/client.send)-fm(0x7f4638804b90, 0xc420945440, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:413 +0x76\ngithub.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x7f4638804b90, 0xc420945440, 0xc4210770f8, 0xc420507c00, 0x40, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:436 +0xf7\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Run(0xc420438140, 0x7f4638804b90, 0xc420945440, 0xc420507c00, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:413 +0xda\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).adminSplitWithDescriptor.func1(0x7f4638804b90, 0xc420945440, 0xc420438140, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:2672 +0x6db\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn.func1(0x7f4638804b90, 0xc420945440, 0xc420438140, 0xc4204a8698, 0xc420021350, 0xc420021300)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:490 +0x43\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Exec(0xc420438140, 0x7f4638804b90, 0xc420945440, 0xc420010101, 0xc4204a86c0, 0xc4210775d0, 0xa66bd4)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:612 +0xb6\ngithub.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn(0xc4202cc7c0, 0x7f4638804b90, 0xc420945440, 0xc4206957d0, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:491 +0x238\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Replica).adminSplitWithDescriptor(0xc42045a900, 0x7f46387278c0, 0xc42103fe60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc42013b230, ...)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:2673 +0xa73\ngithub.com/cockroachdb/cockroach/pkg/storage.(*splitQueue).process(0xc4204950f0, 0x7f46387278c0, 0xc42103fe60, 0xc42045a900, 0xc421220000, 0x13, 0x20, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/split_queue.go:99 +0x259\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica(0xc42000b700, 0x7f4638804bd0, 0xc4206956b0, 0xc42045a900, 0xc420de4100, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:609 +0x3b8\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1.2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:514 +0xf2\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTask(0xc4200185a0, 0xc421077ea0, 0x0, 0x0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:240 +0x105\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:523 +0x368\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc421122020)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33135 [semacquire]:\nsync.runtime_notifyListWait(0xc420de4790, 0xc40000001f)\n\t/usr/local/go/src/runtime/sema.go:298 +0x10b\nsync.(*Cond).Wait(0xc420de4780)\n\t/usr/local/go/src/sync/cond.go:57 +0x89\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203563f0, 0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xac\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4202cdfe0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33126 [semacquire]:\nsync.runtime_notifyListWait(0xc420de4790, 0xc400000015)\n\t/usr/local/go/src/runtime/sema.go:298 +0x10b\nsync.(*Cond).Wait(0xc420de4780)\n\t/usr/local/go/src/sync/cond.go:57 +0x89\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203563f0, 0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xac\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4202cdc40)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33140 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1355 +0x518\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc42123d310)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33122 [semacquire]:\nsync.runtime_notifyListWait(0xc420de4790, 0xc400000013)\n\t/usr/local/go/src/runtime/sema.go:298 +0x10b\nsync.(*Cond).Wait(0xc420de4780)\n\t/usr/local/go/src/sync/cond.go:57 +0x89\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203563f0, 0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xac\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4202cda20)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33120 [semacquire]:\nsync.runtime_notifyListWait(0xc420de4790, 0xc400000021)\n\t/usr/local/go/src/runtime/sema.go:298 +0x10b\nsync.(*Cond).Wait(0xc420de4780)\n\t/usr/local/go/src/sync/cond.go:57 +0x89\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203563f0, 0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xac\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4202cd9e0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33037 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess(0xc420278480, 0x7f46387ff0c8, 0xc420016e28, 0xed07fc285, 0xc41d438106, 0x2dfe4c0, 0xc420de4100, 0xc4200185a0, 0xc4200a6000, 0xc4200a60f8)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:207 +0x241\ngithub.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1.1(0xc4200a6000, 0xc420393c80)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:269 +0x90\ngithub.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit(0xc420393c80, 0xc420d960a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:350 +0x2ff\ngithub.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:271 +0x29f\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4211220e0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33132 [semacquire]:\nsync.runtime_notifyListWait(0xc420de4790, 0xc40000001c)\n\t/usr/local/go/src/runtime/sema.go:298 +0x10b\nsync.(*Cond).Wait(0xc420de4780)\n\t/usr/local/go/src/sync/cond.go:57 +0x89\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203563f0, 0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xac\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4202cdf00)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33036 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:490 +0x39e\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4211220c0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33118 [select]:\ngithub.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).printStatsLoop(0xc42125c0b0, 0x7f46387ff0c8, 0xc420016e28)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:207 +0xb4a\ngithub.com/cockroachdb/cockroach/pkg/kv.NewTxnCoordSender.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:191 +0x98\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc420494690)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33125 [semacquire]:\nsync.runtime_notifyListWait(0xc420de4790, 0xc400000016)\n\t/usr/local/go/src/runtime/sema.go:298 +0x10b\nsync.(*Cond).Wait(0xc420de4780)\n\t/usr/local/go/src/sync/cond.go:57 +0x89\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203563f0, 0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xac\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4202cdaa0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33127 [semacquire]:\nsync.runtime_notifyListWait(0xc420de4790, 0xc400000017)\n\t/usr/local/go/src/runtime/sema.go:298 +0x10b\nsync.(*Cond).Wait(0xc420de4780)\n\t/usr/local/go/src/sync/cond.go:57 +0x89\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203563f0, 0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xac\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4202cdc60)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33033 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:490 +0x39e\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc421122060)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33032 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:490 +0x39e\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc421122040)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33136 [semacquire]:\nsync.runtime_Semacquire(0xc420356474)\n\t/usr/local/go/src/runtime/sema.go:47 +0x34\nsync.(*WaitGroup).Wait(0xc420356468)\n\t/usr/local/go/src/sync/waitgroup.go:131 +0x7a\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(0xc4203563f0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:171 +0x31\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait-fm()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3439 +0x2a\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc420495af0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33129 [semacquire]:\nsync.runtime_notifyListWait(0xc420de4790, 0xc40000001a)\n\t/usr/local/go/src/runtime/sema.go:298 +0x10b\nsync.(*Cond).Wait(0xc420de4780)\n\t/usr/local/go/src/sync/cond.go:57 +0x89\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203563f0, 0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xac\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4202cde80)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33134 [semacquire]:\nsync.runtime_notifyListWait(0xc420de4790, 0xc40000001e)\n\t/usr/local/go/src/runtime/sema.go:298 +0x10b\nsync.(*Cond).Wait(0xc420de4780)\n\t/usr/local/go/src/sync/cond.go:57 +0x89\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203563f0, 0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xac\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4202cdfc0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33121 [semacquire]:\nsync.runtime_notifyListWait(0xc420de4790, 0xc400000012)\n\t/usr/local/go/src/runtime/sema.go:298 +0x10b\nsync.(*Cond).Wait(0xc420de4780)\n\t/usr/local/go/src/sync/cond.go:57 +0x89\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203563f0, 0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xac\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4202cda00)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33103 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*idAllocator).start.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/id_alloc.go:133 +0x79c\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc42032d220)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33138 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).startCoalescedHeartbeatsLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3487 +0x167\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc420495b10)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33142 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1355 +0x518\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc42123d3b0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33114 [semacquire]:\nsync.runtime_notifyListWait(0xc420de4010, 0xc400000001)\n\t/usr/local/go/src/runtime/sema.go:298 +0x10b\nsync.(*Cond).Wait(0xc420de4000)\n\t/usr/local/go/src/sync/cond.go:57 +0x89\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Quiesce(0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:496 +0x1f2\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop(0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:439 +0x1f2\ngithub.com/cockroachdb/cockroach/pkg/storage_test.TestTxnPutOutOfOrder(0xc420394000)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/client_replica_test.go:319 +0xc38\ntesting.tRunner(0xc420394000, 0x1daad38)\n\t/usr/local/go/src/testing/testing.go:657 +0x96\ncreated by testing.(*T).Run\n\t/usr/local/go/src/testing/testing.go:697 +0x2ca\n\ngoroutine 33173 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).maybeAddToPurgatory.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:671 +0xb17\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4204aeea0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33123 [semacquire]:\nsync.runtime_notifyListWait(0xc420de4790, 0xc400000020)\n\t/usr/local/go/src/runtime/sema.go:298 +0x10b\nsync.(*Cond).Wait(0xc420de4780)\n\t/usr/local/go/src/sync/cond.go:57 +0x89\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203563f0, 0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xac\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4202cda40)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33035 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:490 +0x39e\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4211220a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33139 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1226 +0x165\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc420bde000)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33030 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:490 +0x39e\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc421122000)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33131 [semacquire]:\nsync.runtime_notifyListWait(0xc420de4790, 0xc400000018)\n\t/usr/local/go/src/runtime/sema.go:298 +0x10b\nsync.(*Cond).Wait(0xc420de4780)\n\t/usr/local/go/src/sync/cond.go:57 +0x89\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203563f0, 0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xac\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4202cdec0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33137 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3456 +0x337\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc420495b00)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33119 [chan receive]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:155 +0x55\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4202cd9c0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33141 [select]:\ngithub.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1355 +0x518\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc42123d360)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33115 [chan receive]:\ngithub.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open.func2(0xc4202dcdc0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:430 +0x44\ncreated by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:431 +0x4c1\n\ngoroutine 33124 [semacquire]:\nsync.runtime_notifyListWait(0xc420de4790, 0xc400000014)\n\t/usr/local/go/src/runtime/sema.go:298 +0x10b\nsync.(*Cond).Wait(0xc420de4780)\n\t/usr/local/go/src/sync/cond.go:57 +0x89\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203563f0, 0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xac\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4202cda80)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33128 [semacquire]:\nsync.runtime_notifyListWait(0xc420de4790, 0xc400000019)\n\t/usr/local/go/src/runtime/sema.go:298 +0x10b\nsync.(*Cond).Wait(0xc420de4780)\n\t/usr/local/go/src/sync/cond.go:57 +0x89\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203563f0, 0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xac\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4202cde60)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33133 [semacquire]:\nsync.runtime_notifyListWait(0xc420de4790, 0xc40000001d)\n\t/usr/local/go/src/runtime/sema.go:298 +0x10b\nsync.(*Cond).Wait(0xc420de4780)\n\t/usr/local/go/src/sync/cond.go:57 +0x89\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203563f0, 0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xac\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4202cdfa0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\ngoroutine 33130 [semacquire]:\nsync.runtime_notifyListWait(0xc420de4790, 0xc40000001b)\n\t/usr/local/go/src/runtime/sema.go:298 +0x10b\nsync.(*Cond).Wait(0xc420de4780)\n\t/usr/local/go/src/sync/cond.go:57 +0x89\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc4203563f0, 0xc4200185a0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:196 +0xac\ngithub.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2()\n\t/go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:165 +0x33\ngithub.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc4200185a0, 0xc4202cdea0)\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:210 +0x7d\ncreated by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker\n\t/go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:211 +0x66\n\n\nERROR: exit status 2\n\nmake: *** [stress] Error 1\n73 runs completed, 1 failures, over 9m31s\nMakefile:173: recipe for target 'stress' failed\n```"},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/293588172","html_url":"https://github.com/cockroachdb/cockroach/issues/14835#issuecomment-293588172","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/14835","id":293588172,"user":{"login":"a-robinson","id":7085343,"avatar_url":"https://avatars0.githubusercontent.com/u/7085343?v=3","gravatar_id":"","url":"https://api.github.com/users/a-robinson","html_url":"https://github.com/a-robinson","followers_url":"https://api.github.com/users/a-robinson/followers","following_url":"https://api.github.com/users/a-robinson/following{/other_user}","gists_url":"https://api.github.com/users/a-robinson/gists{/gist_id}","starred_url":"https://api.github.com/users/a-robinson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a-robinson/subscriptions","organizations_url":"https://api.github.com/users/a-robinson/orgs","repos_url":"https://api.github.com/users/a-robinson/repos","events_url":"https://api.github.com/users/a-robinson/events{/privacy}","received_events_url":"https://api.github.com/users/a-robinson/received_events","type":"User","site_admin":false},"created_at":"2017-04-12T14:07:31Z","updated_at":"2017-04-12T14:07:31Z","body":"It's very possible my changes to the key in that test broke something. I'll take a look this afternoon if it can wait a bit."}}
{"actor":{"display_login":"a-robinson"}}