{"action":"created","issue":{"url":"https://api.github.com/repos/jaegertracing/jaeger/issues/589","repository_url":"https://api.github.com/repos/jaegertracing/jaeger","labels_url":"https://api.github.com/repos/jaegertracing/jaeger/issues/589/labels{/name}","comments_url":"https://api.github.com/repos/jaegertracing/jaeger/issues/589/comments","events_url":"https://api.github.com/repos/jaegertracing/jaeger/issues/589/events","html_url":"https://github.com/jaegertracing/jaeger/issues/589","id":281268373,"number":589,"title":"[es] Span write fails if index already exists due to race condition","user":{"login":"jackychee","id":11421069,"avatar_url":"https://avatars1.githubusercontent.com/u/11421069?v=4","gravatar_id":"","url":"https://api.github.com/users/jackychee","html_url":"https://github.com/jackychee","followers_url":"https://api.github.com/users/jackychee/followers","following_url":"https://api.github.com/users/jackychee/following{/other_user}","gists_url":"https://api.github.com/users/jackychee/gists{/gist_id}","starred_url":"https://api.github.com/users/jackychee/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jackychee/subscriptions","organizations_url":"https://api.github.com/users/jackychee/orgs","repos_url":"https://api.github.com/users/jackychee/repos","events_url":"https://api.github.com/users/jackychee/events{/privacy}","received_events_url":"https://api.github.com/users/jackychee/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2017-12-12T06:05:52Z","updated_at":"2017-12-23T18:43:18Z","closed_at":null,"author_association":"NONE","body":"ES version is 5.1.1\r\n{\r\n    \"level\": \"error\", \r\n    \"ts\": 1513045072.7463179, \r\n    \"caller\": \"app/span_processor.go:103\", \r\n    \"msg\": \"Failed to save span\", \r\n    \"error\": \"Failed to create index: elastic: Error 400 (Bad Request): index [jaeger-service-2017-12-12/CV-EQal-Q1KQ3vEyxG1W7g] already exists [type=index_already_exists_exception]\", \r\n    \"errorVerbose\": \"elastic: Error 400 (Bad Request): index [jaeger-service-2017-12-12/CV-EQal-Q1KQ3vEyxG1W7g] already exists [type=index_already_exists_exception]\r\nFailed to create index\r\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanWriter).logError\r\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/writer.go:191\r\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanWriter).createIndex\r\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/writer.go:148\r\ngithub.com/jaegertracing/jaeger/plugin/storage/es/spanstore.(*SpanWriter).WriteSpan\r\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/plugin/storage/es/spanstore/writer.go:120\r\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).saveSpan\r\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:102\r\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).(github.com/jaegertracing/jaeger/cmd/collector/app.saveSpan)-fm\r\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:89\r\ngithub.com/jaegertracing/jaeger/cmd/collector/app.ChainedProcessSpan.func1\r\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/model_consumer.go:32\r\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).processItemFromQueue\r\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:126\r\ngithub.com/jaegertracing/jaeger/cmd/collector/app.NewSpanProcessor.func1\r\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:57\r\ngithub.com/jaegertracing/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1\r\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/queue/bounded_queue.go:65\r\nruntime.goexit\r\n\t/home/travis/.gimme/versions/go1.9.linux.amd64/src/runtime/asm_amd64.s:2337\", \r\n    \"stacktrace\": \"github.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).saveSpan\r\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:103\r\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).(github.com/jaegertracing/jaeger/cmd/collector/app.saveSpan)-fm\r\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:89\r\ngithub.com/jaegertracing/jaeger/cmd/collector/app.ChainedProcessSpan.func1\r\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/model_consumer.go:32\r\ngithub.com/jaegertracing/jaeger/cmd/collector/app.(*spanProcessor).processItemFromQueue\r\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:126\r\ngithub.com/jaegertracing/jaeger/cmd/collector/app.NewSpanProcessor.func1\r\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/cmd/collector/app/span_processor.go:57\r\ngithub.com/jaegertracing/jaeger/pkg/queue.(*BoundedQueue).StartConsumers.func1\r\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/pkg/queue/bounded_queue.go:65\"\r\n}\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/jaegertracing/jaeger/issues/comments/353741836","html_url":"https://github.com/jaegertracing/jaeger/issues/589#issuecomment-353741836","issue_url":"https://api.github.com/repos/jaegertracing/jaeger/issues/589","id":353741836,"user":{"login":"yurishkuro","id":3523016,"avatar_url":"https://avatars2.githubusercontent.com/u/3523016?v=4","gravatar_id":"","url":"https://api.github.com/users/yurishkuro","html_url":"https://github.com/yurishkuro","followers_url":"https://api.github.com/users/yurishkuro/followers","following_url":"https://api.github.com/users/yurishkuro/following{/other_user}","gists_url":"https://api.github.com/users/yurishkuro/gists{/gist_id}","starred_url":"https://api.github.com/users/yurishkuro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yurishkuro/subscriptions","organizations_url":"https://api.github.com/users/yurishkuro/orgs","repos_url":"https://api.github.com/users/yurishkuro/repos","events_url":"https://api.github.com/users/yurishkuro/events{/privacy}","received_events_url":"https://api.github.com/users/yurishkuro/received_events","type":"User","site_admin":false},"created_at":"2017-12-23T18:43:18Z","updated_at":"2017-12-23T18:43:18Z","author_association":"OWNER","body":"There is a race condition in the span writer, and the [comment in the code](https://github.com/jaegertracing/jaeger/blob/master/plugin/storage/es/spanstore/writer.go#L142) calls it out:\r\n```\r\n// if there are multiple collectors writing to the same elasticsearch host, if the collectors pass\r\n// the exists check above and try to create the same index all at once, this might fail and\r\n// drop a couple spans (~1 per collector). Creating indices ahead of time alleviates this issue.\r\n```\r\n\r\n@black-adder did we envision any mechanisms for \"creating indices ahead of time\"?\r\n\r\nAnother, possibly a lot simpler alternative, would be to check if the error was specifically \"index already exists\"."}}
{"actor":{"display_login":"yurishkuro"}}