{"action":"closed","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/27447","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/27447/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/27447/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/27447/events","html_url":"https://github.com/elastic/elasticsearch/issues/27447","id":275070423,"number":27447,"title":"Elasticsearch freezes on nested agg with tiny buckets and Painless","user":{"login":"cwurm","id":694597,"avatar_url":"https://avatars2.githubusercontent.com/u/694597?v=4","gravatar_id":"","url":"https://api.github.com/users/cwurm","html_url":"https://github.com/cwurm","followers_url":"https://api.github.com/users/cwurm/followers","following_url":"https://api.github.com/users/cwurm/following{/other_user}","gists_url":"https://api.github.com/users/cwurm/gists{/gist_id}","starred_url":"https://api.github.com/users/cwurm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cwurm/subscriptions","organizations_url":"https://api.github.com/users/cwurm/orgs","repos_url":"https://api.github.com/users/cwurm/repos","events_url":"https://api.github.com/users/cwurm/events{/privacy}","received_events_url":"https://api.github.com/users/cwurm/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-11-18T10:38:45Z","updated_at":"2017-11-20T09:18:34Z","closed_at":"2017-11-20T09:18:33Z","author_association":"MEMBER","body":"**Elasticsearch version** (`bin/elasticsearch --version`):\r\nVersion: 6.0.0, Build: 8f0685b/2017-11-10T18:41:22.859Z, JVM: 1.8.0_111\r\n\r\n**Plugins installed**: []\r\nno additional plugins, OSS download\r\n\r\n**JVM version** (`java -version`):\r\njava version \"1.8.0_111\"\r\nJava(TM) SE Runtime Environment (build 1.8.0_111-b14)\r\nJava HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)\r\n\r\n**OS version** (`uname -a` if on a Unix-like system):\r\nDarwin MacBook-Pro.local 17.2.0 Darwin Kernel Version 17.2.0: Fri Sep 29 18:27:05 PDT 2017; root:xnu-4570.20.62~3/RELEASE_X86_64 x86_64\r\n\r\nI've also observed this with two Elastic Cloud clusters, one on 5.5.1 and one on 6.0.0-rc2. The former is decently sized with 3 x 64 GB memory.\r\n\r\n**Description of the problem including expected versus actual behavior**:\r\nWhen using the Time Series Visual Builder (TSVB) in Kibana, I consistently encountered a situation where Elasticsearch would \"freeze up\" indefinitely and require a hard restart.\r\n\r\n**Steps to reproduce**:\r\n\r\n 1. Needs a cluster with some data - not too much, I could reproduce it on a fresh cluster with only 65M of web logs.\r\n 2. Run below query a few times - e.g. paste it into the Console and hit Cmd+Enter a few times very quickly.\r\n 5. Give it a few seconds, and Elasticsearch will get stuck - I haven't seen it come out of this state without being killed. It will respond to curl requests to management APIs (e.g. hot threads), but not to a `/_search`. Kibana's ES connection will turn red.\r\n\r\n**Query**\r\nThis is the query Kibana's TSVB runs (captured with Wireshark).\r\n\r\n***Console format***\r\n```\r\nGET _msearch\r\n{\"index\":\"logstas*\",\"ignore\":[404],\"timeout\":\"90s\",\"requestTimeout\":90000,\"ignoreUnavailable\":true}\r\n{\"size\":0,\"query\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"gte\":1353234127746,\"lte\":1511000527746,\"format\":\"epoch_millis\"}}},{\"bool\":{\"must\":[{\"match_all\":{}}],\"must_not\":[]}}]}},\"aggs\":{\"61ca57f1-469d-11e7-af02-69e470af7417\":{\"filter\":{\"match_all\":{}},\"aggs\":{\"timeseries\":{\"date_histogram\":{\"field\":\"@timestamp\",\"interval\":\"1s\",\"min_doc_count\":0,\"time_zone\":\"America/Los_Angeles\",\"extended_bounds\":{\"min\":1353234127746,\"max\":1511000527746}},\"aggs\":{\"61ca57f2-469d-11e7-af02-69e470af7417\":{\"bucket_script\":{\"buckets_path\":{\"count\":\"_count\"},\"script\":{\"inline\":\"count * 1\",\"lang\":\"expression\"},\"gap_policy\":\"skip\"}},\"d92bba50-cc49-11e7-87cf-99d990a46dbe\":{\"bucket_script\":{\"buckets_path\":{\"c\":\"61ca57f2-469d-11e7-af02-69e470af7417\"},\"script\":{\"inline\":\"params.c\",\"lang\":\"painless\"},\"gap_policy\":\"skip\"}}}}}}}}\r\n```\r\n\r\n***Expanded (same query)***\r\n```\r\nGET _msearch\r\n{\r\n  \"index\": \"logstas*\",\r\n  \"ignore\": [\r\n    404\r\n  ],\r\n  \"timeout\": \"90s\",\r\n  \"requestTimeout\": 90000,\r\n  \"ignoreUnavailable\": true\r\n}\r\n{\r\n  \"size\": 0,\r\n  \"query\": {\r\n    \"bool\": {\r\n      \"must\": [\r\n        {\r\n          \"range\": {\r\n            \"@timestamp\": {\r\n              \"gte\": 1353234127746,\r\n              \"lte\": 1511000527746,\r\n              \"format\": \"epoch_millis\"\r\n            }\r\n          }\r\n        },\r\n        {\r\n          \"bool\": {\r\n            \"must\": [\r\n              {\r\n                \"match_all\": {}\r\n              }\r\n            ],\r\n            \"must_not\": []\r\n          }\r\n        }\r\n      ]\r\n    }\r\n  },\r\n  \"aggs\": {\r\n    \"61ca57f1-469d-11e7-af02-69e470af7417\": {\r\n      \"filter\": {\r\n        \"match_all\": {}\r\n      },\r\n      \"aggs\": {\r\n        \"timeseries\": {\r\n          \"date_histogram\": {\r\n            \"field\": \"@timestamp\",\r\n            \"interval\": \"1s\",\r\n            \"min_doc_count\": 0,\r\n            \"time_zone\": \"America/Los_Angeles\",\r\n            \"extended_bounds\": {\r\n              \"min\": 1353234127746,\r\n              \"max\": 1511000527746\r\n            }\r\n          },\r\n          \"aggs\": {\r\n            \"61ca57f2-469d-11e7-af02-69e470af7417\": {\r\n              \"bucket_script\": {\r\n                \"buckets_path\": {\r\n                  \"count\": \"_count\"\r\n                },\r\n                \"script\": {\r\n                  \"inline\": \"count * 1\",\r\n                  \"lang\": \"expression\"\r\n                },\r\n                \"gap_policy\": \"skip\"\r\n              }\r\n            },\r\n            \"d92bba50-cc49-11e7-87cf-99d990a46dbe\": {\r\n              \"bucket_script\": {\r\n                \"buckets_path\": {\r\n                  \"c\": \"61ca57f2-469d-11e7-af02-69e470af7417\"\r\n                },\r\n                \"script\": {\r\n                  \"inline\": \"params.c\",\r\n                  \"lang\": \"painless\"\r\n                },\r\n                \"gap_policy\": \"skip\"\r\n              }\r\n            }\r\n          }\r\n        }\r\n      }\r\n    }\r\n  }\r\n}\r\n```\r\n\r\n**Hot threads**\r\n\r\nLooking at hot threads, everything is stuck in the same place:\r\n```\r\n50.0% (250.1ms out of 500ms) cpu usage by thread 'elasticsearch[RwyD12E][search][T#6]'\r\n     10/10 snapshots sharing following 21 elements\r\n       org.elasticsearch.common.rounding.Rounding$TimeUnitRounding.nextRoundingValue(Rounding.java:170)\r\n       org.elasticsearch.search.aggregations.bucket.histogram.InternalDateHistogram.nextKey(InternalDateHistogram.java:488)\r\n       org.elasticsearch.search.aggregations.bucket.histogram.InternalDateHistogram.addEmptyBuckets(InternalDateHistogram.java:401)\r\n       org.elasticsearch.search.aggregations.bucket.histogram.InternalDateHistogram.doReduce(InternalDateHistogram.java:440)\r\n       org.elasticsearch.search.aggregations.InternalAggregation.reduce(InternalAggregation.java:120)\r\n       org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:77)\r\n       org.elasticsearch.search.aggregations.bucket.InternalSingleBucketAggregation.doReduce(InternalSingleBucketAggregation.java:108)\r\n       org.elasticsearch.search.aggregations.InternalAggregation.reduce(InternalAggregation.java:120)\r\n       org.elasticsearch.search.aggregations.InternalAggregations.reduce(InternalAggregations.java:77)\r\n       org.elasticsearch.action.search.SearchPhaseController.reduceAggs(SearchPhaseController.java:523)\r\n       org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:500)\r\n       org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(SearchPhaseController.java:417)\r\n       org.elasticsearch.action.search.SearchPhaseController$1.reduce(SearchPhaseController.java:736)\r\n       org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:102)\r\n       org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:45)\r\n       org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:87)\r\n       org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638)\r\n       org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\r\n       java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n       java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n       java.lang.Thread.run(Thread.java:745)\r\n```\r\n \r\n\r\n**Provide logs (if relevant)**:\r\nLots of GC:\r\n```\r\n[2017-11-18T01:30:59,490][INFO ][o.e.n.Node               ] [RwyD12E] started\r\n[2017-11-18T01:30:59,642][INFO ][o.e.g.GatewayService     ] [RwyD12E] recovered [2] indices into cluster_state\r\n[2017-11-18T01:30:59,949][INFO ][o.e.c.r.a.AllocationService] [RwyD12E] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0], [logstash_elk_demo_v1][3]] ...]).\r\n[2017-11-18T01:31:53,426][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][62] overhead, spent [372ms] collecting in the last [1s]\r\n[2017-11-18T01:32:07,474][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][76] overhead, spent [266ms] collecting in the last [1s]\r\n[2017-11-18T01:37:07,502][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][375] overhead, spent [262ms] collecting in the last [1s]\r\n[2017-11-18T01:38:25,745][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][453] overhead, spent [272ms] collecting in the last [1s]\r\n[2017-11-18T01:39:03,843][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][491] overhead, spent [378ms] collecting in the last [1s]\r\n[2017-11-18T01:39:34,916][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][522] overhead, spent [355ms] collecting in the last [1s]\r\n[2017-11-18T01:39:46,939][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][534] overhead, spent [483ms] collecting in the last [1s]\r\n[2017-11-18T01:40:20,008][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][567] overhead, spent [260ms] collecting in the last [1s]\r\n[2017-11-18T01:42:25,358][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][692] overhead, spent [266ms] collecting in the last [1s]\r\n[2017-11-18T01:43:07,436][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][734] overhead, spent [277ms] collecting in the last [1s]\r\n[2017-11-18T01:43:49,559][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][776] overhead, spent [270ms] collecting in the last [1s]\r\n[2017-11-18T01:44:19,643][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][806] overhead, spent [352ms] collecting in the last [1s]\r\n[2017-11-18T01:44:22,882][WARN ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][809] overhead, spent [633ms] collecting in the last [1.2s]\r\n[2017-11-18T01:44:53,071][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][839] overhead, spent [410ms] collecting in the last [1.1s]\r\n[2017-11-18T01:45:28,175][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][874] overhead, spent [314ms] collecting in the last [1s]\r\n[2017-11-18T01:46:09,254][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][915] overhead, spent [312ms] collecting in the last [1s]\r\n[2017-11-18T01:46:48,358][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][954] overhead, spent [388ms] collecting in the last [1s]\r\n[2017-11-18T01:47:35,824][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][old][995][15] duration [7s], collections [1]/[7.3s], total [7s]/[9.1s], memory [3.8gb]->[3.6gb]/[3.9gb], all_pools {[young] [265.7mb]->[25.1mb]/[266.2mb]}{[survivor] [33.2mb]->[0b]/[33.2mb]}{[old] [3.5gb]->[3.6gb]/[3.6gb]}\r\n[2017-11-18T01:47:35,835][WARN ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][995] overhead, spent [7s] collecting in the last [7.3s]\r\n[2017-11-18T01:48:24,318][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][old][1038][17] duration [6.2s], collections [1]/[6.3s], total [6.2s]/[15.5s], memory [3.9gb]->[3.8gb]/[3.9gb], all_pools {[young] [266.2mb]->[148.6mb]/[266.2mb]}{[survivor] [32.5mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]}\r\n[2017-11-18T01:48:24,320][WARN ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][1038] overhead, spent [6.2s] collecting in the last [6.3s]\r\n[2017-11-18T01:48:53,762][INFO ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][old][1060][18] duration [7.3s], collections [1]/[8s], total [7.3s]/[23s], memory [3.9gb]->[3.8gb]/[3.9gb], all_pools {[young] [266.2mb]->[220mb]/[266.2mb]}{[survivor] [28.1mb]->[0b]/[33.2mb]}{[old] [3.6gb]->[3.6gb]/[3.6gb]}\r\n[2017-11-18T01:48:53,775][WARN ][o.e.m.j.JvmGcMonitorService] [RwyD12E] [gc][1060] overhead, spent [7.3s] collecting in the last [8s]\r\n```\r\n"}}
{"actor":{"display_login":"colings86"}}