{"action":"created","issue":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/26293","repository_url":"https://api.github.com/repos/elastic/elasticsearch","labels_url":"https://api.github.com/repos/elastic/elasticsearch/issues/26293/labels{/name}","comments_url":"https://api.github.com/repos/elastic/elasticsearch/issues/26293/comments","events_url":"https://api.github.com/repos/elastic/elasticsearch/issues/26293/events","html_url":"https://github.com/elastic/elasticsearch/issues/26293","id":251334121,"number":26293,"title":"Shards get stuck initializing on a large cluster (5.5.1)","user":{"login":"gndcshv","id":13341666,"avatar_url":"https://avatars0.githubusercontent.com/u/13341666?v=4","gravatar_id":"","url":"https://api.github.com/users/gndcshv","html_url":"https://github.com/gndcshv","followers_url":"https://api.github.com/users/gndcshv/followers","following_url":"https://api.github.com/users/gndcshv/following{/other_user}","gists_url":"https://api.github.com/users/gndcshv/gists{/gist_id}","starred_url":"https://api.github.com/users/gndcshv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gndcshv/subscriptions","organizations_url":"https://api.github.com/users/gndcshv/orgs","repos_url":"https://api.github.com/users/gndcshv/repos","events_url":"https://api.github.com/users/gndcshv/events{/privacy}","received_events_url":"https://api.github.com/users/gndcshv/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2017-08-18T19:12:36Z","updated_at":"2017-08-21T18:32:39Z","closed_at":null,"body":"## Shards get stuck initializing on a large cluster\r\n\r\n**Elasticsearch version**:\r\n```\r\n\"version\": {\r\n  \"number\": \"5.5.1\",\r\n  \"build_hash\": \"19c13d0\",\r\n  \"build_date\": \"2017-07-18T20:44:24.823Z\",\r\n  \"build_snapshot\": false,\r\n  \"lucene_version\": \"6.6.0\"\r\n}\r\n```\r\n\r\n**Plugins installed**: [analysis-icu, analysis-kuromoji, mapper-murmur3, repository-s3, custom-discovery]\r\n\r\n**JVM version** (`java -version`):\r\n```\r\njava version \"1.8.0_144\"\r\nJava(TM) SE Runtime Environment (build 1.8.0_144-b01)\r\n**Java** HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode)\r\n```\r\n\r\n**OS version** (`uname -a`):\r\n```\r\nLinux ... 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux\r\n```\r\n\r\n## Description of the problem and steps to reproduce\r\n\r\nIn this particular case I have a cluster with 3 dedicated masters and 120 data nodes, I create an index of 360 shards and 2 replicas. It is possible to reproduce it on an index with less shards, just takes longer.\r\n\r\nAfter several successful attempts to create 360 shard indices, you get an index that would have a shard stuck in the initializing state. After ~ 15 minutes, problematic shard gets unstuck, allocated, and started.\r\n\r\n## Cluster configuration\r\n\r\nHere's index template that is applied to these indices:\r\n```\r\n{\r\n  \"order\": 0,\r\n  \"template\": \"test360_*\",\r\n  \"settings\": {\r\n    \"index\": {\r\n      \"refresh_interval\": \"15s\",\r\n      \"number_of_shards\": \"360\",\r\n      \"number_of_replicas\": \"2\"\r\n    }\r\n  },\r\n  \"mappings\": {}\r\n}\r\n```\r\n\r\nHere's the request I send to create an index:\r\n```\r\nPOST /test360_19/default\r\n{\r\n  \"msg\": \"hello\"\r\n}\r\n```\r\n\r\n**elasticsearch.yml**:\r\n```\r\nbootstrap.memory_lock: true\r\nnode.max_local_storage_nodes: 1\r\naction.destructive_requires_name: true\r\nnetwork.bind_host:\r\n- _local_\r\n- _global_\r\nnetwork.publish_host: _global_\r\ntransport.tcp.port: 9002\r\nhttp.port: 9004\r\ndiscovery.zen.ping_timeout: 30s\r\ndiscovery.zen.publish_timeout: 60s\r\ndiscovery.zen.fd.ping_interval: 30s\r\ndiscovery.zen.fd.ping_timeout: 30s\r\ndiscovery.zen.fd.ping_retries: 10\r\ndiscovery.zen.minimum_master_nodes: 2\r\nscript.engine.painless.inline: true\r\nscript.engine.painless.stored.search: true\r\ncluster.routing.rebalance.enable: all\r\ncluster.routing.allocation.cluster_concurrent_rebalance: 4\r\ncluster.routing.allocation.node_concurrent_recoveries: 4\r\ncluster.routing.allocation.disk.watermark.low: 80%\r\ncluster.routing.allocation.disk.watermark.high: 85%\r\nthread_pool.bulk.queue_size: 2000\r\nthread_pool.index.queue_size: 2000\r\nthread_pool.search.queue_size: 1000\r\nlogger.org.elasticsearch.indices.recovery: DEBUG\r\nlogger.org.elasticsearch.cluster.routing: DEBUG\r\nlogger.org.elasticsearch.cluster.action: DEBUG\r\nlogger.org.elasticsearch.cluster.service: DEBUG\r\nlogger.org.elasticsearch.indices.cluster: DEBUG\r\nreindex.remote.whitelist: '*:9004'\r\ncluster.name: es_cluster_120\r\nnode.name: node_az3_006a1fb63b6\r\npath.data: /data/es\r\npath.logs: /logs/es\r\nhttp.cors.enabled: true\r\nhttp.cors.allow-origin: '*'\r\ndiscovery.zen.hosts_provider: custom-discovery\r\ncluster.routing.allocation.awareness.attributes: rack_id\r\nnode.attr.rack_id: az3\r\nnode.master: false\r\nnode.data: true\r\nnode.ingest: false\r\n```\r\nThat is for data nodes, master nodes have `node.master: true` and `node.data: false`.\r\n\r\n**Additional settings**:\r\n```\r\n{\r\n    \"persistent\": {},\r\n    \"transient\": {\r\n        \"logger\": {\r\n            \"org\": {\r\n                \"elasticsearch\": {\r\n                    \"TransportService\": {\r\n                        \"tracer\": \"TRACE\"\r\n                    }\r\n                }\r\n            }\r\n        },\r\n        \"transport\": {\r\n            \"tracer\": {\r\n                \"include\": \"internal:index/shard/*\"\r\n            }\r\n        }\r\n    }\r\n}\r\n```\r\n\r\n**jvm.options**:\r\n```\r\n## JVM configuration\r\n\r\n################################################################\r\n## IMPORTANT: JVM heap size\r\n################################################################\r\n##\r\n## You should always set the min and max JVM heap\r\n## size to the same value. For example, to set\r\n## the heap to 4 GB, set:\r\n##\r\n## -Xms4g\r\n## -Xmx4g\r\n##\r\n## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html\r\n## for more information\r\n##\r\n################################################################\r\n\r\n# Xms represents the initial size of total heap space\r\n# Xmx represents the maximum size of total heap space\r\n\r\n-Xms15338M\r\n-Xmx15338M\r\n\r\n################################################################\r\n## Expert settings\r\n################################################################\r\n##\r\n## All settings below this section are considered\r\n## expert settings. Don't tamper with them unless\r\n## you understand what you are doing\r\n##\r\n################################################################\r\n\r\n## GC configuration\r\n-XX:+UseConcMarkSweepGC\r\n-XX:CMSInitiatingOccupancyFraction=75\r\n-XX:+UseCMSInitiatingOccupancyOnly\r\n\r\n\r\n## optimizations\r\n\r\n# pre-touch memory pages used by the JVM during initialization\r\n-XX:+AlwaysPreTouch\r\n\r\n## basic\r\n\r\n# force the server VM (remove on 32-bit client JVMs)\r\n-server\r\n\r\n# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)\r\n-Xss1m\r\n\r\n# set to headless, just in case\r\n-Djava.awt.headless=true\r\n\r\n# ensure UTF-8 encoding by default (e.g. filenames)\r\n-Dfile.encoding=UTF-8\r\n\r\n# use our provided JNA always versus the system one\r\n-Djna.nosys=true\r\n\r\n# use old-style file permissions on JDK9\r\n-Djdk.io.permissionsUseCanonicalPath=true\r\n\r\n# flags to configure Netty\r\n-Dio.netty.noUnsafe=true\r\n-Dio.netty.noKeySetOptimization=true\r\n-Dio.netty.recycler.maxCapacityPerThread=0\r\n\r\n# log4j 2\r\n-Dlog4j.shutdownHookEnabled=false\r\n-Dlog4j2.disable.jmx=true\r\n-Dlog4j.skipJansi=true\r\n\r\n## heap dumps\r\n\r\n# generate a heap dump when an allocation from the Java heap fails\r\n# heap dumps are created in the working directory of the JVM\r\n-XX:+HeapDumpOnOutOfMemoryError\r\n\r\n# specify an alternative path for heap dumps\r\n# ensure the directory exists and has sufficient space\r\n-XX:HeapDumpPath=/logs/es/es_heap_dump.hprof\r\n\r\n## GC logging\r\n\r\n-XX:+PrintGCDetails\r\n-XX:+PrintGCTimeStamps\r\n-XX:+PrintGCDateStamps\r\n-XX:+PrintClassHistogram\r\n-XX:+PrintTenuringDistribution\r\n-XX:+PrintGCApplicationStoppedTime\r\n\r\n# log GC status to a file with time stamps\r\n# ensure the directory exists\r\n-Xloggc:/logs/es/es_gc-%t.log\r\n\r\n# By default, the GC log file will not rotate.\r\n# By uncommenting the lines below, the GC log file\r\n# will be rotated every 128MB at most 32 times.\r\n-XX:+UseGCLogFileRotation\r\n-XX:NumberOfGCLogFiles=16\r\n-XX:GCLogFileSize=128M\r\n\r\n# Elasticsearch 5.0.0 will throw an exception on unquoted field names in JSON.\r\n# If documents were already indexed with unquoted fields in a previous version\r\n# of Elasticsearch, some operations may throw errors.\r\n#\r\n# WARNING: This option will be removed in Elasticsearch 6.0.0 and is provided\r\n# only for migration purposes.\r\n#-Delasticsearch.json.allow_unquoted_field_names=true\r\n```\r\n\r\nCommand line of the actual process:\r\n```\r\nelastic+ 21904     1 10 Aug17 ?        01:57:51 /usr/bin/java -Xms15338M -Xmx15338M -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logs/es/es_heap_dump.hprof -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:/logs/es/es_gc-%t.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=16 -XX:GCLogFileSize=128M -Des.path.home=/apps/elasticsearch -cp /apps/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /run/elasticsearch/elasticsearch.pid --quiet\r\n```\r\n\r\nProcess limits:\r\n```\r\n$ cat /proc/21904/limits\r\nLimit                     Soft Limit           Hard Limit           Units\r\nMax cpu time              unlimited            unlimited            seconds\r\nMax file size             unlimited            unlimited            bytes\r\nMax data size             unlimited            unlimited            bytes\r\nMax stack size            8388608              unlimited            bytes\r\nMax core file size        0                    unlimited            bytes\r\nMax resident set          unlimited            unlimited            bytes\r\nMax processes             1967818              1967818              processes\r\nMax open files            100000               100000               files\r\nMax locked memory         unlimited            unlimited            bytes\r\nMax address space         unlimited            unlimited            bytes\r\nMax file locks            unlimited            unlimited            locks\r\nMax pending signals       1967818              1967818              signals\r\nMax msgqueue size         819200               819200               bytes\r\nMax nice priority         0                    0\r\nMax realtime priority     0                    0\r\nMax realtime timeout      unlimited            unlimited            us\r\n```\r\n\r\n## Logs\r\n\r\nThere's not too many mentions about the problematic shard, however when it gets unstuck I consistently observe the following exception on both the relevant data node and the active master.\r\n\r\n**Data node that had problematic shard**:\r\n```\r\n...\r\n[2017-08-18T17:02:57,201][DEBUG][o.e.c.s.ClusterService   ] [node_az3_006a1fb63b6] applying cluster state version 1171\r\n[2017-08-18T17:02:57,201][DEBUG][o.e.c.s.ClusterService   ] [node_az3_006a1fb63b6] set local cluster state to version 1171\r\n[2017-08-18T17:02:57,212][DEBUG][o.e.c.s.ClusterService   ] [node_az3_006a1fb63b6] processing [zen-disco-receive(from master [master {node_az2_084354b0d5}{86-R436YR56aHOtxCptzjg}{YSViwD-tQhSJqe3y9w6dWw}{100.0.0.2}{100.0.0.2:9002}{rack_id=az2} committed version [1171]])]: took [11ms] done applying updated cluster_state (version: 1171, uuid: -U_RCKbTSIG6kiRDtuWSmg)\r\n[2017-08-18T17:17:55,337][TRACE][o.e.t.T.tracer           ] [node_az3_006a1fb63b6] [3262984][internal:index/shard/recovery/start_recovery] received response from [{node_az2_f79f3497d0}{CTBV-A2tT5ysQ7uVX1vjPw}{4IMXO6pHQ_SNQS5mSzF6kg}{100.0.0.3}{100.0.0.3:9002}{rack_id=az2}]\r\n[2017-08-18T17:17:55,338][WARN ][o.e.i.c.IndicesClusterStateService] [node_az3_006a1fb63b6] [[test360_19][143]] marking and sending shard failed due to [failed recovery]\r\norg.elasticsearch.indices.recovery.RecoveryFailedException: [test360_19][143]: Recovery failed from {node_az2_f79f3497d0}{CTBV-A2tT5ysQ7uVX1vjPw}{4IMXO6pHQ_SNQS5mSzF6kg}{100.0.0.3}{100.0.0.3:9002}{rack_id=az2} into {node_az3_006a1fb63b6}{b-sQwH5dQQixJ9fN27l2eA}{NBnUGKW7R2CUXlI7byGm5g}{100.0.0.1}{100.0.0.1:9002}{rack_id=az3}\r\n\tat org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:314) [elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:73) [elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:556) [elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.5.1.jar:5.5.1]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]\r\n\tat java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]\r\nCaused by: org.elasticsearch.transport.RemoteTransportException: [node_az2_0ff81340d0bc][100.0.0.3:9002][internal:index/shard/recovery/start_recovery]\r\nCaused by: org.elasticsearch.index.engine.RecoveryEngineException: Phase[1] phase1 failed\r\n\tat org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:140) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:132) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$100(PeerRecoverySourceService.java:54) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:141) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:138) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\t... 5 more\r\nCaused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: Failed to transfer [1] files with total size of [162b]\r\n\tat org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:337) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:138) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:132) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$100(PeerRecoverySourceService.java:54) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:141) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:138) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\t... 5 more\r\nCaused by: org.elasticsearch.transport.ReceiveTimeoutTransportException: [node_az3_006a1fb63b6][100.0.0.1:9002][internal:index/shard/recovery/prepare_translog] request_id [1106203] timed out after [900000ms]\r\n\tat org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:951) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\t... 3 more\r\n[2017-08-18T17:17:55,339][DEBUG][o.e.c.a.s.ShardStateAction] [node_az3_006a1fb63b6] [test360_19][143] sending [internal:cluster/shard/failure] to [86-R436YR56aHOtxCptzjg] for shard entry [shard id [[test360_19][143]], allocation id [uLI9lUk3To2entCZ-KQcAw], primary term [0], message [failed recovery], failure [RecoveryFailedException[[test360_19][143]: Recovery failed from {node_az2_f79f3497d0}{CTBV-A2tT5ysQ7uVX1vjPw}{4IMXO6pHQ_SNQS5mSzF6kg}{100.0.0.3}{100.0.0.3:9002}{rack_id=az2} into {node_az3_006a1fb63b6}{b-sQwH5dQQixJ9fN27l2eA}{NBnUGKW7R2CUXlI7byGm5g}{100.0.0.1}{100.0.0.1:9002}{rack_id=az3}]; nested: RemoteTransportException[[node_az2_f79f3497d0][100.0.0.3:9002][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [1] files with total size of [162b]]; nested: ReceiveTimeoutTransportException[[node_az3_006a1fb63b6][100.0.0.1:9002][internal:index/shard/recovery/prepare_translog] request_id [1106203] timed out after [900000ms]]; ]]\r\n[2017-08-18T17:17:55,619][DEBUG][o.e.c.s.ClusterService   ] [node_az3_006a1fb63b6] processing [zen-disco-receive(from master [master {node_az2_084354b0d5}{86-R436YR56aHOtxCptzjg}{YSViwD-tQhSJqe3y9w6dWw}{100.0.0.2}{100.0.0.2:9002}{rack_id=az2} committed version [1172]])]: execute\r\n[2017-08-18T17:17:55,619][DEBUG][o.e.c.s.ClusterService   ] [node_az3_006a1fb63b6] cluster state updated, version [1172], source [zen-disco-receive(from master [master {node_az2_084354b0d5}{86-R436YR56aHOtxCptzjg}{YSViwD-tQhSJqe3y9w6dWw}{100.0.0.2}{100.0.0.2:9002}{rack_id=az2} committed version [1172]])]\r\n[2017-08-18T17:17:55,620][DEBUG][o.e.c.s.ClusterService   ] [node_az3_006a1fb63b6] applying cluster state version 1172\r\n[2017-08-18T17:17:55,620][DEBUG][o.e.c.s.ClusterService   ] [node_az3_006a1fb63b6] set local cluster state to version 1172\r\n...\r\n```\r\n\r\nEven though this is connection related exception, I don't see any other exceptions that would show that there's a genuine network issue between the two nodes.\r\n\r\n**Active master**:\r\n```\r\n...\r\n[2017-08-18T17:02:56,068][DEBUG][o.e.c.a.s.ShardStateAction] [us-east-1d.i-086432e5a1d4b01d5] [test360_19][143] starting shard [test360_19][143], node[ljMLTQZbRce8245_yhGLGA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=FBeIYPFeSWq8HwuzBwdxQw], unassigned_info[[reason=INDEX_CREATED], at[2017-08-18T17:02:53.615Z], delayed=false, allocation_status[no_attempt]] (shard started task: [shard id [[test360_19][143]], allocation id [FBeIYPFeSWq8HwuzBwdxQw], primary term [0], message [after peer recovery]])\r\n...\r\n\r\n[2017-08-18T17:02:57,189][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] processing [put-mapping[default]]: execute\r\n[2017-08-18T17:02:57,192][INFO ][o.e.c.m.MetaDataMappingService] [node_az2_084354b0d5] [test360_19/Okz9GdoYTgqnBBDr0KDRdQ] create_mapping [default]\r\n[2017-08-18T17:02:57,192][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] cluster state updated, version [1171], source [put-mapping[default]]\r\n[2017-08-18T17:02:57,192][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] publishing cluster state version [1171]\r\n[2017-08-18T17:02:57,303][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] applying cluster state version 1171\r\n[2017-08-18T17:02:57,303][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] set local cluster state to version 1171\r\n[2017-08-18T17:02:57,315][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] processing [put-mapping[default]]: took [126ms] done applying updated cluster_state (version: 1171, uuid: -U_RCKbTSIG6kiRDtuWSmg)\r\n[2017-08-18T17:17:55,340][WARN ][o.e.c.a.s.ShardStateAction] [node_az2_084354b0d5] [test360_19][143] received shard failed for shard id [[test360_19][143]], allocation id [uLI9lUk3To2entCZ-KQcAw], primary term [0], message [failed recovery], failure [RecoveryFailedException[[test360_19][143]: Recovery failed from {node_az2_f79f3497d0}{CTBV-A2tT5ysQ7uVX1vjPw}{4IMXO6pHQ_SNQS5mSzF6kg}{100.0.0.3}{100.0.0.3:9002}{rack_id=az2} into {node_az3_006a1fb63b6}{b-sQwH5dQQixJ9fN27l2eA}{NBnUGKW7R2CUXlI7byGm5g}{100.0.0.1}{100.0.0.1:9002}{rack_id=az3}]; nested: RemoteTransportException[[node_az2_f79f3497d0][100.0.0.3:9002][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [1] files with total size of [162b]]; nested: ReceiveTimeoutTransportException[[node_az3_006a1fb63b6][100.0.0.1:9002][internal:index/shard/recovery/prepare_translog] request_id [1106203] timed out after [900000ms]]; ]\r\norg.elasticsearch.indices.recovery.RecoveryFailedException: [test360_19][143]: Recovery failed from {node_az2_f79f3497d0}{CTBV-A2tT5ysQ7uVX1vjPw}{4IMXO6pHQ_SNQS5mSzF6kg}{100.0.0.3}{100.0.0.3:9002}{rack_id=az2} into {node_az3_006a1fb63b6}{b-sQwH5dQQixJ9fN27l2eA}{NBnUGKW7R2CUXlI7byGm5g}{100.0.0.1}{100.0.0.1:9002}{rack_id=az3}\r\n\tat org.elasticsearch.indices.recovery.PeerRecoveryTargetService.doRecovery(PeerRecoveryTargetService.java:314) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoveryTargetService.access$900(PeerRecoveryTargetService.java:73) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoveryTargetService$RecoveryRunner.doRun(PeerRecoveryTargetService.java:556) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_144]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_144]\r\n\tat java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]\r\nCaused by: org.elasticsearch.transport.RemoteTransportException: [node_az2_f79f3497d0][100.0.0.3:9002][internal:index/shard/recovery/start_recovery]\r\nCaused by: org.elasticsearch.index.engine.RecoveryEngineException: Phase[1] phase1 failed\r\n\tat org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:140) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:132) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$100(PeerRecoverySourceService.java:54) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:141) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:138) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\t... 5 more\r\nCaused by: org.elasticsearch.indices.recovery.RecoverFilesRecoveryException: Failed to transfer [1] files with total size of [162b]\r\n\tat org.elasticsearch.indices.recovery.RecoverySourceHandler.phase1(RecoverySourceHandler.java:337) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:138) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoverySourceService.recover(PeerRecoverySourceService.java:132) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoverySourceService.access$100(PeerRecoverySourceService.java:54) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:141) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.indices.recovery.PeerRecoverySourceService$StartRecoveryTransportRequestHandler.messageReceived(PeerRecoverySourceService.java:138) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\t... 5 more\r\nCaused by: org.elasticsearch.transport.ReceiveTimeoutTransportException: [node_az3_006a1fb63b6][100.0.0.1:9002][internal:index/shard/recovery/prepare_translog] request_id [1106203] timed out after [900000ms]\r\n\tat org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:951) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) ~[elasticsearch-5.5.1.jar:5.5.1]\r\n\t... 3 more\r\n[2017-08-18T17:17:55,342][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] processing [shard-failed[shard id [[test360_19][143]], allocation id [uLI9lUk3To2entCZ-KQcAw], primary term [0], message [failed recovery], failure [RecoveryFailedException[[test360_19][143]: Recovery failed from {node_az2_f79f3497d0}{CTBV-A2tT5ysQ7uVX1vjPw}{4IMXO6pHQ_SNQS5mSzF6kg}{100.0.0.3}{100.0.0.3:9002}{rack_id=az2} into {node_az3_006a1fb63b6}{b-sQwH5dQQixJ9fN27l2eA}{NBnUGKW7R2CUXlI7byGm5g}{100.0.0.1}{100.0.0.1:9002}{rack_id=az3}]; nested: RemoteTransportException[[node_az2_f79f3497d0][100.0.0.3:9002][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [1] files with total size of [162b]]; nested: ReceiveTimeoutTransportException[[node_az3_006a1fb63b6][100.0.0.1:9002][internal:index/shard/recovery/prepare_translog] request_id [1106203] timed out after [900000ms]]; ]]]: execute\r\n[2017-08-18T17:17:55,342][DEBUG][o.e.c.a.s.ShardStateAction] [node_az2_084354b0d5] [test360_19][143] failing shard [test360_19][143], node[b-sQwH5dQQixJ9fN27l2eA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=uLI9lUk3To2entCZ-KQcAw], unassigned_info[[reason=INDEX_CREATED], at[2017-08-18T17:02:53.615Z], delayed=false, allocation_status[no_attempt]] (shard failed task: [shard id [[test360_19][143]], allocation id [uLI9lUk3To2entCZ-KQcAw], primary term [0], message [failed recovery], failure [RecoveryFailedException[[test360_19][143]: Recovery failed from {node_az2_f79f3497d0}{CTBV-A2tT5ysQ7uVX1vjPw}{4IMXO6pHQ_SNQS5mSzF6kg}{100.0.0.3}{100.0.0.3:9002}{rack_id=az2} into {node_az3_006a1fb63b6}{b-sQwH5dQQixJ9fN27l2eA}{NBnUGKW7R2CUXlI7byGm5g}{100.0.0.1}{100.0.0.1:9002}{rack_id=az3}]; nested: RemoteTransportException[[node_az2_f79f3497d0][100.0.0.3:9002][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [1] files with total size of [162b]]; nested: ReceiveTimeoutTransportException[[node_az3_006a1fb63b6][100.0.0.1:9002][internal:index/shard/recovery/prepare_translog] request_id [1106203] timed out after [900000ms]]; ]])\r\n[2017-08-18T17:17:55,348][DEBUG][o.e.c.r.a.AllocationService] [node_az2_084354b0d5] [test360_19][143] failing shard [test360_19][143], node[b-sQwH5dQQixJ9fN27l2eA], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=uLI9lUk3To2entCZ-KQcAw], unassigned_info[[reason=INDEX_CREATED], at[2017-08-18T17:02:53.615Z], delayed=false, allocation_status[no_attempt]] with unassigned info ([reason=ALLOCATION_FAILED], at[2017-08-18T17:17:55.348Z], failed_attempts[1], delayed=false, details[failed recovery, failure RecoveryFailedException[[test360_19][143]: Recovery failed from {node_az2_f79f3497d0}{CTBV-A2tT5ysQ7uVX1vjPw}{4IMXO6pHQ_SNQS5mSzF6kg}{100.0.0.3}{100.0.0.3:9002}{rack_id=az2} into {node_az3_006a1fb63b6}{b-sQwH5dQQixJ9fN27l2eA}{NBnUGKW7R2CUXlI7byGm5g}{100.0.0.1}{100.0.0.1:9002}{rack_id=az3}]; nested: RemoteTransportException[[node_az2_f79f3497d0][100.0.0.3:9002][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [1] files with total size of [162b]]; nested: ReceiveTimeoutTransportException[[node_az3_006a1fb63b6][100.0.0.1:9002][internal:index/shard/recovery/prepare_translog] request_id [1106203] timed out after [900000ms]]; ], allocation_status[no_attempt])\r\n[2017-08-18T17:17:55,582][DEBUG][o.e.c.r.a.a.BalancedShardsAllocator] [node_az2_084354b0d5] skipping rebalance due to in-flight shard/store fetches\r\n[2017-08-18T17:17:55,600][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] cluster state updated, version [1172], source [shard-failed[shard id [[test360_19][143]], allocation id [uLI9lUk3To2entCZ-KQcAw], primary term [0], message [failed recovery], failure [RecoveryFailedException[[test360_19][143]: Recovery failed from {node_az2_f79f3497d0}{CTBV-A2tT5ysQ7uVX1vjPw}{4IMXO6pHQ_SNQS5mSzF6kg}{100.0.0.3}{100.0.0.3:9002}{rack_id=az2} into {node_az3_006a1fb63b6}{b-sQwH5dQQixJ9fN27l2eA}{NBnUGKW7R2CUXlI7byGm5g}{100.0.0.1}{100.0.0.1:9002}{rack_id=az3}]; nested: RemoteTransportException[[node_az2_f79f3497d0][100.0.0.3:9002][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [1] files with total size of [162b]]; nested: ReceiveTimeoutTransportException[[node_az3_006a1fb63b6][100.0.0.1:9002][internal:index/shard/recovery/prepare_translog] request_id [1106203] timed out after [900000ms]]; ]]]\r\n[2017-08-18T17:17:55,600][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] publishing cluster state version [1172]\r\n[2017-08-18T17:17:55,843][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] applying cluster state version 1172\r\n[2017-08-18T17:17:55,843][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] set local cluster state to version 1172\r\n[2017-08-18T17:17:55,852][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] processing [shard-failed[shard id [[test360_19][143]], allocation id [uLI9lUk3To2entCZ-KQcAw], primary term [0], message [failed recovery], failure [RecoveryFailedException[[test360_19][143]: Recovery failed from {node_az2_f79f3497d0}{CTBV-A2tT5ysQ7uVX1vjPw}{4IMXO6pHQ_SNQS5mSzF6kg}{100.0.0.3}{100.0.0.3:9002}{rack_id=az2} into {node_az3_006a1fb63b6}{b-sQwH5dQQixJ9fN27l2eA}{NBnUGKW7R2CUXlI7byGm5g}{100.0.0.1}{100.0.0.1:9002}{rack_id=az3}]; nested: RemoteTransportException[[node_az2_f79f3497d0][100.0.0.3:9002][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [1] files with total size of [162b]]; nested: ReceiveTimeoutTransportException[[node_az3_006a1fb63b6][100.0.0.1:9002][internal:index/shard/recovery/prepare_translog] request_id [1106203] timed out after [900000ms]]; ]]]: took [510ms] done applying updated cluster_state (version: 1172, uuid: I7EG7cqgTKGbqp2thv2ZKw)\r\n[2017-08-18T17:17:55,852][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] processing [cluster_reroute(async_shard_fetch)]: execute\r\n[2017-08-18T17:17:56,155][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] cluster state updated, version [1173], source [cluster_reroute(async_shard_fetch)]\r\n[2017-08-18T17:17:56,156][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] publishing cluster state version [1173]\r\n[2017-08-18T17:17:56,205][DEBUG][o.e.c.a.s.ShardStateAction] [node_az2_084354b0d5] [test360_19][143] received shard started for [shard id [[test360_19][143]], allocation id [uAUMBccQRji9i6oeO9yCSA], primary term [0], message [after peer recovery]]\r\n[2017-08-18T17:17:56,459][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] applying cluster state version 1173\r\n[2017-08-18T17:17:56,459][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] set local cluster state to version 1173\r\n[2017-08-18T17:17:56,467][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] processing [cluster_reroute(async_shard_fetch)]: took [614ms] done applying updated cluster_state (version: 1173, uuid: F8Vn_O1RSV-A4I1deTv_8Q)\r\n[2017-08-18T17:17:56,467][DEBUG][o.e.c.s.ClusterService   ] [node_az2_084354b0d5] processing [shard-started shard id [[test360_19][143]], allocation id [uAUMBccQRji9i6oeO9yCSA], primary term [0], message [after peer recovery][shard id [[test360_19][143]], allocation id [uAUMBccQRji9i6oeO9yCSA], primary term [0], message [after peer recovery]]]: execute\r\n[2017-08-18T17:17:56,467][DEBUG][o.e.c.a.s.ShardStateAction] [node_az2_084354b0d5] [test360_19][143] starting shard [test360_19][143], node[pw31xd1SR6Gino4TrbqZpw], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=uAUMBccQRji9i6oeO9yCSA], unassigned_info[[reason=ALLOCATION_FAILED], at[2017-08-18T17:17:55.348Z], failed_attempts[1], delayed=false, details[failed recovery, failure RecoveryFailedException[[test360_19][143]: Recovery failed from {node_az2_f79f3497d0}{CTBV-A2tT5ysQ7uVX1vjPw}{4IMXO6pHQ_SNQS5mSzF6kg}{100.0.0.3}{100.0.0.3:9002}{rack_id=az2} into {node_az3_006a1fb63b6}{b-sQwH5dQQixJ9fN27l2eA}{NBnUGKW7R2CUXlI7byGm5g}{100.0.0.1}{100.0.0.1:9002}{rack_id=az3}]; nested: RemoteTransportException[[node_az2_f79f3497d0][100.0.0.3:9002][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [1] files with total size of [162b]]; nested: ReceiveTimeoutTransportException[[node_az3_006a1fb63b6][100.0.0.1:9002][internal:index/shard/recovery/prepare_translog] request_id [1106203] timed out after [900000ms]]; ], allocation_status[no_attempt]], expected_shard_size[162] (shard started task: [shard id [[test360_19][143]], allocation id [uAUMBccQRji9i6oeO9yCSA], primary term [0], message [after peer recovery]])\r\n[2017-08-18T17:17:57,044][DEBUG][o.e.c.r.a.a.BalancedShardsAllocator] [node_az2_084354b0d5] Relocate shard [[test360_19][1], node[pw31xd1SR6Gino4TrbqZpw], [R], s[STARTED], a[id=7MkVj-PkQXePR-Htukpl7g]] from node [pw31xd1SR6Gino4TrbqZpw] to node [b-sQwH5dQQixJ9fN27l2eA]\r\n[2017-08-18T17:18:01,307][INFO ][o.e.c.r.a.AllocationService] [node_az2_084354b0d5] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[test360_19][143]] ...]).\r\n...\r\n```\r\n\r\nAny insight will be appreciated.\r\nThank you!"},"comment":{"url":"https://api.github.com/repos/elastic/elasticsearch/issues/comments/323818517","html_url":"https://github.com/elastic/elasticsearch/issues/26293#issuecomment-323818517","issue_url":"https://api.github.com/repos/elastic/elasticsearch/issues/26293","id":323818517,"user":{"login":"gndcshv","id":13341666,"avatar_url":"https://avatars0.githubusercontent.com/u/13341666?v=4","gravatar_id":"","url":"https://api.github.com/users/gndcshv","html_url":"https://github.com/gndcshv","followers_url":"https://api.github.com/users/gndcshv/followers","following_url":"https://api.github.com/users/gndcshv/following{/other_user}","gists_url":"https://api.github.com/users/gndcshv/gists{/gist_id}","starred_url":"https://api.github.com/users/gndcshv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gndcshv/subscriptions","organizations_url":"https://api.github.com/users/gndcshv/orgs","repos_url":"https://api.github.com/users/gndcshv/repos","events_url":"https://api.github.com/users/gndcshv/events{/privacy}","received_events_url":"https://api.github.com/users/gndcshv/received_events","type":"User","site_admin":false},"created_at":"2017-08-21T18:32:39Z","updated_at":"2017-08-21T18:32:39Z","body":"> The include rule is fine. There should be at least a log line on node_az2_f79f3497d0 that the prepare_translog request with id 1106203 has been sent (around timestamp 2017-08-18T17:02:55, i.e. 15 mins before the error got reported). Can you confirm that? This would help identify the network stack as the issue.\r\n\r\nCan't find any mention of that request on the node specified in the RemoteTransportException - I'm guessing that's the node that was supposed to stream the data. That exception itself in not there either - the only nodes have it are the active master and the data node that had that shard stuck initializing.\r\n\r\n> Another wild guess, but can you try to switch transport.type and http.type to netty3 (requires full cluster restart) and see if that helps?\r\n\r\nUpdated `elasticsearch.yml` and restarted the cluster. Transport settings look like this now:\r\n```\r\n...\r\n\"transport\": {\r\n  \"ping_schedule\": \"10s\",\r\n  \"tcp\": {\r\n    \"port\": \"9002\"\r\n  },\r\n  \"type\": \"netty3\"\r\n},\r\n...\r\n```\r\nand HTTP:\r\n```\r\n...\r\n\"http\": {\r\n  \"type\": \"netty3\",\r\n  \"port\": \"9004\",\r\n  \"cors\": {\r\n    \"allow-origin\": \"*\",\r\n    \"enabled\": \"true\"\r\n  }\r\n}\r\n...\r\n```\r\n\r\nAnd good news - so far this seems to work, I could create 38 indices 360 * 3 shards each which brought total shard count in the cluster to 40,000+. Not a single shard got stuck and all shards were allocated and started successfully just as expected. And 40,000 shards is quite a lot and we would never have this in a real use case.\r\n\r\nThis makes sense - we never saw this issue in 2.x which is uses netty3. I should have tried it earlier :-)\r\n\r\nLet me reinstall the plugins and try this again. Will report on the finding later.\r\n\r\nThanks much!"}}
{"actor":{"display_login":"gndcshv"}}