title: Elasticsearch 6.4.2 fails to start
body: <p>I have an elasticsearch (6.4.2) instance that was running fine for months, but has suddenly stopped running and I am unable to restart it.</p>
  
  <p>I have tried restarting elasticsearch. It tries to start for 30 sec and then fails.</p>
  
  <p>Here are the logs </p>
  
  <blockquote>
    <ul>
    <li>Starting periodic command scheduler cron    ...done.  * Starting Elasticsearch Server    ...done. waiting for Elasticsearch to be up
    (1/30) waiting for Elasticsearch to be up (2/30) waiting for
    Elasticsearch to be up (3/30) waiting for Elasticsearch to be up
    (4/30) waiting for Elasticsearch to be up (5/30) waiting for
    Elasticsearch to be up (6/30) waiting for Elasticsearch to be up
    (7/30) waiting for Elasticsearch to be up (8/30) waiting for
    Elasticsearch to be up (9/30) waiting for Elasticsearch to be up
    (10/30) waiting for Elasticsearch to be up (11/30) waiting for
    Elasticsearch to be up (12/30) waiting for Elasticsearch to be up
    (13/30) waiting for Elasticsearch to be up (14/30) waiting for
    Elasticsearch to be up (15/30) waiting for Elasticsearch to be up
    (16/30) waiting for Elasticsearch to be up (17/30) waiting for
    Elasticsearch to be up (18/30) waiting for Elasticsearch to be up
    (19/30) waiting for Elasticsearch to be up (20/30) waiting for
    Elasticsearch to be up (21/30) waiting for Elasticsearch to be up
    (22/30) waiting for Elasticsearch to be up (23/30) waiting for
    Elasticsearch to be up (24/30) waiting for Elasticsearch to be up
    (25/30) waiting for Elasticsearch to be up (26/30) waiting for
    Elasticsearch to be up (27/30) waiting for Elasticsearch to be up
    (28/30) waiting for Elasticsearch to be up (29/30) waiting for
    Elasticsearch to be up (30/30) Couln't start Elasticsearch. Exiting.
    Elasticsearch log follows below. [2019-01-26T05:24:44,534][INFO
    ][o.e.n.Node               ] [] initializing ...
    [2019-01-26T05:24:44,803][INFO ][o.e.e.NodeEnvironment    ] [N8GO4t2]
    using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/xvda1)]],
    net usable_space [61.2gb], net total_space [96.8gb], types [ext4]
    [2019-01-26T05:24:44,804][INFO ][o.e.e.NodeEnvironment    ] [N8GO4t2]
    heap size [3.9gb], compressed ordinary object pointers [true]
    [2019-01-26T05:24:58,423][INFO ][o.e.n.Node               ] [N8GO4t2]
    node name derived from node ID [N8GO4t2IRwuqQE-QmYZwIw]; set
    [node.name] to override [2019-01-26T05:24:58,424][INFO ][o.e.n.Node<br>
    ] [N8GO4t2] version[6.4.2], pid[90],
    build[default/tar/04711c2/2018-09-26T13:34:09.098244Z],
    OS[Linux/4.4.0-1074-aws/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit
    Server VM/1.8.0_181/25.181-b13] [2019-01-26T05:24:58,424][INFO
    ][o.e.n.Node               ] [N8GO4t2] JVM arguments [-Xms4g, -Xmx4g,
    -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.9OJzjeIq, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/opt/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=tar] [2019-01-26T05:25:03,133][INFO ][o.e.p.PluginsService     ] [N8GO4t2] loaded module
    [aggs-matrix-stats] [2019-01-26T05:25:03,133][INFO
    ][o.e.p.PluginsService     ] [N8GO4t2] loaded module [analysis-common]
    [2019-01-26T05:25:03,134][INFO ][o.e.p.PluginsService     ] [N8GO4t2]
    loaded module [ingest-common] [2019-01-26T05:25:03,134][INFO
    ][o.e.p.PluginsService     ] [N8GO4t2] loaded module [lang-expression]
    [2019-01-26T05:25:03,134][INFO ][o.e.p.PluginsService     ] [N8GO4t2]
    loaded module [lang-mustache] [2019-01-26T05:25:03,134][INFO
    ][o.e.p.PluginsService     ] [N8GO4t2] loaded module [lang-painless]
    [2019-01-26T05:25:03,134][INFO ][o.e.p.PluginsService     ] [N8GO4t2]
    loaded module [mapper-extras] [2019-01-26T05:25:03,134][INFO
    ][o.e.p.PluginsService     ] [N8GO4t2] loaded module [parent-join]
    [2019-01-26T05:25:03,134][INFO ][o.e.p.PluginsService     ] [N8GO4t2]
    loaded module [percolator] [2019-01-26T05:25:03,134][INFO
    ][o.e.p.PluginsService     ] [N8GO4t2] loaded module [rank-eval]
    [2019-01-26T05:25:03,135][INFO ][o.e.p.PluginsService     ] [N8GO4t2]
    loaded module [reindex] [2019-01-26T05:25:03,135][INFO
    ][o.e.p.PluginsService     ] [N8GO4t2] loaded module [repository-url]
    [2019-01-26T05:25:03,135][INFO ][o.e.p.PluginsService     ] [N8GO4t2]
    loaded module [transport-netty4] [2019-01-26T05:25:03,135][INFO
    ][o.e.p.PluginsService     ] [N8GO4t2] loaded module [tribe]
    [2019-01-26T05:25:03,135][INFO ][o.e.p.PluginsService     ] [N8GO4t2]
    loaded module [x-pack-core] [2019-01-26T05:25:03,135][INFO
    ][o.e.p.PluginsService     ] [N8GO4t2] loaded module
    [x-pack-deprecation] [2019-01-26T05:25:03,135][INFO
    ][o.e.p.PluginsService     ] [N8GO4t2] loaded module [x-pack-graph]
    [2019-01-26T05:25:03,135][INFO ][o.e.p.PluginsService     ] [N8GO4t2]
    loaded module [x-pack-logstash] [2019-01-26T05:25:03,136][INFO
    ][o.e.p.PluginsService     ] [N8GO4t2] loaded module [x-pack-ml]
    [2019-01-26T05:25:03,136][INFO ][o.e.p.PluginsService     ] [N8GO4t2]
    loaded module [x-pack-monitoring] [2019-01-26T05:25:03,136][INFO
    ][o.e.p.PluginsService     ] [N8GO4t2] loaded module [x-pack-rollup]
    [2019-01-26T05:25:03,136][INFO ][o.e.p.PluginsService     ] [N8GO4t2]
    loaded module [x-pack-security] [2019-01-26T05:25:03,136][INFO
    ][o.e.p.PluginsService     ] [N8GO4t2] loaded module [x-pack-sql]
    [2019-01-26T05:25:03,136][INFO ][o.e.p.PluginsService     ] [N8GO4t2]
    loaded module [x-pack-upgrade] [2019-01-26T05:25:03,136][INFO
    ][o.e.p.PluginsService     ] [N8GO4t2] loaded module [x-pack-watcher]
    [2019-01-26T05:25:03,137][INFO ][o.e.p.PluginsService     ] [N8GO4t2]
    loaded plugin [repository-s3] [2019-01-26T05:25:13,522][INFO
    ][o.e.x.m.j.p.l.CppLogMessageHandler] [controller/112] [Main.cc@109]
    controller (64 bit): Version 6.4.2 (Build 660eefe6f2ea55) Copyright
    (c) 2018 Elasticsearch BV
    [2019-01-26T05:25:14,092][DEBUG][o.e.a.ActionModule       ] Using REST
    wrapper from plugin org.elasticsearch.xpack.security.Security
    [2019-01-26T05:25:16,584][INFO ][o.e.d.DiscoveryModule    ] [N8GO4t2]
    using discovery type [zen] [2019-01-26T05:25:17,886][INFO ][o.e.n.Node
    ] [N8GO4t2] initialized [2019-01-26T05:25:17,887][INFO ][o.e.n.Node<br>
    ] [N8GO4t2] starting ... [2019-01-26T05:25:18,132][INFO
    ][o.e.t.TransportService   ] [N8GO4t2] publish_address
    {172.17.0.2:9300}, bound_addresses {0.0.0.0:9300}</li>
    </ul>
  </blockquote>