title: ambari on HDP cluster + ambari-metrics-collector service not start
body: <p>we have some issue with <code>ambari-metrics-collector</code> service , ( we have <code>HDP</code> cluster version - <code>2.6.4</code> with 8 nodes )</p>
  <p>ambari metrics collector service can’t start or start of few second then failed</p>
  <p><a href="https://i.stack.imgur.com/2KXvN.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/2KXvN.png" alt="enter image description here" /></a></p>
  <p>the details about metrics collector version</p>
  <pre><code>rpm -qa | grep metrics
  ambari-metrics-grafana-2.6.1.0-143.x86_64
  ambari-metrics-monitor-2.6.1.0-143.x86_64
  ambari-metrics-collector-2.5.0.3-7.x86_64
  ambari-metrics-hadoop-sink-2.6.1.0-143.x86_64
  </code></pre>
  <p>all machines are <code>rhel 7.2</code></p>
  <p>we performed the following steps in order to resolve the problem</p>
  <p>1.restart metrics-collector service</p>
  <pre><code>su - ams -c '/usr/sbin/ambari-metrics-collector --config /etc/ambari-metrics-collector/conf/ stop'
  su - ams -c '/usr/sbin/ambari-metrics-collector --config /etc/ambari-metrics-collector/conf/ start'
  
  or
  
  ambari-metrics-collector stop 
  ambari-metrics-collector start
  </code></pre>
  <p>2.restart ambari-metrics-monitor on all nodes</p>
  <pre><code> ambari-metrics-monitor stop
   ambari-metrics-monitor start
  </code></pre>
  <p>3.clean the folder <code>/var/lib/ambari-metrics-collector/hbase-tmp/zookeeper/</code></p>
  <pre><code>mv /var/lib/ambari-metrics-collector/hbase-tmp/zookeeper/zookeeper_0 /tmp/bck/zookeeper/
  </code></pre>
  <p>Then restart  metrics-collector service</p>
  <p>4.Tuning the metrics-collector  parameters according - <a href="https://docs.cloudera.com/HDPDocuments/Ambari-2.2.1.0/bk_ambari_reference_guide/content/_ams_general_guidelines.html" rel="nofollow noreferrer">https://docs.cloudera.com/HDPDocuments/Ambari-2.2.1.0/bk_ambari_reference_guide/content/_ams_general_guidelines.html</a></p>
  <p>we update the follwing parameters in ambari</p>
  <pre><code>metrics_collector_heap_size=1024
  hbase_regionserver_heapsize=1024
  hbase_master_heapsize=512
  hbase_master_xmn_size=128
  </code></pre>
  <p><strong>status for now:</strong> - steps 1-4 doesn’t help</p>
  <p><em>From the logs we can see the following:</em></p>
  <p>log file - <code>ambari-metrics-collector.log</code></p>
  <pre><code>2020-06-25 09:06:14,474 WARN org.apache.zookeeper.ClientCnxn: Session 0x172eab71f310002 for server null, unexpected error, closing socket connection and attempting reconnect
  java.net.ConnectException: Connection refused
          at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
          at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
          at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
          at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
  2020-06-25 09:06:14,575 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper, quorum=master02.sys671.com:61181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /ams-hbase-unsecure/meta-region-server
  </code></pre>
  <p>log file - <code>hbase-ams-master-master02.sys671.com.log</code></p>
  <pre><code>2020-06-25 09:38:18,799 WARN  [RS:0;master02:51842-SendThread(master02.sys671.com:61181)] zookeeper.ClientCnxn: Session 0x172ead5d73a0004 for server null, unexpected error, closing socket connection and attempting reconnect
  java.net.ConnectException: Connection refused
          at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
          at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
          at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
          at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1125)
  2020-06-25 09:38:20,437 INFO  [main-SendThread(master02.sys671.com:61181)] zookeeper.ClientCnxn: Opening socket connection to server master02.sys671.com/23.2.35.171:61181. Will not attempt to authenticate using SASL (unknown error)
  2020-06-25 09:38:20,438 WARN  [main-SendThread(master02.sys671.com:61181)] zookeeper.ClientCnxn: Session 0x172ead5d73a0002 for server null, unexpected error, closing socket connection and attempting reconnect
  java.net.ConnectException: Connection refused
  </code></pre>
  <p>we also not see that port is listening ( timeline.metrics.service.webapp.address )</p>
  <pre><code>netstat -tulpn  | grep  6188
  </code></pre>
  <p>any advice how to continue from this point ?</p>
  <p>we'll appreciate to get any help about this problem</p>