Consider the following example: In a HA cluster with 3 masters, a cold backup is taken  sequentially starting from master 1, then 2, then 3. Each master is powered off (using the  recommended approach in the previous sections), backed up, then powered on again. During  this time period, master 2 and 3 are still running, and the cluster is still fully operational for end  users. When restoring, the same order is used so master 1 is restored first, then 2, then 3.  The issue here is that each backup for the master nodes is taken at a completely different  time, with master 3 being the last. This means that master 3 will have different data when  restored, and when brought online last it will attempt to communicate with the existing cluster.  Therefore, it’s essential to ensure that in the event of a cluster failure, and all 3 masters need  to be restored from a backup, that the order of restoration is the  reverse  of the order of  backup. This process applies to restoring both hot and cold backups. Restoring a cluster entirely depends on the hosting virtualization platform. The examples in  this section uses VMWare to host the IBM Cloud Private installation, and therefore uses the  Virtual Machine (VM) snapshot capabilities provided with the platform. Note that a VMWare  snapshot is not a complete backup mechanism and is used here to demonstrate the practical  application of the backup methods described in this chapter. Using the previous example of a HA cluster, master nodes 1, 2, and 3 were backed up using  VMWare snapshots while the machines were offline (cold snapshot) in sequence starting  from master 1. Nginx workloads were deployed to simulate user deployments so that the  cluster state in etcd was changed. To simulate a disaster, masters 1 and 2 were destroyed. At  this point etcd is now running as a single node cluster, in read-only mode and cannot accept  any write requests from Kubernetes.  https://coreos.com/etcd/docs/latest/op-guide/failures.html ) provides a good  explanation about different etcd failure scenarios and what it can or can’t tolerate. In this  situation, quorum is lost and as you cannot currently replace IBM Cloud Private master nodes  using different IP addresses, you cannot recover. Therefore, the snapshots must be used to  deploy masters 1 and 2, and restore all 3 masters to a previously working state. Restore all 3 masters to a previous snapshot, power on the nodes and allow some time for all  the containers to start. To ensure the etcd cluster has started successfully and the cluster is  healthy, etcd provides a useful command line utility to query the cluster status, which can be  downloaded to your local machine or ran from the etcd container itself. To get the cluster  health status, complete the following steps from the IBM Cloud Private boot node (or  whatever system has cluster access with  kubectl ): The CoreOS documentation (found at  Verifying a restored backup 1. Run  kubectl -n kube-system get pods | grep k8s-etcd  to retrieve the etc pod name  (the format is  k8s-etcd-<node-ip> ): 2. Use any one of the pod names returned to execute an  etcdctl cluster-health command, using one of the master node IP addresses in the  endpoint  parameter: [root@icp-ha-boot ~]# kubectl -n kube-system get pods -o name| grep k8s-etcd k8s-etcd-172.24.19.201 k8s-etcd-172.24.19.202 k8s-etcd-172.24.19.203 [root@icp-ha-boot ~]# kubectl -n kube-system exec k8s-etcd-172.24.19.201 -- sh  -c "export ETCDCTL_API=2; etcdctl --cert-file /etc/cfc/conf/etcd/client.pem  --key-file /etc/cfc/conf/etcd/client-key.pem --ca-file  /etc/cfc/conf/etcd/ca.pem --endpoints https://172.24.19.201:4001  cluster-health" 80  IBM Cloud Private System Administrator’s Guide