title: Check memory status of every node in alerts for prometheus
body: <p>I want to write a alert query for prometheus.
  Which will raise alert if any node is consuming memory more than specific % of memory.</p>
  
  <p>I want to dynamically take node and check memory status of it.</p>
  
  <ol>
  <li><p>read all nodes </p></li>
  <li><p>loop over each node with query</p>
  
  <p>sum(container_memory_usage_bytes{job="kubernetes-nodes", id="/", instance="$node"}) / sum(machine_memory_bytes{job="kubernetes-nodes", instance="$node"}) * 100</p></li>
  </ol>
  
  <p>this alert works if i provide static name of node</p>
  
  <pre><code> ALERT HighMemoryAlert
    IF 100 * sum(container_memory_usage_bytes{job="kubernetes-nodes", id="/",instance="ip-170-10-10-17.eu-west-1.compute.internal"}) / sum(machine_memory_bytes{job="kubernetes-nodes",instance="ip-170-10-10-17.eu-west-1.compute.internal"}) &gt; 90
    FOR 15m
    LABELS {
      service = "k8s",
      severity = "warning"
    }
    ANNOTATIONS {
      summary = "HighMemoryAlert"
    }
  </code></pre>
  
  <p>I dont want to add ip statically as k8s node ip can b changed over the period of time if any node crashes and created with different ip</p>