title: AlwaysOn - cluster lease timeouts and PREEMPTIVE_HADR_LEASE_MECHANISM
body: <p>We have recently installed some WSUS updates + SQL 2012 SP3 (yes, all tested without a problem in UAT :) and since than it seems that AO and cluster is having few issues - it seems that cluster's lease is timing out and I am unable to figure out why.. ;/ this results in short blip and lost connectivity.</p>
  
  <p>Any help would be appreciated!</p>
  
  <p>AlwaysOn Extended Events:</p>
  
  <pre><code>availability_group_lease_expired; state: LeaseEpxired; Timestamp: 2016-06-12 04:58:40.34
  availability_replica_state_change: current state: Resolving_Normal; previous_sate: Primary_Normal;Timestamp: 2016-06-12 04:58:40.34
  ..
  availability_replica_state_change: current state: Primary_Normal; previous_sate: Primary_Pending;Timestamp: 2016-06-12 04:58:52.96
  </code></pre>
  
  <p>SQL Log: </p>
  
  <pre><code>Date: 12/06/2016 04:58:40; Error: 19421, Severity: 16, State: 1.
  SQL Server hosting availability group did not receive a process event signal from the Windows Server Failover Cluster within the lease timeout period.
  
  Date: 12/06/2016 04:58:40; Error: 19407, Severity: 16, State: 1.
  The lease between availability group and the Windows Server Failover Cluster has expired. A connectivity issue occurred between the instance of SQL Server and the Windows Server Failover Cluster. To determine whether the availability group is failing over correctly, check the corresponding availability group resource in the Windows Server Failover Cluster.
  
  Date: 12/06/2016 04:58:40
  AlwaysOn: The local replica of availability group is going offline because either the lease expired or lease renewal failed. This is an informational message only. No user action is required.
  </code></pre>
  
  <p>Cluster log (do not ask my why it's -1h, date on all nodes is ok):</p>
  
  <pre><code>2016/06/12-03:58:40.587 INFO  [RCM] rcm::RcmApi::FailResource: (AlwaysOn)
  2016/06/12-03:58:40.588 INFO  [RCM] HandleMonitorReply: FAILURENOTIFICATION for 'AlwaysOn', gen(3) result 0/0.
  2016/06/12-03:58:40.588 INFO  [RCM] Res AlwaysOn: Online -&gt; ProcessingFailure( StateUnknown )
  2016/06/12-03:58:40.588 INFO  [RCM] TransitionToState(AlwaysOn) Online--&gt;ProcessingFailure.
  2016/06/12-03:58:40.588 INFO  [RCM] rcm::RcmGroup::UpdateStateIfChanged: (AlwaysOn, Online --&gt; Pending)
  2016/06/12-03:58:40.588 ERR   [RCM] rcm::RcmResource::HandleFailure: (AlwaysOn)
  2016/06/12-03:58:40.588 INFO  [RCM] resource AlwaysOn: failure count: 1, restartAction: 2 persistentState: 1.
  2016/06/12-03:58:40.588 INFO  [RCM] numDependents is zero, auto-returning true
  2016/06/12-03:58:40.588 INFO  [RCM] Greater than restartPeriod time has elapsed since first failure of AlwaysOn, resetting failureTime and failureCount.
  2016/06/12-03:58:40.588 INFO  [RCM] Will queue immediate restart (500 milliseconds) of AlwaysOn after terminate is complete.
  2016/06/12-03:58:40.588 INFO  [RCM] Res AlwaysOn: ProcessingFailure -&gt; WaitingToTerminate( DelayRestartingResource )
  2016/06/12-03:58:40.588 INFO  [RCM] TransitionToState(AlwaysOn) ProcessingFailure--&gt;[WaitingToTerminate to DelayRestartingResource].
  2016/06/12-03:58:40.588 INFO  [RCM] Res AlwaysOn: [WaitingToTerminate to DelayRestartingResource] -&gt; Terminating( DelayRestartingResource )
  2016/06/12-03:58:40.588 INFO  [RCM] TransitionToState(AlwaysOn) [WaitingToTerminate to DelayRestartingResource]--&gt;[Terminating to DelayRestartingResource].
  2016/06/12-03:58:40.588 ERR   [RES] SQL Server Availability Group &lt;AlwaysOn&gt;: [hadrag] Lease Thread terminated
  2016/06/12-03:58:40.588 ERR   [RES] SQL Server Availability Group &lt;AlwaysOn&gt;: [hadrag] The lease is expired. The lease should have been renewed by 2016/06/12-03:58:30.348
  2016/06/12-03:58:40.588 INFO  [RES] SQL Server Availability Group: [hadrag] Stopping Health Worker Thread
  2016/06/12-03:58:40.588 INFO  [RES] SQL Server Availability Group: [hadrag] Health worker was asked to terminate
  </code></pre>
  
  <p>Something odd - SQL wait times from last 12h:</p>
  
  <pre><code>wait type                        Wait Time      % of Total Wait
  PREEMPTIVE_HADR_LEASE_MECHANISM  80,183,360 ms  39.09%
  PREEMPTIVE_SP_SERVER_DIAGNOSTICS 80,183,265 ms  39.09%
  HADR_CLUSAPI_CALL                40,534,655 ms  19.76%
  </code></pre>
  
  <p>Dodgy update somewhere? Let me know if you have any hints.</p>
  
  <p>Thanks in advance,
  Tomasz</p>