title: Rails after_remove not getting triggered with destroy
body: <p>I have venues and venue_managers, with a HABTM relationship, and I am rolling my own counter cache via after_add and after_remove callbacks. Those callbacks on the venue_managers are getting added to the venue model via a mixin. I am using am mixin because there are other models that need this same logic.</p>
  
  <p>If there are 2 venue_managers and I re-assign venue_managers to just have 1, with venue.venue_managers = a_user, then the after_remove callback gets triggered and all is well. If I have 2 venue_managers and I destroy the last one with venue.venue_managers.last.destroy, the after_remove callback is NOT triggered and the counter cache does not update, which is my problem.</p>
  
  <pre><code>class Venue &lt; ActiveRecord::Base
    include HABTMCounterCache
  
    has_and_belongs_to_many :venue_managers, :class_name =&gt; 'User', :join_table =&gt; :venues_venue_managers
  end
  
  module HABTMCounterCache
    extend ::ActiveSupport::Concern
  
    module ClassMethods
      def count_cache_associations
        klass = self.name.downcase
        self.send("after_add_for_#{klass}_managers") &lt;&lt; lambda do |obj, manager|
          obj.update_counter_cache(manager)
        end
  
        self.send("after_remove_for_#{klass}_managers") &lt;&lt; lambda do |obj, manager|
          obj.update_counter_cache(manager)
        end
      end
    end
  
    def update_counter_cache(noop)
      klass = self.class.name.downcase
      self.assign_attributes(:"#{klass}_managers_count" =&gt; self.send("#{klass}_managers").count)
    end
  end
  </code></pre>