title: Incensistent partition lag values when using high level consumer for reading Kafka logs
body: <p>All of our 30 topics are created with 10 partitions in our kafka.  We are monitoring the lag by partition for all the topics/group-ids.</p>
  
  <p>We are using Fluentd plugin to read and route logs from kafka.  The plugin is implemented using a high level consumer. We have configured some consumers for individual topics and some for multiple topics for the plugin.  Overall, the data is flowing through with no problem except for 3 of the topics.</p>
  
  <p>The problem is that for 3 out of the 30 topics being processed we see that the partition lag values are inconsistent ie. looking at lag values for a specific topic/group-id, the lag for some partitions are much higher than other partitions, sometimes by as much as 30k.  However, for the other 27 topics the lags numbers for all partitions stay uniform, all partitions of one topic/group-id stay within close range of each other (for ex. all between 12 and 18).  </p>
  
  <p>Almost every time we restart Fluentd agent (which restarts the high level consumers) we see that the lag starts to smooth out for those 3 topics and sometimes they stay consistent for a little while and then again lag numbers start to become zigzagy.  This is only happening for the 3 topics.  But when we check the distribution for those 3 topics, everything looks normal.</p>
  
  <p>We are at a loss as the reason for this.  High level consumers do not code for managing the retrieval of the data from the partitions.  Its the kafka lib that handles that part. All that consumer code specifies is the number of threads.  We have tried 10, 5 and in all cases (especially 10 and 5 threads) the lag inconsistency keeps showing up for these 3 topics. The data volume is less than 30k per hour for each of these topics.</p>
  
  <p>Any suggestion as to what could be the reason?  What can be done about it?</p>
  
  <p>Really appreciate your help in advance.</p>