title: Suggestions on breaking down an IOT application consisting of jobs and services hosted in kubernetes in a way which would enable horizontal scaling
body: <p>I have an IOT application which is architected in the following way :-</p>
  
  <p>There are plants which has its own sets of devices.</p>
  
  <p>Now the entire pipeline is deployed in kubernetes consisting of the following units :-</p>
  
  <ol>
  <li>A job which wakes up every x seconds, reads data from all plants, pushes the data to a mqtt broker.</li>
  <li>An MQTT broker.</li>
  <li>A subscriber service which receives data from all plants and pushes it to a timeseries database.</li>
  <li>Jobs running at intervals of 5min, 15min, 1hr, 4hr, 1day and performs downsampling on the data of all the projects and pushes it to separate downsampled tables.</li>
  <li>Jobs running every day to check if there was in holes/gaps in the data and tries to fill it up if possible.</li>
  </ol>
  
  <p>Now this works fine for few Plants, but when the number of plants increases it becomes difficult perform data retrieval, push, downsampling using single service/job as it becomes too much memory intensive and chokes at multiple places. As a temporary fix scaling it vertically fixes the issue to some extent but in that case I need to put all the pods in a single machine which I am scaling vertically and scaling multiple nodes vertically is quite expensive.
  Hence I am planning to break down the system in a way so that I can scale horizontally and I am looking for suggestions for the possible ways I can achieve this.</p>