title: How spark jobs are created?
body: <p>Could someone please let me know how spark jobs are being created. I have a framework which ingests the data to Hive table. However, when I am checking spark UI it's showing that 50,000 job has been created with each stage not processing more than few bytes of data.</p>
  <p>Would this create performance issue ?
  How to find out why these many jobs are being created?
  is it possible to restrict jobs?</p>
  <p><a href="https://i.stack.imgur.com/ph5Ib.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ph5Ib.png" alt="enter image description here" /></a></p>
  <p><strong>Code</strong>
  Showing code is very difficult for me. It's more than 2000 lines of spark framework. And making MVP is also difficult. However, this is the process.</p>
  <p><strong>I am doing following process for 10k tables.</strong></p>
  <ol>
  <li>Read a file (not using spark)</li>
  <li>Parse a file (not using spark)</li>
  <li>create dataframe from read data.</li>
  <li>Figure out deletes and write it to s3</li>
  <li>Figure out inserts and write it to s3.</li>
  </ol>