title: BigQuery replaced most of my Spark jobs, am I missing something?
body: <p>I've been developing Spark jobs for some years using on-premise clusters and our team recently moved to the Google Cloud Platform allowing us to leverage the power of BigQuery and such.</p>
  
  <p>The thing is, I now often find myself writing processing steps in SQL more than in PySpark since it is : </p>
  
  <ul>
  <li>easier to reason about (less verbose)</li>
  <li>easier to maintain (SQL vs scala/python code)</li>
  <li>you can run it easily on the GUI if needed</li>
  <li>fast without having to really reason about partitioning, caching and so on...</li>
  </ul>
  
  <p>In the end, I only use Spark when I've got something to do that I can't express using SQL. </p>
  
  <p>To be clear, my workflow is often like : </p>
  
  <ul>
  <li>preprocessing (previously in Spark, now in SQL)</li>
  <li>feature engineering (previously in Spark, now <strong>mainly</strong> in SQL)</li>
  <li>machine learning model and predictions (Spark ML)</li>
  </ul>
  
  <p>Am I missing something ?
  Is there any con in using BigQuery this way instead of Spark ?</p>
  
  <p>Thanks</p>