title: Which would be a quicker (and better) tool for querying data stored in the Parquet format - Spark SQL, Athena or ElasticSearch?
body: <p>Am currently building an ETL pipeline, which outputs tables of data (order of ~100+ GBs) to a downstream interactive dashboard, which allows filtering the data dynamically (based on pre-defined &amp; indexed filters).</p>
  
  <p>Have zeroed in on using <strong>PySpark / Spark for the initial ETL phase</strong>.
  Next, this processed data will be <strong>summarised</strong> (<em>simple counts, averages, etc.</em>) &amp; <strong>then visualised</strong> in the interactive dashboard.</p>
  
  <p>Towards the interactive querying part, I was wondering which tool might work best with my structured &amp; transactional data (stored in Parquet format) -</p>
  
  <ol>
  <li>Spark SQL (in memory dynamic querying)</li>
  <li>AWS Athena (Serverless SQL querying, based on Presto)</li>
  <li>Elastic Search (search engine)</li>
  <li>Redis (Key Value DB)</li>
  </ol>
  
  <p>Feel free to suggest alternative tools, if you know of a better option. </p>