title: Pass %%local variables from dotenv run in jupyter to Azure HDInsight pyspark cluster
body: <h1>Intro</h1>
  <p><a href="https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-spark-jupyter-notebook-install-locally" rel="nofollow noreferrer">This link</a> details how to install jupyter locally and work against an Azure HDInsight cluster. This works well getting things setup.</p>
  <p>However:</p>
  <ol>
  <li><p>Not all python packages that we have available locally are available on the cluster.</p>
  </li>
  <li><p>Some local processing may want to be done before 'submitting' a cell to the cluster.</p>
  </li>
  </ol>
  <p>I'm aware that python packages that are not installed can be installed via <a href="https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-spark-python-package-installation" rel="nofollow noreferrer">script actions</a> and <a href="https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-spark-jupyter-notebook-use-external-packages" rel="nofollow noreferrer"><code>%%configure</code></a>, however given our use of <code>dotenv</code> locally these don't seem to be viable solutions.</p>
  <h1>Problem</h1>
  <ol>
  <li>Source control with git</li>
  <li>Git repos are local on dev machines We store
  configuration/sensitive environment variables in <code>.env</code> files
  locally (they are not checked into git)</li>
  <li><code>dotenv</code> package is used to
  read sensitive variables and set locally for execution</li>
  <li>blob storage
  account names and keys are example of these variables</li>
  <li><strong>how to pass these locally set variables to a pyspark cell?</strong></li>
  </ol>
  <hr />
  <h1>Local cell example</h1>
  <p><a href="https://i.stack.imgur.com/4MpRa.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/4MpRa.png" alt="enter image description here" /></a></p>
  <h1>Followed by pyspark cell</h1>
  <p><a href="https://i.stack.imgur.com/aT0WY.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/aT0WY.png" alt="enter image description here" /></a></p>