title: Spark-submit: undefined function parse_url
body: <p>The function - <strong><em>parse_url</em></strong> always works fine if we working with spark-sql throw sql-client (via thrift server), IPython, pyspark-shell, but it doesn't work throw <strong>spark-submit</strong> mode:</p>
  
  <pre><code>/opt/spark/bin/spark-submit --driver-memory 4G --executor-memory 8G main.py
  </code></pre>
  
  <p>The error is:</p>
  
  <pre><code>Traceback (most recent call last):
    File "/home/spark/***/main.py", line 167, in &lt;module&gt;
      )v on registrations.ga = v.ga and reg_path = oldtrack_page and registration_day = day_cl_log  and date_cl_log &lt;= registration_date""")
    File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/context.py", line 552, in sql
    File "/opt/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
    File "/opt/spark/python/lib/pyspark.zip/pyspark/sql/utils.py", line 40, in deco
  pyspark.sql.utils.AnalysisException: undefined function parse_url;
  Build step 'Execute shell' marked build as failure
  Finished: FAILURE
  </code></pre>
  
  <p>So, we are using workaround here:</p>
  
  <pre><code>def python_parse_url(url, que, key):
      import urlparse
      ians = None
      if que == "QUERY":
          ians = urlparse.parse_qs(urlparse.urlparse(url).query)[key][0]
      elif que == "HOST":
          ians = urlparse.urlparse(url).hostname
      elif que == "PATH":
          ians = urlparse.urlparse(url).path
      return ians
  
  def dc_python_parse_url(url, que, key):
      ians = None
      try:
          ians = python_parse_url(url, que, key)
      except:
          pass
      return ians
  
  sqlCtx.registerFunction('my_parse_url', dc_python_parse_url)
  </code></pre>
  
  <p>Please, any help with this issue?</p>