title: Can't install pyarrow on AWS Glue python shell
body: <p>I want to import <code>pyarrow</code> in a Python shell Glue script because I need to export a dataframe as parquet (i.e. with <code>DataFrame.to_parquet()</code>).</p>
  
  <p>The way to add custom dependencies suggested in the AWS docs is to use <code>.egg</code> or <code>.whl</code> files (<a href="https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html#create-python-extra-library" rel="nofollow noreferrer">https://docs.aws.amazon.com/glue/latest/dg/add-job-python.html#create-python-extra-library</a>).</p>
  
  <p>The library <code>pyarrow</code> has <code>numpy</code> and <code>six</code> as dependencies:</p>
  
  <ul>
  <li><p><code>numpy</code> is already pre-installed on Glue, with version <code>1.16.2</code> as I checked with a simple <code>print(numpy.version.version)</code></p></li>
  <li><p><code>six</code> is not pre-installed so I downloaded <code>six-1.14.0-py2.py3-none-any.whl</code> from Pypi and uploaded it to S3.</p></li>
  <li><p><code>pyarrow</code> is not pre-installed so I downloaded from Pypi the wheel file <code>pyarrow-0.16.0-cp36-cp36m-manylinux2014_x86_64.whl</code> and uploaded it to S3.</p></li>
  </ul>
  
  <p>The script itself is this:</p>
  
  <pre><code>import pandas as pd
  import six
  import numpy
  from pyarrow import *
  
  data = [['Alex',10],['Bob',12],['Clarke',13]]
  df = pd.DataFrame(data,columns=['Name','Age'],dtype=float)
  df.to_parquet('test.parquet')
  </code></pre>
  
  <p>When I run the script adding as libraries the wheel files of <code>six</code> and <code>pyarrow</code>, I get the following message:</p>
  
  <pre><code>Processing ./glue-python-libs-f8nyy9el/six-1.14.0-py2.py3-none-any.whl
  Installing collected packages: six
  Successfully installed six-1.14.0
  Processing ./glue-python-libs-f8nyy9el/pyarrow-0.16.0-cp36-cp36m-manylinux2014_x86_64.whl
  </code></pre>
  
  <p>and the following error:</p>
  
  <pre><code>WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(&lt;pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f7d78c96e10&gt;, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/numpy/
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(&lt;pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f7d78c96c88&gt;, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/numpy/
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(&lt;pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f7d78c96dd8&gt;, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/numpy/
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(&lt;pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f7d78c969b0&gt;, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/numpy/
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(&lt;pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7f7d78c96898&gt;, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/numpy/
  ERROR: Could not find a version that satisfies the requirement numpy&gt;=1.14 (from pyarrow==0.16.0) (from versions: none)
  ERROR: No matching distribution found for numpy&gt;=1.14 (from pyarrow==0.16.0)
  Traceback (most recent call last):
    File "/tmp/runscript.py", line 112, in &lt;module&gt;
      download_and_install(args.extra_py_files)
    File "/tmp/runscript.py", line 62, in download_and_install
      subprocess.check_call([sys.executable, "-m", "pip", "install", "--target=
  {}
  ".format(install_path), local_file_path])
    File "/usr/local/lib/python3.6/subprocess.py", line 311, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-m', 'pip', 'install', '--target=/glue/lib/installation', '/tmp/glue-python-libs-f8nyy9el/pyarrow-0.16.0-cp36-cp36m-manylinux2014_x86_64.whl']' returned non-zero exit status 1.
  </code></pre>
  
  <p>So at first, it seems that six is installed correctly, but then it looks like the job does not realize that <code>numpy</code> is already present with a compatible version.</p>
  
  <p>Then I tried to upload to S3 also the wheel file <code>s3://risultati-navigazione-wt-ga/libs/numpy-1.18.2-cp36-cp36m-manylinux1_x86_64.whl</code> that I downloaded from Pypi. In this case I get the message:</p>
  
  <pre><code>Processing ./glue-python-libs-xzfdvgzd/numpy-1.18.2-cp36-cp36m-manylinux1_x86_64.whl
  Installing collected packages: numpy
  Successfully installed numpy-1.18.2
  Processing ./glue-python-libs-xzfdvgzd/six-1.14.0-py2.py3-none-any.whl
  Installing collected packages: six
  Successfully installed six-1.14.0
  Processing ./glue-python-libs-xzfdvgzd/pyarrow-0.16.0-cp36-cp36m-manylinux2014_x86_64.whl
  </code></pre>
  
  <p>and the error:</p>
  
  <pre><code>WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(&lt;pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fca02861cc0&gt;, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/six/
  WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(&lt;pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fca02861cc0&gt;, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/six/
  WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(&lt;pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fca02861cc0&gt;, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/six/
  WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(&lt;pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fca02861cc0&gt;, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/six/
  WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(&lt;pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x7fca02861cc0&gt;, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/six/
  ERROR: Could not find a version that satisfies the requirement six&gt;=1.0.0 (from pyarrow==0.16.0) (from versions: none)
  ERROR: No matching distribution found for six&gt;=1.0.0 (from pyarrow==0.16.0)
  Traceback (most recent call last):
    File "/tmp/runscript.py", line 112, in &lt;module&gt;
      download_and_install(args.extra_py_files)
    File "/tmp/runscript.py", line 62, in download_and_install
      subprocess.check_call([sys.executable, "-m", "pip", "install", "--target=
  {}
  ".format(install_path), local_file_path])
    File "/usr/local/lib/python3.6/subprocess.py", line 311, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-m', 'pip', 'install', '--target=/glue/lib/installation', '/tmp/glue-python-libs-xzfdvgzd/pyarrow-0.16.0-cp36-cp36m-manylinux2014_x86_64.whl']' returned non-zero exit status 1.
  </code></pre>
  
  <p>so, this time, <code>numpy</code> is recognized during the installation of pyarrow but, as far I understand, although<code>six</code> is installed correctly, for some reason pyarrow can't find it during the installation and indeed it tries to download from the Internet (it gets stuck a few minutes during that operation).</p>
  
  <p>Can anybody help me? Thanks!</p>