title: Unable to load libhdfs when using pyarrow
body: <p>I'm trying to connect to HDFS through Pyarrow, but it does not work because <code>libhdfs</code> library cannot be loaded.</p>
  
  <p><code>libhdfs.so</code> is in <code>$HADOOP_HOME/lib/native</code> as well as in <code>$ARROW_LIBHDFS_DIR</code>.</p>
  
  <pre><code>print(os.environ['ARROW_LIBHDFS_DIR'])
  fs = hdfs.connect()
  
  
  bash-3.2$ ls $ARROW_LIBHDFS_DIR
  examples        libhadoop.so.1.0.0  libhdfs.a       libnativetask.a
  libhadoop.a     libhadooppipes.a    libhdfs.so      libnativetask.so
  libhadoop.so        libhadooputils.a    libhdfs.so.0.0.0    libnativetask.so.1.0.0
  </code></pre>
  
  <p>The error I'm getting:</p>
  
  <pre><code>Traceback (most recent call last):
    File "wine-pred-ml.py", line 31, in &lt;module&gt;
      fs = hdfs.connect()
    File "/Users/PVZP/Library/Python/2.7/lib/python/site-packages/pyarrow/hdfs.py", line 183, in connect
      extra_conf=extra_conf)
    File "/Users/PVZP/Library/Python/2.7/lib/python/site-packages/pyarrow/hdfs.py", line 37, in __init__
      self._connect(host, port, user, kerb_ticket, driver, extra_conf)
    File "pyarrow/io-hdfs.pxi", line 89, in pyarrow.lib.HadoopFileSystem._connect
    File "pyarrow/error.pxi", line 83, in pyarrow.lib.check_status
  pyarrow.lib.ArrowIOError: Unable to load libhdfs
  </code></pre>