title: dask read_parquet with pyarrow memory blow up
body: <p>I am using dask to write and read parquet. I am writing using fastparquet engine and reading using pyarrow engine.
  My worker has 1 gb of memory. With fastparquet the memory usage is fine, but when i switch to pyarrow, it just blows up and causes the worker to restart.
  I have a reproducible example below which fails with pyarrow on a worker of 1gb memory limit.
  In reality my dataset is much more bigger than this. The only reason of using pyarrow is it gives me speed boost while scanning compared to fastparquet(somewhere around 7x-8x)</p>
  
  <p>dask : 0.17.1 </p>
  
  <p>pyarrow : 0.9.0.post1</p>
  
  <p>fastparquet : 0.1.3</p>
  
  <pre><code>import dask.dataframe as dd
  import numpy as np
  import pandas as pd
  
  size = 9900000
  tmpdir = '/tmp/test/outputParquet1'
  
  d = {'a': np.random.normal(0, 0.3, size=size).cumsum() + 50,
      'b': np.random.choice(['A', 'B', 'C'], size=size),
      'c': np.random.choice(['D', 'E', 'F'], size=size),
      'd': np.random.normal(0, 0.4, size=size).cumsum() + 50,
      'e': np.random.normal(0, 0.5, size=size).cumsum() + 50,
      'f': np.random.normal(0, 0.6, size=size).cumsum() + 50,
      'g': np.random.normal(0, 0.7, size=size).cumsum() + 50}
  df = dd.from_pandas(pd.DataFrame(d), 200)
  df.to_parquet(tmpdir, compression='snappy', write_index=True, 
           engine='fastparquet')
  
  #engine = 'pyarrow' #fails due to worker restart
  engine = 'fastparquet' #works fine
  df_partitioned = dd.read_parquet(tmpdir + "/*.parquet", engine=engine)
  print(df_partitioned.count().compute())
  df_partitioned.query("b=='A'").count().compute()
  </code></pre>
  
  <p>Edit: My original setup has spark jobs running that writes data parallely into partitions using fastparquet. So the metadata file is created in the innermost partition rather than the parent directory.Hence using glob paths instead of parent directory(fastparquet is much faster with parent directory read whereas pyarrow wins when scanning with glob path)</p>