title: How to use Interpolations on SharedArray in worker process without each process creating its own copy of data?
body: <p>I'm using the Julia 0.4.6 currently in 64bit Debian Stretch ("testing").  I'm currently using it in a 4 core 4Gbyte VM.</p>
  
  <p>What I'm basically trying to do is have the main process set up a 3D SharedArray, and then parallelize the production of a 2D array derived from that 3D array over worker processes.  The catch is, if I want the workers to access that volume (all of it, as I can't predict which bits they'll actually need) via an interpolation, I seem to be having problems with rampant memory usage.</p>
  
  <p>Here's some minimal code demonstrating what I'm trying to do...</p>
  
  <p>First an interpolation free version:</p>
  
  <pre><code># Initialize 1GByte of SharedArray
  volume=SharedArray(Float32,(512,512,1024))  # 1 GByte of shared data
  volume[:,:,:]=rand(Float32,(512,512,1024))
  
  # Function to compute directly from array
  function test0()
    image=SharedArray(Float32,(512,512))
    @sync @parallel for y=1:512
      for x=1:512
        image[x,y]=volume[x,y,512]
      end
    end
    sdata(image)
  end
  
  println(mean(test0()))
  </code></pre>
  
  <p>That works fine whether I run it with no workers or -p 2, -p 4 , -p 8 or -p 16 (there's not a lot of performance improvement, but bear in mind this is just a model for something which does much more compute in each iteration). </p>
  
  <p>However this version:</p>
  
  <pre><code>using Interpolations
  
  # Function to compute via interpolator
  function test1()
    sampler=interpolate(volume,BSpline(Linear()),OnGrid())
  
    image=SharedArray(Float32,(512,512))
  
    @sync @parallel for i=1:512*512
      x,y=ind2sub((512,512),i)
      image[x,y]=sampler[x,y,512]
    end
    sdata(image)
  end
  
  println(mean(test1()))
  </code></pre>
  
  <p>starts the system swapping with -p 2, and -p 4 has the workers terminating with <code>OutOfMemoryError</code>s.</p>
  
  <p>What I'm <em>guessing</em> is that the serialization of the interpolator to the worker processes loses sight of the fact it's sitting on a  SharedArray, and just copies the data wholesale into a non-shared copy in each worker.</p>
  
  <p>My question is: what can I do to improve this?  Is there a better pattern for what I'm trying to do here, or some cunning use of <code>@&lt;something&gt;</code> that'll result in each worker interpolating off the shared array instead of it's own copy?</p>
  
  <p><strong>Update:</strong></p>
  
  <p>This approach seems to usefully push the creation of an interpolation object into each worker processes:</p>
  
  <pre><code>function test(volume)
  
    image=SharedArray(Float32,(512,512))
  
    sampler=Nullable()
  
    @sync @parallel for i=1:512*512
      x,y=ind2sub((512,512),i)
      if isnull(sampler)
        warn(STDERR,"Allocating sampler...")
        sampler=Nullable(Interpolations.interpolate(sdata(volume),Interpolations.BSpline(Interpolations.Linear()),Interpolations.OnGrid()))
        warn(STDERR,"...allocated sampler")
      end
      image[x,y]=get(sampler)[x,y,512]
    end
    sdata(image)
  end
  </code></pre>
  
  <p>...however it still suffers from exactly the same problem and with more than a few worker processes will start swapping or fail to create the samplers due to memory exhaustion.</p>
  
  <p>A closer inspection of the <a href="https://github.com/tlycken/Interpolations.jl" rel="nofollow">Interpolations.jl</a> code suggests it generally expects to pad and/or pre-filter the array given (which obviously means it taking a copy)... however I can't yet read Julia well enough to understand whether <code>BSpline(Linear())</code> has any chance of bypassing those copies being made and avoiding the massive per-process memory overhead.  I see the Quadratic interpolation offers the intriguing option of prefiltering being done "InPlace" but I haven't explored this further yet (a significant issue would seem to be avoiding <em>all</em> the workers attempting to prefilter the shared data).</p>