title: Rasterizing and mosaicking a large (~15GB) CSV in parallel
body: <p>I'm trying to rasterize a large CSV file which contains X and Y coordinates of raster pixels and an ID representing different individuals that have moved through the pixel.</p>
  
  <p>I want 2 rasters from this operation: In the first, each pixel of the raster contains the number of times individuals have crossed it and in the second, each pixel contains the number of unique individuals that have crossed it. </p>
  
  <p>I currently create rasters for each ID and then mosaic the rasters. I've parallelized this operation, using 16 cores and a CPU with 200 GB RAM. However, the CSV contains over a million rows and it seems a large overhead to copy it into each core. It also takes upwards of 4 days to complete and I have multiple such CSV files.</p>
  
  <p>Is there a better, faster way to do this?</p>
  
  <p>This is what I do currently:</p>
  
  <pre><code>n.cores &lt;- 16 #this is half of the available cores
  
  #Register CoreCluster
  cl       &lt;- makeCluster(n.cores)
  registerDoSNOW(cl)
  
  clusterExport(cl=cl, "fil2" ) #fil2 is the CSV
  
  
  #uq_id is a vector of all unique IDs
   rasterList= foreach(i=1:length(uq_id), .combine='comb', 
              .init=list()) %dopar% {
  
    library(raster)
  
   #create a raster for each unique ID 
    dfXYID &lt;- subset(fil2, uqid %in% i)
    dfXYID$uqid &lt;- as.numeric(dfXYID$uqid)
    dfXYID=as.data.frame(dfXYID)
  
    x &lt;- raster(xmn=min(dfXYID$V1), xmx=max(dfXYID$V1), ymn=min(dfXYID$V2), ymx=max(dfXYID$V2), res=220, crs="+proj=utm +zone=43 +datum=WGS84 +units=m +no_defs")
  
  #create a raster of the number of times individual with ID i have moved through a pixel 
  lenList &lt;- rasterize(dfXYID[, c('V1', 'V2')], x, dfXYID[, 'uqid'], fun=function(x,...){length((na.omit(x)))})
  
  
    return(lenList)
  
  }
  </code></pre>
  
  <p>rasterList is then a list of rasters, each representing movement of individual i. I then have to do another time-consuming operation (can I do this in parallel within the above foreach?): apply a mosaic operation on this large list to get the 2 raster ouputs that I need. </p>
  
  <p>I would be very grateful if anyone has any suggestions on how I can speed this up.</p>
  
  <p>I understand gdal is an option, but I am constrained as I have to work with a CSV and I don't want to be writing each raster in <code>rasterList</code> to the disk to do further processing with gdal.</p>