title: Google Cloud Function - Memory Limit Exceeded (2GB) - Heavy data processing
body: <p>I have a heavy data processing script written in Python. Each time the script processes one job, about 500MB of RAM is used. (The reason is because the script looks up historical records from a very large database.) The processing script also takes about 3 minutes for each row to run.</p>
  
  <p>We have deployed our python script to a Google Cloud Function. When we invoke the function to process three jobs simultaneously, the function works fine, memory usage is about 1500-1600MB; all is dandy.</p>
  
  <p>However, when we try to invoke the function to process 10 jobs or 100 jobs simultaneously, the function is killed as memory is exceeded. We noticed in the documentation that the memory limit for a function at any one time is 2GB. Would it be safe to say that we can't increase that to 10GB or 100GB or 1000GB so we can run more instances of the script in parallel? To be honest, why is it 2GB per function, not 2GB per invocation? I would love to have access to serverless capabilities for heavy data processing work on Google; this does not seem to be available.</p>
  
  <p>If so, would you say that the best way to achieve our goal is just use a stock-standard Google VM with 1000GB of RAM?</p>
  
  <p>Thanks.</p>