title: Independent python subprocess from AWS Lambda function
body: <p>I have successfully created a Lambda function (app1) that reads and writes to RDS.</p>
  
  <p>My Lambda function is written in python2.7 and uploaded as a zipped package.</p>
  
  <p>I created and tested the zipped package on an EC2 instance in the same VPC as my RDS and Lambda function.</p>
  
  <p>Next, I added the functionality to my Lambda function to popen an independent subprocess (app2) using subprocess.popen and had app1 return while the app2 subprocess continued on its own.  I tested that app1 would successfully return its handler's output while app2 continued by putting a 60 second sleep in app2 and tailed the output file of app2.</p>
  
  <p>I successfully tested app1 and app2 functionality in the EC2 instance.</p>
  
  <p>After uploading the new package, my app1 appears to behave exactly as expected, and returns its handler's output immediately, but the app2 functionality doesn't "appear" to instantiated, but there is no logs, errors, or output to capture from app2.</p>
  
  <p>In app1, I tested that subprocess worked by performing a subprocess.check_output(['ls','-la']) prior to and after the independent subproccess.popen, and the local folder is displayed with my files.  Except there isn't a app2output file created as expected.</p>
  
  <p><strong>Two questions</strong></p>
  
  <ol>
  <li>Is there something special that I am missing in AWS-Lambda concepts
  that is causing app2 to "fail"?  By "fail" I mean not writing creating the new file and writing to it, nor creating any logs in Cloudwatch the same way app1 successfully does, nor printing out to the Lambda console like app1 does.</li>
  <li>How do I catch any output (logging info and errors) for app2 in an AWS-Lambda environment?</li>
  </ol>
  
  <p><strong>app1.py</strong></p>
  
  <pre><code>import subprocess
  import sys
  import logging
  import rds_config
  import pymysql
  #rds settings
  rds_host  = "rdshost"
  name = rds_config.db_username
  password = rds_config.db_password
  db_name = rds_config.db_name
  port = 3306
  
  logger = logging.getLogger()
  logger.setLevel(logging.INFO)
  
  server_address = (rds_host, port)
  try:
      conn = pymysql.connect(rds_host, user=name, passwd=password, db=db_name, connect_timeout=5)
  except:
      logger.error("ERROR: Unexpected error: Could not connect to MySql instance.")
      sys.exit()
  
  def handler(event, context):
  
      cur = conn.cursor()
      isql = "INSERT ..."
      cur.execute(isql)
      conn.commit()
      newid = cur.lastrowid
      cur.close()
  
      args = [str(newid),str(event['name'])]
  
      logger.info('ARGS: '+str(args))
      print 'pwd: '
      output = subprocess.check_output(['pwd'])
      print output
      print 'ls -la'
      output = subprocess.check_output(['ls','-l'])
      print output
  
      pid = subprocess.Popen([sys.executable, "app2.py"]+args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE)
  
      logger.info('PID: '+str(pid))
      output = subprocess.check_output(['ls','-l'])
      print output
  
      return "{'status':'success','newid':'"+str(newid)+"'}";
  </code></pre>
  
  <p>The output from "logger.info('PID: '+str(pid))" in app1.py</p>
  
  <p>is like: "PID: &lt;subprocess.Popen object at 0x7f51aba2a550&gt;"</p>
  
  <p><strong>app2</strong></p>
  
  <pre><code>import sys
  import logging
  from datetime import datetime
  import time
  
  fo = open('app2output','a+')
  fo.write("starting with: "+str(sys.argv)+"\n")
  
  logger = logging.getLogger()
  logger.setLevel(logging.INFO)
  logger.info("Starting with: "+str(sys.argv)+"\n")
  
  #log accumulated processing time
  t1 = datetime.now();
  sleep(60)
  t2 = datetime.now();
  tstring = "{'t1':'"+str(t1)+"','t2':'"+str(t2)+"','args':'"+str(sys.argv[1])+"'}"
  logger.info(tstring+"\n")
  fo.write(tstring+"\n")
  fo.close()
  sys.exit()
  </code></pre>