title: php application suddenly slow on AWS
body: <p>I'm running an old PHP (7.1) / NGINX (1.10.2) application on AWS. The application runs on AWS more than few months. Since 2 days we experiencing high latency issues. But it does not affect the whole page. Only "intensive" PHP processes seems to have problems deliver the content. </p>
  
  <p>I looked up so many other related topics now but nothing was pointing me to the correct direction. </p>
  
  <p>First of all: The latency is not related to networking as I get these latencies also when sending a request from the server to localhost. It doesnt seem to be related to the database as well (the website is able to connect to the RDS DB within &lt;3ms and DB CPU ~20% and Memory free >2GB looks good). Connecting to the database and running some queries made by the webserver are also performing fine.</p>
  
  <p>The Webserver itself doesnt consume so much hardware resources (CPU 10-25% and Memory free ~2GB). No cronjobs/scheduled tasks are installed on this server. More than 50% iNodes still available on the server. The network gateway is retrieving/transmitting 8-25MB/seconds. We do not monitor any kind of DoS at all.</p>
  
  <p>I already checked and tried to adjust the PHP FPM settings (memory_limit, process management, children processes etc.) Nothing helped here. Deactiving/Activating OPCache doesnt really has an impact.</p>
  
  <p>Even when I use an AMI from a previous installation and start a new server the same latency issue happens again. Same happens when running the application in multiple availability zones.</p>
  
  <p>To see where PHP is spending time I used blackfire.io and actually its telling me it spends most of the time on mysql interactions (which is not surprisingly as the application sends a lot of dirty queries with a lot of joins etc. and its the only performance expensive thing here..). I also added some debug output to the code itself. It usually gets completed within less than 6 seconds (which is sadly the normal average we know from our search..)</p>
  
  <p>The latency according to the target group is by average between 3-8 seconds but we also discover a lot of latencies where the request times out (30-60seconds).</p>
  
  <p>At this point I'm even quite unsure about what to provide here. I don't want to paste every related config etc here. So please tell me what you need to be able to help here :/</p>
  
  <p>php-fpm/nginx log doesn't log anything related to this issue. Same with the syslog. The only thing which can be found there is <code>Timed out waiting for reply from 91.189.89.199:123 (ntp.ubuntu.com)</code> but even <code>date</code> is still in sync.. PHP FPM slow-log (timeout set to 5sec.) is empty as well. ELB Access logs are monitoring high "backend_processing_time"s only.</p>
  
  <p>Nginx actually routes requests to an S3 bucket and besides one S3 mount we do not have any huge amount of temp files or other stuff on the server. </p>
  
  <p>Requests send to the internet are performing as expected. DNS seems also not to be the problem (can reach db and other services in the internet as usual).</p>
  
  <p>Does anyone has ideas what could cause these latency issues? What else should/<strong>can</strong> be investigated?? I really appreciate every help or question which can point me to the correct direction.<br>
  Best Regards.</p>