{"action":"created","issue":{"url":"https://api.github.com/repos/docker/docker/issues/30995","repository_url":"https://api.github.com/repos/docker/docker","labels_url":"https://api.github.com/repos/docker/docker/issues/30995/labels{/name}","comments_url":"https://api.github.com/repos/docker/docker/issues/30995/comments","events_url":"https://api.github.com/repos/docker/docker/issues/30995/events","html_url":"https://github.com/docker/docker/issues/30995","id":207387363,"number":30995,"title":"High CPU usage and possible memory leak on docker logs with SpringBoot applications.","user":{"login":"pythianarceri","id":10517648,"avatar_url":"https://avatars.githubusercontent.com/u/10517648?v=3","gravatar_id":"","url":"https://api.github.com/users/pythianarceri","html_url":"https://github.com/pythianarceri","followers_url":"https://api.github.com/users/pythianarceri/followers","following_url":"https://api.github.com/users/pythianarceri/following{/other_user}","gists_url":"https://api.github.com/users/pythianarceri/gists{/gist_id}","starred_url":"https://api.github.com/users/pythianarceri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pythianarceri/subscriptions","organizations_url":"https://api.github.com/users/pythianarceri/orgs","repos_url":"https://api.github.com/users/pythianarceri/repos","events_url":"https://api.github.com/users/pythianarceri/events{/privacy}","received_events_url":"https://api.github.com/users/pythianarceri/received_events","type":"User","site_admin":false},"labels":[{"id":182417459,"url":"https://api.github.com/repos/docker/docker/labels/status/more-info-needed","name":"status/more-info-needed","color":"C6A49A","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2017-02-14T01:06:00Z","updated_at":"2017-02-14T19:27:15Z","closed_at":null,"body":"**Description**\r\nHigh CPU usage and possible memory leak on docker logs with SpringBoot applications.\r\nWe have a rancher environment with several containers running SpringBoot Applications, we have an ELK stack to which we send container logs using logspout with the logstash adapter.\r\nFor some unknown reason, if we allow logspout to collect logs from these containers dockerd cpu utilization and memory utilization start to climb (i've seen load averages of up to 70), to the point that the server needs to be rebooted in order to recover it.\r\nFurthermore when looking at the container logs using \"docker logs\"  i get different output every time i run the command, and that output last ten lines never match what docker logs --tail=10 is displaying\r\n```[rancher@ip-10-30-0-193 ~]$ docker logs c61f0d662ef5\r\n\r\n[rancher@ip-10-30-0-193 ~]$ docker logs c61f0d662ef5\r\n\r\n[rancher@ip-10-30-0-193 ~]$ docker logs c61f0d662ef5\r\n\r\n[rancher@ip-10-30-0-193 ~]$ docker logs c61f0d662ef5\r\n\r\n  .   ____          _            __ _ _\r\n /\\\\ / ___'_ __ _ _(_)_ __  __ _ \\ \\ \\ \\\r\n( ( )\\___ | '_ | '_| | '_ \\/ _` | \\ \\ \\ \\\r\n \\\\/  ___)| |_)| | | | | || (_| |  ) ) ) )\r\n  '  |____| .__|_| |_|_| |_\\__, | / / / /\r\n =========|_|==============|___/=/_/_/_/\r\n[rancher@ip-10-30-0-193 ~]$ docker logs c61f0d662ef5\r\n\r\n[rancher@ip-10-30-0-193 ~]$ docker logs c61f0d662ef5\r\n\r\n  .   ____          _            __ _ _\r\n[rancher@ip-10-30-0-193 ~]$ docker logs --tail=10 c61f0d662ef5\r\n2017-02-10 17:50:55.161  INFO 1 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]\r\n2017-02-10 17:50:56.193  INFO 1 --- [           main] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler]\r\n2017-02-10 17:51:01.775  INFO 1 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup\r\n2017-02-10 17:51:01.921  INFO 1 --- [           main] o.s.c.support.DefaultLifecycleProcessor  : Starting beans in phase 0\r\n2017-02-10 17:51:19.895  INFO 1 --- [           main] /                                        : Initializing Spring FrameworkServlet 'dispatcherServlet'\r\n2017-02-10 17:51:19.917  INFO 1 --- [           main] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization started\r\n2017-02-10 17:51:20.495  INFO 1 --- [           main] o.s.web.servlet.DispatcherServlet        : FrameworkServlet 'dispatcherServlet': initialization completed in 570 ms\r\n2017-02-10 17:51:21.140  INFO 1 --- [           main] o.eclipse.jetty.server.ServerConnector   : Started ServerConnector@48433ada{HTTP/1.1}{0.0.0.0:8100}\r\n2017-02-10 17:51:21.217  INFO 1 --- [           main] .s.b.c.e.j.JettyEmbeddedServletContainer : Jetty started on port(s) 8100 (http/1.1)\r\n2017-02-10 17:51:21.272  INFO 1 --- [           main] com.bps.statement.analysis.Application   : Started Application in 161.074 seconds (JVM running for 178.074)\r\n```\r\nIs it possible that multiline messages are somehow \"confusing\" docker ?\r\n\r\n**Steps to reproduce the issue:**\r\n1. Deploy springboot application\r\n2. Configure logspout to collect logs from the application and send them to logstash\r\n3. Server performance starts degrading\r\n\r\n**Describe the results you received:**\r\nServer will ultimately crash, dockerd can be seen using 80%+ CPU and its memory utilization starts climbing.\r\n\r\n**Describe the results you expected:**\r\n- dockerd should not hog the cpu/memory\r\n- `docker logs` output should be the same everytime the command is executed\r\n\r\n**Additional information you deem important (e.g. issue happens only occasionally):**\r\n\r\n**Output of `docker version`:**\r\n```\r\n[rancher@ip-10-30-0-193 ~]$ docker version\r\nClient:\r\n Version:      1.12.3\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   6b644ec\r\n Built:        Wed Oct 26 23:26:11 2016\r\n OS/Arch:      linux/amd64\r\n\r\nServer:\r\n Version:      1.12.3\r\n API version:  1.24\r\n Go version:   go1.6.3\r\n Git commit:   6b644ec\r\n Built:        Wed Oct 26 23:26:11 2016\r\n OS/Arch:      linux/amd64\r\n```\r\n\r\n**Output of `docker info`:**\r\n\r\n```\r\n[rancher@ip-10-30-0-193 ~]$ docker info\r\nContainers: 62\r\n Running: 40\r\n Paused: 0\r\n Stopped: 22\r\nImages: 94\r\nServer Version: 1.12.3\r\nStorage Driver: overlay\r\n Backing Filesystem: extfs\r\nLogging Driver: json-file\r\nCgroup Driver: cgroupfs\r\nPlugins:\r\n Volume: local rancher-nfs\r\n Network: host null bridge overlay\r\nSwarm: inactive\r\nRuntimes: runc\r\nDefault Runtime: runc\r\nSecurity Options: seccomp\r\nKernel Version: 4.4.24-rancher\r\nOperating System: RancherOS v0.7.1\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 2\r\nTotal Memory: 14.94 GiB\r\nName: ip-10-30-0-193.ec2.internal\r\nID: DVSV:NLW5:VD5U:UOYY:FIOB:4RTH:7NGQ:GIBO:KKTR:VPYO:J6A2:4SZE\r\nDocker Root Dir: /var/lib/docker\r\nDebug Mode (client): false\r\nDebug Mode (server): false\r\nRegistry: https://index.docker.io/v1/\r\nInsecure Registries:\r\n 127.0.0.0/8\r\n```\r\n\r\n**Additional environment details (AWS, VirtualBox, physical, etc.):**\r\nAWS"},"comment":{"url":"https://api.github.com/repos/docker/docker/issues/comments/279809149","html_url":"https://github.com/docker/docker/issues/30995#issuecomment-279809149","issue_url":"https://api.github.com/repos/docker/docker/issues/30995","id":279809149,"user":{"login":"pythianarceri","id":10517648,"avatar_url":"https://avatars.githubusercontent.com/u/10517648?v=3","gravatar_id":"","url":"https://api.github.com/users/pythianarceri","html_url":"https://github.com/pythianarceri","followers_url":"https://api.github.com/users/pythianarceri/followers","following_url":"https://api.github.com/users/pythianarceri/following{/other_user}","gists_url":"https://api.github.com/users/pythianarceri/gists{/gist_id}","starred_url":"https://api.github.com/users/pythianarceri/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pythianarceri/subscriptions","organizations_url":"https://api.github.com/users/pythianarceri/orgs","repos_url":"https://api.github.com/users/pythianarceri/repos","events_url":"https://api.github.com/users/pythianarceri/events{/privacy}","received_events_url":"https://api.github.com/users/pythianarceri/received_events","type":"User","site_admin":false},"created_at":"2017-02-14T19:27:15Z","updated_at":"2017-02-14T19:27:15Z","body":"I think the customer has made some modifications to the container, which apparently is not generating any logs now, i'll keep you updated."}}
{"actor":{"display_login":"pythianarceri"}}