{"action":"created","issue":{"url":"https://api.github.com/repos/joyent/containerpilot/issues/464","repository_url":"https://api.github.com/repos/joyent/containerpilot","labels_url":"https://api.github.com/repos/joyent/containerpilot/issues/464/labels{/name}","comments_url":"https://api.github.com/repos/joyent/containerpilot/issues/464/comments","events_url":"https://api.github.com/repos/joyent/containerpilot/issues/464/events","html_url":"https://github.com/joyent/containerpilot/issues/464","id":247948780,"number":464,"title":"ContainerPilot memory leak (v3.3.0)","user":{"login":"JeromeGe","id":5278754,"avatar_url":"https://avatars1.githubusercontent.com/u/5278754?v=4","gravatar_id":"","url":"https://api.github.com/users/JeromeGe","html_url":"https://github.com/JeromeGe","followers_url":"https://api.github.com/users/JeromeGe/followers","following_url":"https://api.github.com/users/JeromeGe/following{/other_user}","gists_url":"https://api.github.com/users/JeromeGe/gists{/gist_id}","starred_url":"https://api.github.com/users/JeromeGe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JeromeGe/subscriptions","organizations_url":"https://api.github.com/users/JeromeGe/orgs","repos_url":"https://api.github.com/users/JeromeGe/repos","events_url":"https://api.github.com/users/JeromeGe/events{/privacy}","received_events_url":"https://api.github.com/users/JeromeGe/received_events","type":"User","site_admin":false},"labels":[{"id":567778717,"url":"https://api.github.com/repos/joyent/containerpilot/labels/v3.x","name":"v3.x","color":"c5def5","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-08-04T09:16:27Z","updated_at":"2017-08-04T13:42:04Z","closed_at":null,"body":"I was using ContainerPilot v3.3.0 and found the ContainerPilot process memory leak.\r\n\r\nLinux 192-168-161-100 4.4.0-79-generic #100-Ubuntu SMP Wed May 17 19:58:14 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux\r\nubuntu 16.04\r\ndocker image: phusion/baseimage:0.9.18\r\n\r\nContainerPilot configuration:\r\n\r\n```\r\n{\r\n  consul: \"127.0.0.1:8500\",\r\n  jobs: [\r\n    {\r\n      name: \"p-job\",\r\n      port: 80,\r\n      exec: \"p-job start\",\r\n      restarts: \"unlimited\",\r\n      when: {\r\n        source: \"generate-file\",\r\n        once: \"exitSuccess\",\r\n      },\r\n      health: {\r\n        exec: \"curl 127.0.0.1:80\",\r\n        interval: 2,\r\n        ttl: 10,\r\n        timeout: \"10s\"\r\n      },\r\n    },\r\n    {\r\n      name: \"consul-client\",\r\n      exec: [\r\n        \"consul\",\r\n        \"agent\",\r\n        \"-config-dir=/etc/consulClient.conf\",\r\n        \"-data-dir=/var/lib/consul\",\r\n        \"-retry-join=consul-server\",\r\n        \"-disable-host-node-id\"\r\n      ],\r\n      health: {\r\n        exec: \"consul operator raft -list-peers | grep leader\",\r\n        interval: 2,\r\n        ttl: 10,\r\n        timeout: \"10s\"\r\n      },\r\n      restarts: \"unlimited\"\r\n    },\r\n    {\r\n      name: \"generate-file\",\r\n      exec: [\r\n        \"consul-template\",\r\n        \"-once\",\r\n        \"-consul-addr\",\r\n        \"127.0.0.1:8500\",\r\n        \"-template\",\r\n        \"/etc/containerpilot/servicesTemplate.ctmpl:/etc/containerpilot/services.json\"\r\n      ],\r\n      when: {\r\n        source: \"consul-client\",\r\n        once: \"healthy\"\r\n      }\r\n    },\r\n    {\r\n      name: \"consul-leave\",\r\n      exec: \"consul leave\",\r\n      when: {\r\n        source: \"consul-client\",\r\n        once: \"stopping\"\r\n      }\r\n    },\r\n    {\r\n      name: \"regenerate-file-a\",\r\n      exec: [\r\n        \"consul-template\",\r\n        \"-once\",\r\n        \"-consul-addr\",\r\n        \"127.0.0.1:8500\",\r\n        \"-template\",\r\n        \"/etc/containerpilot/servicesTemplate.ctmpl:/etc/containerpilot/services.json\"\r\n      ],\r\n      when: {\r\n        source: \"watch.a\",\r\n        each: \"changed\"\r\n      }\r\n    },\r\n    {\r\n      name: \"regenerate-file-b\",\r\n      exec: [\r\n        \"consul-template\",\r\n        \"-once\",\r\n        \"-consul-addr\",\r\n        \"127.0.0.1:8500\",\r\n        \"-template\",\r\n        \"/etc/containerpilot/servicesTemplate.ctmpl:/etc/containerpilot/services.json\"\r\n      ],\r\n      when: {\r\n        source: \"watch.b\",\r\n        each: \"changed\"\r\n      }\r\n    },\r\n    {\r\n      name: \"regenerate-file-c\",\r\n      exec: [\r\n        \"consul-template\",\r\n        \"-once\",\r\n        \"-consul-addr\",\r\n        \"127.0.0.1:8500\",\r\n        \"-template\",\r\n        \"/etc/containerpilot/servicesTemplate.ctmpl:/etc/containerpilot/services.json\"\r\n      ],\r\n      when: {\r\n        source: \"watch.c\",\r\n        each: \"changed\"\r\n      }\r\n    },\r\n    {\r\n      name: \"send-sighup-a\",\r\n      exec: \"kill -HUP ${p-job PID}\",\r\n      when: {\r\n        source: \"regenerate-file-a\",\r\n        each: \"exitSuccess\"\r\n      }\r\n    },\r\n    {\r\n      name: \"send-sighup-b\",\r\n      exec: \"kill -HUP ${p-job PID}\",\r\n      when: {\r\n        source: \"regenerate-file-b\",\r\n        each: \"exitSuccess\"\r\n      }\r\n    },\r\n    {\r\n      name: \"send-sighup-c\",\r\n      exec: \"kill -HUP ${p-job PID}\",\r\n      when: {\r\n        source: \"regenerate-file-c\",\r\n        each: \"exitSuccess\"\r\n      }\r\n    },\r\n  ],\r\n  watches: [\r\n    {\r\n      name: \"a\",\r\n      interval: 1\r\n    },\r\n    {\r\n      name: \"b\",\r\n      interval: 1\r\n    },\r\n    {\r\n      name: \"c\",\r\n      interval: 1\r\n    },\r\n    \r\n  ],\r\n  control: {\r\n    socket: \"/var/run/containerpilot.socket\"\r\n  }\r\n}\r\n```\r\n\r\npmap -d ${containerpilot PID}\r\n```\r\nAddress           Kbytes Mode  Offset           Device    Mapping\r\n0000000000400000    3788 r-x-- 0000000000000000 000:000bc containerpilot\r\n00000000007b3000    2868 r---- 00000000003b3000 000:000bc containerpilot\r\n0000000000a80000     216 rw--- 0000000000680000 000:000bc containerpilot\r\n0000000000ab6000     140 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c000000000      56 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c41fe58000    1696 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c420000000    7168 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c420700000    3072 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c420a00000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c420b00000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c420c00000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c420d00000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c420e00000    2048 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c421000000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c421100000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c421200000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c421300000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c421400000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c421500000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c421600000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c421700000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c421800000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c421900000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c421a00000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c421b00000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c421c00000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c421d00000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c421e00000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c421f00000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c422000000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c422100000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c422200000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c422300000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c422400000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c422500000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c422600000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c422700000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c422800000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c422900000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c422a00000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c422b00000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c422c00000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c422d00000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c422e00000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c422f00000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c423000000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c423100000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c423200000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c423300000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n000000c423400000    1024 rw--- 0000000000000000 000:00000   [ anon ]\r\n00007fa8b9817000    3584 rw--- 0000000000000000 000:00000   [ anon ]\r\n00007ffee4d07000     136 rw--- 0000000000000000 000:00000   [ stack ]\r\n00007ffee4da6000       8 r---- 0000000000000000 000:00000   [ anon ]\r\n00007ffee4da8000       8 r-x-- 0000000000000000 000:00000   [ anon ]\r\nffffffffff600000       4 r-x-- 0000000000000000 000:00000   [ anon ]\r\nmapped: 66776K    writeable/private: 60100K    shared: 0K\r\n```\r\n\r\ndump and view ContainerPilot memory, the memory stores a large number of duplicate the following:\r\n\r\n- container environment variable\r\n- a, b, c services registered to consul information\r\n- Send-sighup event"},"comment":{"url":"https://api.github.com/repos/joyent/containerpilot/issues/comments/320252046","html_url":"https://github.com/joyent/containerpilot/issues/464#issuecomment-320252046","issue_url":"https://api.github.com/repos/joyent/containerpilot/issues/464","id":320252046,"user":{"login":"tgross","id":1409219,"avatar_url":"https://avatars3.githubusercontent.com/u/1409219?v=4","gravatar_id":"","url":"https://api.github.com/users/tgross","html_url":"https://github.com/tgross","followers_url":"https://api.github.com/users/tgross/followers","following_url":"https://api.github.com/users/tgross/following{/other_user}","gists_url":"https://api.github.com/users/tgross/gists{/gist_id}","starred_url":"https://api.github.com/users/tgross/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tgross/subscriptions","organizations_url":"https://api.github.com/users/tgross/orgs","repos_url":"https://api.github.com/users/tgross/repos","events_url":"https://api.github.com/users/tgross/events{/privacy}","received_events_url":"https://api.github.com/users/tgross/received_events","type":"User","site_admin":false},"created_at":"2017-08-04T13:42:04Z","updated_at":"2017-08-04T13:42:04Z","body":"Because I don't have your application code I can't reproduce your scenario exactly. You're reporting a problem with ContainerPilot itself allegedly leaking memory and not the overall container, so it shouldn't matter what the underlying applications are anyways. But I'm trying to come up with a more minimal reproduction and this is what I've done so far (without success). \r\n\r\nI wrote a quickie hello world golang server and built it for Linux:\r\n\r\n```go\r\npackage main\r\n\r\nimport (\r\n\t\"fmt\"\r\n\t\"log\"\r\n\t\"net/http\"\r\n)\r\n\r\nfunc main() {\r\n\thttp.HandleFunc(\"/\", func(w http.ResponseWriter, r *http.Request) {\r\n\t\tfmt.Fprintf(w, \"Hello, world\")\r\n\t})\r\n\tlog.Println(\"listening on 8080...\")\r\n\tlog.Fatal(http.ListenAndServe(\":8080\", nil))\r\n}\r\n```\r\n\r\nDropped it into this Dockerfile:\r\n\r\n```\r\nFROM alpine:latest\r\n\r\nRUN apk --no-cache add curl unzip procps\r\n\r\nRUN curl --fail -Lso /tmp/consul-template.zip \"https://releases.hashicorp.com/consul-template/0.18.3/consul-template_0.18.3_linux_amd64.zip\" \\\r\n    && unzip /tmp/consul-template.zip -d /usr/local/bin \\\r\n    && rm /tmp/consul-template.zip\r\n\r\nRUN mkdir -p /etc/containerpilot \\\r\n    && touch /etc/containerpilot/servicesTemplate.ctmpl\r\n\r\nCOPY containerpilot /bin/containerpilot\r\nCOPY local.json5 /etc/containerpilot.json5\r\nCOPY pjob /bin/pjob\r\n```\r\n\r\nAnd used this ContainerPilot config:\r\n\r\n```json5\r\n{\r\n  consul: \"consul:8500\",\r\n  jobs: [\r\n    {\r\n      name: \"pjob\",\r\n      port: 8080,\r\n      exec: \"/bin/pjob start\",\r\n      restarts: \"unlimited\",\r\n      health: {\r\n        exec: \"curl --fail -s http://127.0.0.1:8080\",\r\n        interval: 2,\r\n        ttl: 10,\r\n        timeout: \"10s\"\r\n      },\r\n    },\r\n    {\r\n      name: \"regen-file\",\r\n      exec: [\r\n        \"consul-template\",\r\n        \"-once\",\r\n        \"-consul-addr\",\r\n        \"consul:8500\",\r\n        \"-template\",\r\n        \"/etc/containerpilot/servicesTemplate.ctmpl:/etc/containerpilot/services.json\"\r\n      ],\r\n      when: {\r\n        source: \"watch.aaa\",\r\n        each: \"changed\"\r\n      }\r\n    },\r\n    {\r\n      name: \"send-sighup-a\",\r\n      exec: \"pkill -SIGHUP pjob\",\r\n      when: {\r\n        source: \"regen-file\",\r\n        each: \"exitSuccess\"\r\n      }\r\n    },\r\n  ],\r\n  watches: [\r\n    {\r\n      name: \"aaa\",\r\n      interval: 1\r\n    }\r\n  ]\r\n}\r\n```\r\n\r\nThen I `docker exec` into a shell the container and take a look at what we've got:\r\n\r\n```bash\r\n/ # ps -ef\r\nUID        PID  PPID  C STIME TTY          TIME CMD\r\nroot         1     0  0 13:25 ?        00:00:00 /bin/containerpilot -config=/etc/containerpilot.json\r\nroot        10     1  0 13:25 ?        00:00:01 /bin/containerpilot -config=/etc/containerpilot.json\r\nroot       111     0  0 13:27 pts/0    00:00:00 /bin/sh\r\nroot       336    10  0 13:33 ?        00:00:00 /bin/pjob start\r\nroot       350   111  0 13:33 pts/0    00:00:00 ps -ef\r\n\r\n# pmap the supervisor\r\n/ # pmap 1\r\n1: /bin/containerpilot -config=/etc/containerpilot.json5\r\n0000000000400000    3788K r-xp  /bin/containerpilot\r\n00000000007b3000    2868K r--p  /bin/containerpilot\r\n0000000000a80000     216K rw-p  /bin/containerpilot\r\n0000000000ab6000     140K rw-p    [ anon ]\r\n000000c000000000       4K rw-p    [ anon ]\r\n000000c41fff0000      64K rw-p    [ anon ]\r\n000000c420000000    2048K rw-p    [ anon ]\r\n00007f60e676a000     640K rw-p    [ anon ]\r\n00007ffc5f40d000     132K rw-p  [stack]\r\n00007ffc5f5e7000       8K r--p  [vvar]\r\n00007ffc5f5e9000       8K r-xp  [vdso]\r\nffffffffff600000       4K r-xp  [vsyscall]\r\nmapped: 9920K\r\n\r\n# pmap the worker\r\n/ # pmap 10\r\n10: /bin/containerpilot -config=/etc/containerpilot.json5\r\n0000000000400000    3788K r-xp  /bin/containerpilot\r\n00000000007b3000    2868K r--p  /bin/containerpilot\r\n0000000000a80000     216K rw-p  /bin/containerpilot\r\n0000000000ab6000     140K rw-p    [ anon ]\r\n000000c000000000       8K rw-p    [ anon ]\r\n000000c41ffd0000    6336K rw-p    [ anon ]\r\n00007fc7efffe000    2176K rw-p    [ anon ]\r\n00007fffc763b000     132K rw-p  [stack]\r\n00007fffc77dc000       8K r--p  [vvar]\r\n00007fffc77de000       8K r-xp  [vdso]\r\nffffffffff600000       4K r-xp  [vsyscall]\r\nmapped: 15684K\r\n\r\n/ # free -m\r\n             total       used       free     shared    buffers     cached\r\nMem:          1998        610       1388        163         51        391\r\n-/+ buffers/cache:        167       1831\r\nSwap:          964          0        964\r\n```\r\n\r\nThe pmap doesn't change over time, nor does the free memory. I've `pkill -SIGHUP` the pjob application manually and this doesn't seem to cause an increase in the use of memory either."}}
{"actor":{"display_login":"tgross"}}