{"action":"created","issue":{"url":"https://api.github.com/repos/mesosphere/marathon-lb/issues/441","repository_url":"https://api.github.com/repos/mesosphere/marathon-lb","labels_url":"https://api.github.com/repos/mesosphere/marathon-lb/issues/441/labels{/name}","comments_url":"https://api.github.com/repos/mesosphere/marathon-lb/issues/441/comments","events_url":"https://api.github.com/repos/mesosphere/marathon-lb/issues/441/events","html_url":"https://github.com/mesosphere/marathon-lb/issues/441","id":218402698,"number":441,"title":"[question] How to improve the stability?","user":{"login":"robsonpeixoto","id":124390,"avatar_url":"https://avatars3.githubusercontent.com/u/124390?v=3","gravatar_id":"","url":"https://api.github.com/users/robsonpeixoto","html_url":"https://github.com/robsonpeixoto","followers_url":"https://api.github.com/users/robsonpeixoto/followers","following_url":"https://api.github.com/users/robsonpeixoto/following{/other_user}","gists_url":"https://api.github.com/users/robsonpeixoto/gists{/gist_id}","starred_url":"https://api.github.com/users/robsonpeixoto/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robsonpeixoto/subscriptions","organizations_url":"https://api.github.com/users/robsonpeixoto/orgs","repos_url":"https://api.github.com/users/robsonpeixoto/repos","events_url":"https://api.github.com/users/robsonpeixoto/events{/privacy}","received_events_url":"https://api.github.com/users/robsonpeixoto/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-03-31T05:36:27Z","updated_at":"2017-03-31T11:41:23Z","closed_at":null,"body":"Hi guys! I'm using the stack Mesos + Marathon + Marathon-lb but we are getting some troubles :(\r\n\r\nFor some unknown reason, the marathon-lb are showing these problem:\r\n\r\n- A new app are add/updated and the marathon-lb do not get the service\r\n- I have 2 instances of marathon-lb. When I access the haproxy status page, it show for the same service in one as healthy and to another one unhealthy\r\n- Just stop do respond and the docker stop do respond.\r\n\r\n\r\nAnd I have some question:\r\n\r\n- Is everyone running marathon in a docker container?\r\n- Is possible to run outside docker container?\r\n- How do you deal with persistent connection? Do you kill old haproxy process?\r\n- What's the docker version are you using? And Storage Driver?\r\n\r\nI'll try to reduce the number of old haproxy process using the below solution. What your opinion?\r\n\r\nWe are using marathon-lb as as L4 and L7 load balance. But for each created task(deploy/task failure/...) the marathon-lb will create a new configuration version and will reload haproxy.\r\nBut it keep old process running waiting to all ports close.\r\n\r\nTo avoid this problem I'll create two marathon-lb instances. One instance to http(L7) services and another one to tcp(L4) ((redis/thrift/...)) services.\r\n\r\nOn the L7 LB(load balancer) I'll use this [script](https://gist.github.com/robsonpeixoto/b44e5eef56db181b5a5a462a2eb3a6f0) to kill old process in a cron running every 5 minutes. It will ensure that all haproxy hold by a websocket connection will be killed.\r\n\r\nAs our tcp service are very stable and has few deploys, it will not be affect other apps problems.\r\n\r\nAny suggestion how to make it works better?\r\n\r\nSome of my server info server info:\r\n- Docker:\r\n```\r\nServer Version: 1.10.3\r\nStorage Driver: aufs\r\nKernel Version: 3.13.0-112-generic\r\nOperating System: Ubuntu 14.04.5 LTS\r\nOSType: linux\r\nArchitecture: x86_64\r\nCPUs: 8\r\nTotal Memory: 6.826 GiB\r\n```\r\n- Mesos: 0.28.2-2.0.27.ubuntu1404\r\n`/usr/sbin/mesos-master --cluster=jusbrasil-mesos-prod --log_dir=/var/log/mesos --logging_level=INFO --port=5050 --quorum=2 --work_dir=/tmp/mesos --zk=zk://zk-1:2181,zk-2:2181,zk-3:2181/mesos_20160425`\r\n\r\n`/usr/sbin/mesos-slave --advertise_ip=10.0.0.1 --cgroups_enable_cfs --cgroups_hierarchy=/cgroup --containerizers=docker,mesos --docker_stop_timeout=50secs --executor_registration_timeout=10mins --executor_shutdown_grace_period=60secs --isolation=cgroups/cpu,cgroups/mem --log_dir=/var/log/mesos --logging_level=INFO --master=zk://zk-1:2181,zk-2:2181,zk-3:2181/mesos_20160425 --port=5051 --recover=reconnect --strict --no-switch_user --work_dir=/tmp/mesos`\r\n\r\n- Marathon: marathon-1.1.7\r\n`java -Xms512m -Xmx2048m -server -jar /opt/marathon/marathon-1.1.7/target/scala-2.11/marathon-assembly-1.1.7.jar --enable_features task_killing --event_subscriber http_callback --master zk://zk-1:2181,zk-2:2181,zk-3:2181/mesos_20160425 --task_launch_timeout 600000 --task_lost_expunge_gc 75000 --task_lost_expunge_initial_delay 300000 --task_lost_expunge_interval 30000 --zk zk://zk-1:2181,zk-2:2181,zk-3:2181/marathon_20160425`\r\n\r\n- Marathon-lb: v1.6.0\r\n`sse -m http://mesos-1:8080 http://mesos-2:8080 http://mesos-3:8080 --group external --group internal --syslog-socket /dev/log`\r\n\r\nThanks"},"comment":{"url":"https://api.github.com/repos/mesosphere/marathon-lb/issues/comments/290690383","html_url":"https://github.com/mesosphere/marathon-lb/issues/441#issuecomment-290690383","issue_url":"https://api.github.com/repos/mesosphere/marathon-lb/issues/441","id":290690383,"user":{"login":"robsonpeixoto","id":124390,"avatar_url":"https://avatars3.githubusercontent.com/u/124390?v=3","gravatar_id":"","url":"https://api.github.com/users/robsonpeixoto","html_url":"https://github.com/robsonpeixoto","followers_url":"https://api.github.com/users/robsonpeixoto/followers","following_url":"https://api.github.com/users/robsonpeixoto/following{/other_user}","gists_url":"https://api.github.com/users/robsonpeixoto/gists{/gist_id}","starred_url":"https://api.github.com/users/robsonpeixoto/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robsonpeixoto/subscriptions","organizations_url":"https://api.github.com/users/robsonpeixoto/orgs","repos_url":"https://api.github.com/users/robsonpeixoto/repos","events_url":"https://api.github.com/users/robsonpeixoto/events{/privacy}","received_events_url":"https://api.github.com/users/robsonpeixoto/received_events","type":"User","site_admin":false},"created_at":"2017-03-31T11:41:23Z","updated_at":"2017-03-31T11:41:23Z","body":"@JayH5, do you have any tip?"}}
{"actor":{"display_login":"robsonpeixoto"}}