{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/prometheus-operator/issues/481","repository_url":"https://api.github.com/repos/coreos/prometheus-operator","labels_url":"https://api.github.com/repos/coreos/prometheus-operator/issues/481/labels{/name}","comments_url":"https://api.github.com/repos/coreos/prometheus-operator/issues/481/comments","events_url":"https://api.github.com/repos/coreos/prometheus-operator/issues/481/events","html_url":"https://github.com/coreos/prometheus-operator/issues/481","id":243305630,"number":481,"title":"unable to change configuration","user":{"login":"lwolf","id":6144353,"avatar_url":"https://avatars6.githubusercontent.com/u/6144353?v=4","gravatar_id":"","url":"https://api.github.com/users/lwolf","html_url":"https://github.com/lwolf","followers_url":"https://api.github.com/users/lwolf/followers","following_url":"https://api.github.com/users/lwolf/following{/other_user}","gists_url":"https://api.github.com/users/lwolf/gists{/gist_id}","starred_url":"https://api.github.com/users/lwolf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lwolf/subscriptions","organizations_url":"https://api.github.com/users/lwolf/orgs","repos_url":"https://api.github.com/users/lwolf/repos","events_url":"https://api.github.com/users/lwolf/events{/privacy}","received_events_url":"https://api.github.com/users/lwolf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-07-17T06:27:36Z","updated_at":"2017-07-17T16:02:57Z","closed_at":null,"body":"**What did you do?**\r\nI have a tectonic cluster. Prometheus-operator was installed as part of it. I want to increase resource requests and retention by editing Prometheus object (not sure if this is the best way)\r\n`kubectl edit Prometheus k8s --namespace=tectonic-system`\r\nAfter saving, Statefulset is getting recreated but after a few minutes, its getting killed. The new one, recreated after this has default settings.\r\n\r\n**What did you expect to see?**\r\nStateful set with updated configuration\r\n\r\n**What did you see instead? Under which circumstances?**\r\nAlways the same configuration\r\n\r\n**Environment**\r\nTectonic 1.6.7-tectonic.1\r\nKubernetes 1.6.7+tectonic.1\r\ntectonic-monitoring 1.3.0\r\n\r\n* Kubernetes version information:\r\n\r\n```\r\n$ kubectl version\r\nClient Version: version.Info{Major:\"1\", Minor:\"6\", GitVersion:\"v1.6.6\", GitCommit:\"7fa1c1756d8bc963f1a389f4a6937dc71f08ada2\", GitTreeState:\"clean\", BuildDate:\"2017-06-16T18:34:20Z\", GoVersion:\"go1.7.6\", Compiler:\"gc\", Platform:\"darwin/amd64\"}\r\nServer Version: version.Info{Major:\"1\", Minor:\"6\", GitVersion:\"v1.6.7+coreos.0\", GitCommit:\"c8c505ee26ac3ab4d1dff506c46bc5538bc66733\", GitTreeState:\"clean\", BuildDate:\"2017-07-06T17:38:33Z\", GoVersion:\"go1.7.6\", Compiler:\"gc\", Platform:\"linux/amd64\"}\r\n```\r\n\r\n* Kubernetes cluster kind: \r\n\r\ntectonic-installer\r\n\r\n* Prometheus Operator Logs:\r\n\r\nNothing strange. Dozens of similar lines\r\n```\r\nts=2017-07-17T06:18:54Z caller=operator.go:619 component=prometheusoperator msg=\"sync prometheus\" key=tectonic-system/k8s\r\nts=2017-07-17T06:18:54Z caller=operator.go:972 component=prometheusoperator msg=\"updating config skipped, no configuration change\"\r\nts=2017-07-17T06:18:54Z caller=operator.go:584 component=prometheusoperator msg=\"update handler\" old=7003568 cur=7006925\r\nts=2017-07-17T06:18:54Z caller=operator.go:321 component=alertmanageroperator msg=\"update handler\" old=7003568 cur=7006925\r\nts=2017-07-17T06:18:54Z caller=operator.go:619 component=prometheusoperator msg=\"sync prometheus\" key=tectonic-system/k8s\r\nts=2017-07-17T06:18:54Z caller=operator.go:972 component=prometheusoperator msg=\"updating config skipped, no configuration change\"\r\nts=2017-07-17T06:18:54Z caller=operator.go:584 component=prometheusoperator msg=\"update handler\" old=7006925 cur=7006926\r\nts=2017-07-17T06:18:54Z caller=operator.go:321 component=alertmanageroperator msg=\"update handler\" old=7006925 cur=7006926\r\n```\r\nIn kubectl describe pod just before restoring default configuration.\r\n```\r\n  4s\t\t4s\t\t1\tkubelet, <hostname>\tspec.containers{prometheus-config-reloader}\tNormal\t\tKilling\t\tKilling container with id docker://d5a160f8ee734b7423e088711f757f258a20b6120c4dc409a99f12541044e86a:Need to kill Pod\r\n  2s\t\t2s\t\t1\tkubelet, <hostname>\tspec.containers{prometheus}\t\t\tNormal\t\tKilling\t\tKilling container with id docker://0a08bab80d70dc12666940e3fa0a2d2fe51b28ec80ce7305181354eb84942e40:Need to kill Pod\r\n```\r\n\r\nAm I doing something wrong? Is there a correct way to change settings for the operator?\r\n"},"comment":{"url":"https://api.github.com/repos/coreos/prometheus-operator/issues/comments/315798773","html_url":"https://github.com/coreos/prometheus-operator/issues/481#issuecomment-315798773","issue_url":"https://api.github.com/repos/coreos/prometheus-operator/issues/481","id":315798773,"user":{"login":"lwolf","id":6144353,"avatar_url":"https://avatars6.githubusercontent.com/u/6144353?v=4","gravatar_id":"","url":"https://api.github.com/users/lwolf","html_url":"https://github.com/lwolf","followers_url":"https://api.github.com/users/lwolf/followers","following_url":"https://api.github.com/users/lwolf/following{/other_user}","gists_url":"https://api.github.com/users/lwolf/gists{/gist_id}","starred_url":"https://api.github.com/users/lwolf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lwolf/subscriptions","organizations_url":"https://api.github.com/users/lwolf/orgs","repos_url":"https://api.github.com/users/lwolf/repos","events_url":"https://api.github.com/users/lwolf/events{/privacy}","received_events_url":"https://api.github.com/users/lwolf/received_events","type":"User","site_admin":false},"created_at":"2017-07-17T16:02:57Z","updated_at":"2017-07-17T16:02:57Z","body":"I created configmap `tectonic-monitoring` in `tectonic-system` namespace and it worked for rentention and resources. But not for storage.\r\n\r\nI have manually provisioned PV with label `prometheus`.\r\nConfigmap looks like this\r\n```\r\napiVersion: v1\r\nkind: ConfigMap\r\nmetadata:\r\n  name: tectonic-monitoring\r\n  namespace: tectonic-system\r\ndata:\r\n  config.yaml: |+\r\n    prometheusK8s:\r\n      retention: 744h\r\n      resources:\r\n        limits:\r\n          cpu: 800m\r\n          memory: 3000Mi\r\n        requests:\r\n          cpu: 400m\r\n          memory: 2000Mi\r\n      storage:\r\n        volumeClaimTemplate:\r\n          selector:\r\n            matchLabels:\r\n              app: \"prometheus\"\r\n          spec:\r\n            resources:\r\n              storage: 100Gi\r\n```"}}
{"actor":{"display_login":"lwolf"}}