{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/3625","repository_url":"https://api.github.com/repos/prometheus/prometheus","labels_url":"https://api.github.com/repos/prometheus/prometheus/issues/3625/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/3625/comments","events_url":"https://api.github.com/repos/prometheus/prometheus/issues/3625/events","html_url":"https://github.com/prometheus/prometheus/issues/3625","id":284546962,"number":3625,"title":"Is it possible to disable crash recovery?","user":{"login":"paragsomani","id":13316462,"avatar_url":"https://avatars3.githubusercontent.com/u/13316462?v=4","gravatar_id":"","url":"https://api.github.com/users/paragsomani","html_url":"https://github.com/paragsomani","followers_url":"https://api.github.com/users/paragsomani/followers","following_url":"https://api.github.com/users/paragsomani/following{/other_user}","gists_url":"https://api.github.com/users/paragsomani/gists{/gist_id}","starred_url":"https://api.github.com/users/paragsomani/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/paragsomani/subscriptions","organizations_url":"https://api.github.com/users/paragsomani/orgs","repos_url":"https://api.github.com/users/paragsomani/repos","events_url":"https://api.github.com/users/paragsomani/events{/privacy}","received_events_url":"https://api.github.com/users/paragsomani/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-12-26T12:15:44Z","updated_at":"2017-12-26T13:20:40Z","closed_at":"2017-12-26T13:16:44Z","author_association":"NONE","body":"<!--\r\n\r\n    Please do *NOT* ask usage questions in Github issues.\r\n\r\n    If your issue is not a feature request or bug report use:\r\n    https://groups.google.com/forum/#!forum/prometheus-users. If\r\n    you are unsure whether you hit a bug, search and ask in the\r\n    mailing list first.\r\n\r\n    You can find more information at: https://prometheus.io/community/\r\n\r\n-->\r\n\r\n**What did you do?**\r\nI have a prometheus server deployed on openshift. We are monitoring different pods, kublets etc. Ingestion rate is ~10K/second.\r\nPolling/scraping interval: 10s\r\nMemory assigned: 8GiB\r\nCPU: 6 cores\r\nDisk type: NFS/HDD\r\nstorage.local.target-heap-size: 5.7GiB\r\nretention period: 1h\r\n\r\nWith above config, prometheus frequently runs into rushed mode(with urgency score 1) and resulting less scraping of data. If we are trying to stop it(using SIGTERM) it takes too much time(in openshift grace period of shutdown is 15m), and still does not gets shutdown gracefully. This causes crash recovery of data which again causes delay in start for 15-20m.\r\n\r\nCan you advice, whether i can disable with crash recovery of data, so that i can work with some loss of data?\r\n\r\n**What did you expect to see?**\r\nDisabling of crash recovery.\r\n**What did you see instead? Under which circumstances?**\r\n\r\n**Environment**\r\nopenshift\r\n* System information:\r\n\r\nopenshift 3.5\r\n\r\n* Prometheus version:\r\nv 1.8.2\r\n\r\n\r\n* Alertmanager version:\r\n\r\nNA\r\n\r\n* Prometheus configuration file:\r\n```\r\nglobal:\r\n  scrape_interval: 1m\r\n  scrape_timeout: 10s\r\n  evaluation_interval: 1m\r\nscrape_configs:\r\n- job_name: kubernetes-nodes\r\n  scrape_interval: 1m\r\n  scrape_timeout: 10s\r\n  metrics_path: /metrics\r\n  scheme: https\r\n  kubernetes_sd_configs:\r\n  - api_server: null\r\n    role: node\r\n    namespaces:\r\n      names: []\r\n  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token\r\n  tls_config:\r\n    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt\r\n    insecure_skip_verify: false\r\n  relabel_configs:\r\n  - separator: ;\r\n    regex: __meta_kubernetes_node_label_(.+)\r\n    replacement: $1\r\n    action: labelmap\r\n  - separator: ;\r\n    regex: (.*)\r\n    target_label: __address__\r\n    replacement: kubernetes.default.svc:443\r\n    action: replace\r\n  - source_labels: [__meta_kubernetes_node_name]\r\n    separator: ;\r\n    regex: (.+)\r\n    target_label: __metrics_path__\r\n    replacement: /api/v1/nodes/${1}/proxy/metrics\r\n    action: replace\r\n- job_name: kubernetes-pods\r\n  scrape_interval: 10s\r\n  scrape_timeout: 10s\r\n  metrics_path: /metrics\r\n  scheme: http\r\n  kubernetes_sd_configs:\r\n  - api_server: null\r\n    role: pod\r\n    namespaces:\r\n      names:\r\n      - XYZ\r\n      - ABC\r\n      - DEF\r\n  relabel_configs:\r\n  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]\r\n    separator: ;\r\n    regex: \"true\"\r\n    replacement: $1\r\n    action: keep\r\n  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]\r\n    separator: ;\r\n    regex: (.+)\r\n    target_label: __metrics_path__\r\n    replacement: $1\r\n    action: replace\r\n  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]\r\n    separator: ;\r\n    regex: ([^:]+)(?::\\d+)?;(\\d+)\r\n    target_label: __address__\r\n    replacement: $1:$2\r\n    action: replace\r\n  - separator: ;\r\n    regex: __meta_kubernetes_pod_label_(.+)\r\n    replacement: $1\r\n    action: labelmap\r\n  - source_labels: [__meta_kubernetes_namespace]\r\n    separator: ;\r\n    regex: (.*)\r\n    target_label: kubernetes_namespace\r\n    replacement: $1\r\n    action: replace\r\n  - source_labels: [__meta_kubernetes_pod_name]\r\n    separator: ;\r\n    regex: (.*)\r\n    target_label: kubernetes_pod_name\r\n    replacement: $1\r\n    action: replace\r\nremote_write:\r\n- url: http://prometheus-sop.ft-aramse:9096\r\n  remote_timeout: 30s\r\n  queue_config:\r\n    capacity: 100000\r\n    max_shards: 1000\r\n    max_samples_per_send: 100\r\n    batch_send_deadline: 5s\r\n    max_retries: 10\r\n    min_backoff: 30ms\r\n    max_backoff: 100ms\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/comments/353968192","html_url":"https://github.com/prometheus/prometheus/issues/3625#issuecomment-353968192","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/3625","id":353968192,"user":{"login":"jamtur01","id":4365,"avatar_url":"https://avatars3.githubusercontent.com/u/4365?v=4","gravatar_id":"","url":"https://api.github.com/users/jamtur01","html_url":"https://github.com/jamtur01","followers_url":"https://api.github.com/users/jamtur01/followers","following_url":"https://api.github.com/users/jamtur01/following{/other_user}","gists_url":"https://api.github.com/users/jamtur01/gists{/gist_id}","starred_url":"https://api.github.com/users/jamtur01/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jamtur01/subscriptions","organizations_url":"https://api.github.com/users/jamtur01/orgs","repos_url":"https://api.github.com/users/jamtur01/repos","events_url":"https://api.github.com/users/jamtur01/events{/privacy}","received_events_url":"https://api.github.com/users/jamtur01/received_events","type":"User","site_admin":false},"created_at":"2017-12-26T13:20:40Z","updated_at":"2017-12-26T13:20:40Z","author_association":"CONTRIBUTOR","body":"@brian-brazil I note there are a lot of these. Would it be worth adding some variant of this to the issue template?"}}
{"actor":{"display_login":"jamtur01"}}