{"action":"created","issue":{"url":"https://api.github.com/repos/ceph/ceph/issues/14211","repository_url":"https://api.github.com/repos/ceph/ceph","labels_url":"https://api.github.com/repos/ceph/ceph/issues/14211/labels{/name}","comments_url":"https://api.github.com/repos/ceph/ceph/issues/14211/comments","events_url":"https://api.github.com/repos/ceph/ceph/issues/14211/events","html_url":"https://github.com/ceph/ceph/pull/14211","id":217730227,"number":14211,"title":"[DNM] bluestore: adaptive throttle tied to latency","user":{"login":"ivancich","id":742788,"avatar_url":"https://avatars2.githubusercontent.com/u/742788?v=3","gravatar_id":"","url":"https://api.github.com/users/ivancich","html_url":"https://github.com/ivancich","followers_url":"https://api.github.com/users/ivancich/followers","following_url":"https://api.github.com/users/ivancich/following{/other_user}","gists_url":"https://api.github.com/users/ivancich/gists{/gist_id}","starred_url":"https://api.github.com/users/ivancich/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ivancich/subscriptions","organizations_url":"https://api.github.com/users/ivancich/orgs","repos_url":"https://api.github.com/users/ivancich/repos","events_url":"https://api.github.com/users/ivancich/events{/privacy}","received_events_url":"https://api.github.com/users/ivancich/received_events","type":"User","site_admin":false},"labels":[{"id":385981601,"url":"https://api.github.com/repos/ceph/ceph/labels/bluestore","name":"bluestore","color":"1d76db","default":false},{"id":204137300,"url":"https://api.github.com/repos/ceph/ceph/labels/performance","name":"performance","color":"bfdadc","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-03-29T00:21:24Z","updated_at":"2017-04-12T13:23:48Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/ceph/ceph/pulls/14211","html_url":"https://github.com/ceph/ceph/pull/14211","diff_url":"https://github.com/ceph/ceph/pull/14211.diff","patch_url":"https://github.com/ceph/ceph/pull/14211.patch"},"body":"To turn on feature, you'll need to add to your ceph.conf file:\r\n```\r\nbluestore latency managed throttle = true\r\n```\r\n\r\nOther configuration options include bluestore_throttle_latency_min, bluestore_throttle_latency_max, and bluestore_throttle_latency_window_msec.\r\n\r\nThe general strategy is to look at the commit latency every `bluestore_throttle_latency_window_msec` milliseconds. If within five samples three show samples above `bluestore_throttle_latency_max`, increase the throttle. Likewise if there are three samples below `bluestore_throttle_latency_min`, decrease the throttle. The throttle is increased/decreased to values that are powers of the square root of 2 (e.g., 8192, 11585, 16384, 23170, 32768, 46341, ...).\r\n\r\nTo watch the perf monitor latencies run from your build dir (e.g., for osd 0):\r\n```\r\nbin/ceph daemonperf osd.0\r\n```\r\n\r\nTo watch measurements of latency and changes to throttle (e.g., for osd 0):\r\n```\r\ntail -f out/osd.0.log | grep throttle_mgr\r\n```\r\n\r\nCommands I used to generate requests:\r\n```\r\nbin/rados mkpool testp\r\nbin/rados -b 4096 -o 4096 -t 8 --pool=testp bench 600 write\r\n```\r\n\r\nWhat I found is that for periods the latency (**clat** column on `daemonperf` command) was stable. But occasionally it would also spike for a while and even if the throttle was expanded, it would not bring down the latency.\r\n\r\nI suspect that the throttle should be reduced while the submit latency (or throttle latency) is 0. I will work on that code next."},"comment":{"url":"https://api.github.com/repos/ceph/ceph/issues/comments/293574100","html_url":"https://github.com/ceph/ceph/pull/14211#issuecomment-293574100","issue_url":"https://api.github.com/repos/ceph/ceph/issues/14211","id":293574100,"user":{"login":"markhpc","id":1286295,"avatar_url":"https://avatars3.githubusercontent.com/u/1286295?v=3","gravatar_id":"","url":"https://api.github.com/users/markhpc","html_url":"https://github.com/markhpc","followers_url":"https://api.github.com/users/markhpc/followers","following_url":"https://api.github.com/users/markhpc/following{/other_user}","gists_url":"https://api.github.com/users/markhpc/gists{/gist_id}","starred_url":"https://api.github.com/users/markhpc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markhpc/subscriptions","organizations_url":"https://api.github.com/users/markhpc/orgs","repos_url":"https://api.github.com/users/markhpc/repos","events_url":"https://api.github.com/users/markhpc/events{/privacy}","received_events_url":"https://api.github.com/users/markhpc/received_events","type":"User","site_admin":false},"created_at":"2017-04-12T13:23:48Z","updated_at":"2017-04-12T13:23:48Z","body":"sounds good, hoping to start testing this today!"}}
{"actor":{"display_login":"markhpc"}}