{"action":"created","issue":{"url":"https://api.github.com/repos/coreos/bugs/issues/1811","repository_url":"https://api.github.com/repos/coreos/bugs","labels_url":"https://api.github.com/repos/coreos/bugs/issues/1811/labels{/name}","comments_url":"https://api.github.com/repos/coreos/bugs/issues/1811/comments","events_url":"https://api.github.com/repos/coreos/bugs/issues/1811/events","html_url":"https://github.com/coreos/bugs/issues/1811","id":208258113,"number":1811,"title":"\"Error loading buddy information\"; root filesystem remounted read-only","user":{"login":"bgilbert","id":361374,"avatar_url":"https://avatars.githubusercontent.com/u/361374?v=3","gravatar_id":"","url":"https://api.github.com/users/bgilbert","html_url":"https://github.com/bgilbert","followers_url":"https://api.github.com/users/bgilbert/followers","following_url":"https://api.github.com/users/bgilbert/following{/other_user}","gists_url":"https://api.github.com/users/bgilbert/gists{/gist_id}","starred_url":"https://api.github.com/users/bgilbert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgilbert/subscriptions","organizations_url":"https://api.github.com/users/bgilbert/orgs","repos_url":"https://api.github.com/users/bgilbert/repos","events_url":"https://api.github.com/users/bgilbert/events{/privacy}","received_events_url":"https://api.github.com/users/bgilbert/received_events","type":"User","site_admin":false},"labels":[{"id":418444453,"url":"https://api.github.com/repos/coreos/bugs/labels/area/stability","name":"area/stability","color":"c2e0c6","default":false},{"id":269192631,"url":"https://api.github.com/repos/coreos/bugs/labels/component/kernel","name":"component/kernel","color":"0052cc","default":false},{"id":91823380,"url":"https://api.github.com/repos/coreos/bugs/labels/kind/bug","name":"kind/bug","color":"DF0101","default":false},{"id":264376389,"url":"https://api.github.com/repos/coreos/bugs/labels/kind/support","name":"kind/support","color":"eb6420","default":false},{"id":275179790,"url":"https://api.github.com/repos/coreos/bugs/labels/team/os","name":"team/os","color":"ffffff","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2017-02-16T21:51:24Z","updated_at":"2017-02-16T21:53:58Z","closed_at":null,"body":"See kernel logs below.  Prometheus, while generating high I/O load, hits the container memcg limit and is OOM killed.  ext4 immediately reports \"Error loading buddy information\" as a result of a failed `ext4_mb_load_buddy()` call from `ext4_discard_preallocations()`, presumably running due to a Prometheus file descriptor being closed.  That report is made via `ext4_error()`, causing a journal abort and read-only remount of the root FS.  I haven't done an exhaustive audit, but it doesn't appear that any allocations under `ext4_mb_load_buddy()` should be accountable to the memcg.\r\n\r\nThis has happened on 4.7.3-coreos-r2 and 4.8.15-coreos.  The root filesystem is in EBS.  The problem is sporadic but recurring."},"comment":{"url":"https://api.github.com/repos/coreos/bugs/issues/comments/280473686","html_url":"https://github.com/coreos/bugs/issues/1811#issuecomment-280473686","issue_url":"https://api.github.com/repos/coreos/bugs/issues/1811","id":280473686,"user":{"login":"bgilbert","id":361374,"avatar_url":"https://avatars.githubusercontent.com/u/361374?v=3","gravatar_id":"","url":"https://api.github.com/users/bgilbert","html_url":"https://github.com/bgilbert","followers_url":"https://api.github.com/users/bgilbert/followers","following_url":"https://api.github.com/users/bgilbert/following{/other_user}","gists_url":"https://api.github.com/users/bgilbert/gists{/gist_id}","starred_url":"https://api.github.com/users/bgilbert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bgilbert/subscriptions","organizations_url":"https://api.github.com/users/bgilbert/orgs","repos_url":"https://api.github.com/users/bgilbert/repos","events_url":"https://api.github.com/users/bgilbert/events{/privacy}","received_events_url":"https://api.github.com/users/bgilbert/received_events","type":"User","site_admin":false},"created_at":"2017-02-16T21:53:58Z","updated_at":"2017-02-16T21:53:58Z","body":"```\r\nprometheus invoked oom-killer: gfp_mask=0x24000c0(GFP_KERNEL), order=0, oom_score_adj=997\r\nprometheus cpuset=[...] mems_allowed=0\r\nCPU: 1 PID: 20930 Comm: prometheus Tainted: G        W       4.8.15-coreos #1\r\nHardware name: Xen HVM domU, BIOS 4.2.amazon 11/11/2016\r\nEXT4-fs error (device xvda9): ext4_discard_preallocations:4031: comm prometheus: Error loading buddy information for 397\r\nAborting journal on device xvda9-8.\r\nEXT4-fs (xvda9): Remounting filesystem read-only\r\n 0000000000000286 0000000013548d17 ffff953846b9bc60 ffffffffb83157f3\r\n ffff953846b9bd40 ffff9537fffe1d40 ffff953846b9bcc8 ffffffffb81fc36e\r\n 0000000000000002 ffffffffb817d8eb ffff953904fc57c0 0000000000000001\r\nCall Trace:\r\n [<ffffffffb83157f3>] dump_stack+0x63/0x90\r\n [<ffffffffb81fc36e>] dump_header+0x5d/0x1e1\r\n [<ffffffffb817d8eb>] ? find_lock_task_mm+0x3b/0x80\r\n [<ffffffffb817e515>] oom_kill_process+0x225/0x3f0\r\n [<ffffffffb81f38e5>] ? mem_cgroup_iter+0x1a5/0x310\r\n [<ffffffffb81f5bbc>] mem_cgroup_out_of_memory+0x2bc/0x300\r\n [<ffffffffb81f6aa5>] mem_cgroup_oom_synchronize+0x335/0x340\r\n [<ffffffffb81f14f0>] ? high_work_func+0x20/0x20\r\n [<ffffffffb817eb9c>] pagefault_out_of_memory+0x4c/0xc0\r\n [<ffffffffb8067ff1>] mm_fault_error+0x91/0x190\r\n [<ffffffffb80688c6>] __do_page_fault+0x476/0x4a0\r\n [<ffffffffb8068912>] do_page_fault+0x22/0x30\r\n [<ffffffffb85c0af8>] page_fault+0x28/0x30\r\nTask in /docker/[...] killed as a result of limit of /docker/[...]\r\nmemory: usage 3072000kB, limit 3072000kB, failcnt 164128\r\nmemory+swap: usage 3072000kB, limit 9007199254740988kB, failcnt 0\r\nkmem: usage 8168kB, limit 9007199254740988kB, failcnt 0\r\nMemory cgroup stats for /docker/[...]: cache:4KB rss:3063828KB rss_huge:514048KB mapped_file:0KB dirty:0KB writeback:0KB swap:0KB inactive_anon:0KB active_anon:3063828KB inactive_file:0KB active_file:0KB unevictable:0KB\r\n[ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name\r\n[20889]     0 20889   777317   768312    1519       8        0           997 prometheus\r\nMemory cgroup out of memory: Kill process 20889 (prometheus) score 1999 or sacrifice child\r\nKilled process 20889 (prometheus) total-vm:3109268kB, anon-rss:3062540kB, file-rss:10708kB, shmem-rss:0kB\r\n```"}}
{"actor":{"display_login":"bgilbert"}}