{"action":"closed","issue":{"url":"https://api.github.com/repos/rust-lang/rust/issues/38114","repository_url":"https://api.github.com/repos/rust-lang/rust","labels_url":"https://api.github.com/repos/rust-lang/rust/issues/38114/labels{/name}","comments_url":"https://api.github.com/repos/rust-lang/rust/issues/38114/comments","events_url":"https://api.github.com/repos/rust-lang/rust/issues/38114/events","html_url":"https://github.com/rust-lang/rust/issues/38114","id":192912952,"number":38114,"title":"revise metadata hashing to hash the contents of the metadata, not its input dependencies","user":{"login":"nikomatsakis","id":155238,"avatar_url":"https://avatars3.githubusercontent.com/u/155238?v=3","gravatar_id":"","url":"https://api.github.com/users/nikomatsakis","html_url":"https://github.com/nikomatsakis","followers_url":"https://api.github.com/users/nikomatsakis/followers","following_url":"https://api.github.com/users/nikomatsakis/following{/other_user}","gists_url":"https://api.github.com/users/nikomatsakis/gists{/gist_id}","starred_url":"https://api.github.com/users/nikomatsakis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nikomatsakis/subscriptions","organizations_url":"https://api.github.com/users/nikomatsakis/orgs","repos_url":"https://api.github.com/users/nikomatsakis/repos","events_url":"https://api.github.com/users/nikomatsakis/events{/privacy}","received_events_url":"https://api.github.com/users/nikomatsakis/received_events","type":"User","site_admin":false},"labels":[{"id":307747675,"url":"https://api.github.com/repos/rust-lang/rust/labels/A-incr-comp","name":"A-incr-comp","color":"d7e012","default":false},{"id":211668100,"url":"https://api.github.com/repos/rust-lang/rust/labels/T-compiler","name":"T-compiler","color":"bfd4f2","default":false}],"state":"closed","locked":false,"assignee":{"login":"michaelwoerister","id":1825894,"avatar_url":"https://avatars1.githubusercontent.com/u/1825894?v=3","gravatar_id":"","url":"https://api.github.com/users/michaelwoerister","html_url":"https://github.com/michaelwoerister","followers_url":"https://api.github.com/users/michaelwoerister/followers","following_url":"https://api.github.com/users/michaelwoerister/following{/other_user}","gists_url":"https://api.github.com/users/michaelwoerister/gists{/gist_id}","starred_url":"https://api.github.com/users/michaelwoerister/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaelwoerister/subscriptions","organizations_url":"https://api.github.com/users/michaelwoerister/orgs","repos_url":"https://api.github.com/users/michaelwoerister/repos","events_url":"https://api.github.com/users/michaelwoerister/events{/privacy}","received_events_url":"https://api.github.com/users/michaelwoerister/received_events","type":"User","site_admin":false},"assignees":[{"login":"michaelwoerister","id":1825894,"avatar_url":"https://avatars1.githubusercontent.com/u/1825894?v=3","gravatar_id":"","url":"https://api.github.com/users/michaelwoerister","html_url":"https://github.com/michaelwoerister","followers_url":"https://api.github.com/users/michaelwoerister/followers","following_url":"https://api.github.com/users/michaelwoerister/following{/other_user}","gists_url":"https://api.github.com/users/michaelwoerister/gists{/gist_id}","starred_url":"https://api.github.com/users/michaelwoerister/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaelwoerister/subscriptions","organizations_url":"https://api.github.com/users/michaelwoerister/orgs","repos_url":"https://api.github.com/users/michaelwoerister/repos","events_url":"https://api.github.com/users/michaelwoerister/events{/privacy}","received_events_url":"https://api.github.com/users/michaelwoerister/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/rust-lang/rust/milestones/31","html_url":"https://github.com/rust-lang/rust/milestone/31","labels_url":"https://api.github.com/repos/rust-lang/rust/milestones/31/labels","id":2090457,"number":31,"title":"Incremental compilation across crates","description":"After [the incremental beta](https://github.com/rust-lang/rust/milestone/30), another important step is to have the ability to reuse code across crates. In fact, the basic infrastructure for this exists already, but we need to make test cases representing the scenarios we expect and then fix the bugs that prevent those testcases from achieving the reuse we expect.","creator":{"login":"nikomatsakis","id":155238,"avatar_url":"https://avatars3.githubusercontent.com/u/155238?v=3","gravatar_id":"","url":"https://api.github.com/users/nikomatsakis","html_url":"https://github.com/nikomatsakis","followers_url":"https://api.github.com/users/nikomatsakis/followers","following_url":"https://api.github.com/users/nikomatsakis/following{/other_user}","gists_url":"https://api.github.com/users/nikomatsakis/gists{/gist_id}","starred_url":"https://api.github.com/users/nikomatsakis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nikomatsakis/subscriptions","organizations_url":"https://api.github.com/users/nikomatsakis/orgs","repos_url":"https://api.github.com/users/nikomatsakis/repos","events_url":"https://api.github.com/users/nikomatsakis/events{/privacy}","received_events_url":"https://api.github.com/users/nikomatsakis/received_events","type":"User","site_admin":false},"open_issues":1,"closed_issues":3,"state":"closed","created_at":"2016-10-24T19:51:37Z","updated_at":"2017-05-29T08:53:04Z","due_on":null,"closed_at":"2017-03-06T18:08:51Z"},"comments":5,"created_at":"2016-12-01T17:35:07Z","updated_at":"2017-05-29T08:53:04Z","closed_at":"2017-05-29T08:53:04Z","body":"NB: Part of the [roadmap issue on incremental compilation](https://github.com/rust-lang/rust-roadmap/issues/4).\r\n\r\n**Newer writeup:**\r\n\r\nThe plan is to change how we hash metadata. Instead of hashing the hashes of the predecessors of the metadata node, we ought to hash the metadata content *itself*. This would be far more robust in general.\r\n\r\n**Older writeup:**\r\n\r\nWhile playing around with an incremental-enabled rustc, I noticed the following pattern.\r\n\r\n1. We do a make clean.\r\n2. The first build gives a metadata hash H1 for some item X.\r\n3. Touch some file but make no other changes.\r\n4. The next build reuses a lot of stuff -- but the hash for item X changes to H2! This triggers invalidations in downstream crates.\r\n\r\nThis can be produced with this test case:\r\n\r\n```rust\r\n#![crate_type = \"rlib\"]\r\n#![feature(rustc_attrs)]\r\n\r\nuse std::str;\r\n\r\npub const MAX_BASE: u64 = 64;\r\n\r\nconst BASE_64: &'static [u8; MAX_BASE as usize] =\r\n    b\"0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ@$\";\r\n\r\n#[inline]\r\npub fn push_str(mut n: u64, base: u64, output: &mut String) {\r\n    debug_assert!(base >= 2 && base <= MAX_BASE);\r\n    let mut s = [0u8; 64];\r\n    let mut index = 0;\r\n\r\n    loop {\r\n        s[index] = BASE_64[(n % base) as usize];\r\n        index += 1;\r\n        n /= base;\r\n\r\n        if n == 0 {\r\n            break;\r\n        }\r\n    }\r\n    &mut s[0..index].reverse();\r\n    output.push_str(str::from_utf8(&s[0..index]).unwrap());\r\n}\r\n\r\nfn foo() {\r\n    unsafe {\r\n        let x: &mut [u32; 64] = &mut [0; 64];\r\n        ::std::ptr::drop_in_place(x.get_unchecked_mut(0));\r\n    }\r\n}\r\n\r\nfn main() { }\r\n```\r\n\r\nYou can observe this if you run twice with the `-Z incremental-dump-hash` option added by https://github.com/rust-lang/rust/pull/38113:\r\n\r\n```\r\nlunch-box. rm -rf ~/tmp/incr/*\r\nlunch-box. rustc ~/tmp/push_str_regr.rs -Z incremental-info -Z incremental=~/tmp/incr -Z incremental-dump-hash 2>&1 | grep 'metadata hash for.*push_str\\[0\\] } is\\|re-using'\r\nmetadata hash for DefId { krate: CrateNum(0), node: DefIndex(8) => push_str_regr/4089d7c8b778d88cec885baf7b69e6df::push_str[0] } is 37::48::148::96::189::58::104::10::224::200::97::217::109::183::176::21\\\r\n1\r\nincremental: re-using 0 out of 1 modules\r\nlunch-box. rustc ~/tmp/push_str_regr.rs -Z incremental-info -Z incremental=~/tmp/incr -Z incremental-dump-hash 2>&1 | grep 'metadata hash for.*push_str\\[0\\] } is\\|re-using'\r\nmetadata hash for DefId { krate: CrateNum(0), node: DefIndex(8) => push_str_regr/4089d7c8b778d88cec885baf7b69e6df::push_str[0] } is 127::14::237::97::187::11::36::81::62::236::134::168::133::163::8::179\r\nincremental: re-using 1 out of 1 modules\r\n```\r\n\r\nThe reason for this problem is that, in the first run, we have to predefine various symbols, which causes us to do associated type normalization, which adds edges to some `TraitSelect` nodes. However, in the second run, because we are able to reuse, we don't do those normalizations, and thus we have fewer edges in the graph. \r\n\r\nThis interacts poorly with our metadata hashing scheme, because the MIR for `push_str` depends on `TraitSelect`. Thus when we compute the hash of its metadata, we find the (transitive) sources for that metadata, which means all predecessors of `TraitSelect`. In the first round, this is a different set from the second round. You can see this by diffing the output a bit more:\r\n\r\n```\r\nlunch-box. rm -rf ~/tmp/incr/*\r\nlunch-box. rustc ~/tmp/push_str_regr.rs -Z incremental-info -Z incremental=~/tmp/incr -Z incremental-dump-hash 2>&1 | grep 'metadata hash for.*push_str\\[0\\] } depends on' | sort > deps.1\r\nlunch-box. rustc ~/tmp/push_str_regr.rs -Z incremental-info -Z incremental=~/tmp/incr -Z incremental-dump-hash 2>&1 | grep 'metadata hash for.*push_str\\[0\\] } depends on' | sort > deps.2\r\nlunch-box. diff deps.1 deps.2\r\n132d131\r\n< metadata hash for DefId { krate: CrateNum(0), node: DefIndex(8) => push_str_regr/4089d7c8b778d88cec885baf7b69e6df::push_str[0] } depends on MetaData(DefId { krate: CrateNum(2), node: DefIndex(5924) => \\\r\ncore/e5e2346e098515dd30a7b410581e03fa::slice[0]::{{impl}}[3]::Output[0] }) with hash 147::245::50::211::200::154::244::7::0::0::0::0::0::0::0::0\r\n```\r\n\r\nIn my opinion, the best fix for this is probably to change how we hash metadata. Instead of hashing the hashes of the predecessors of the metadata node, we ought to hash the metadata content *itself*. This would be far more robust in general.\r\n\r\ncc @michaelwoerister "}}
{"actor":{"display_login":"michaelwoerister"}}