{"action":"created","issue":{"url":"https://api.github.com/repos/attic-labs/noms/issues/3123","repository_url":"https://api.github.com/repos/attic-labs/noms","labels_url":"https://api.github.com/repos/attic-labs/noms/issues/3123/labels{/name}","comments_url":"https://api.github.com/repos/attic-labs/noms/issues/3123/comments","events_url":"https://api.github.com/repos/attic-labs/noms/issues/3123/events","html_url":"https://github.com/attic-labs/noms/issues/3123","id":203752757,"number":3123,"title":"Removing the *hinting* system","user":{"login":"rafael-atticlabs","id":12848669,"avatar_url":"https://avatars.githubusercontent.com/u/12848669?v=3","gravatar_id":"","url":"https://api.github.com/users/rafael-atticlabs","html_url":"https://github.com/rafael-atticlabs","followers_url":"https://api.github.com/users/rafael-atticlabs/followers","following_url":"https://api.github.com/users/rafael-atticlabs/following{/other_user}","gists_url":"https://api.github.com/users/rafael-atticlabs/gists{/gist_id}","starred_url":"https://api.github.com/users/rafael-atticlabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rafael-atticlabs/subscriptions","organizations_url":"https://api.github.com/users/rafael-atticlabs/orgs","repos_url":"https://api.github.com/users/rafael-atticlabs/repos","events_url":"https://api.github.com/users/rafael-atticlabs/events{/privacy}","received_events_url":"https://api.github.com/users/rafael-atticlabs/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-01-27T22:09:15Z","updated_at":"2017-01-27T22:34:12Z","closed_at":null,"body":"Towards #3122, the noms wire protocol uses a system of hints to reduce the IO load on a server of validating that chunks written to the server contain valid references.\r\n\r\nThe \"hints\" are references to other chunks which the client knows to contain the same references (and already be committed to the server). Thus, if the server can find these references within contained chunks which have already been validated, it can assume they are correct.\r\n\r\nGenerally, this works relatively well, but has some problems:\r\n\r\n1) It's complex. The code within ValueStore which manages it has gotten more & more complex as noms has evolved. As is, it's pretty hard to reason about and we have comparatively low confidence that it's behavior is correct. Thankfully, incorrect behavior only results in greater IO on the server, not corruption.\r\n\r\n2) The RAM required to maintain this hinting information isn't bounded and long-lived processes which do multiple commits continue to accumulate hinting information with no clear way to know how to expire some of it.\r\n\r\n2) We had to relax the server to not *require* all hints necessary to write, and thus it's possible that a lazy client could bypass all hints and force the server to do a large amount of IO. The only obvious recourse here is to create heuristics around how much IO is too much, but those would be fairly arbitrary.\r\n\r\n3) We currently have a *write once at the end* write model for noms because writing in batches and committing at some point in the future doesn't have an obvious way to fit into this model. As a result, we always write locally to disk first. In practice, this means that alot of jobs are forced needlessly to write to disk, and it means that we can parallelize job work and shipping data to the server."},"comment":{"url":"https://api.github.com/repos/attic-labs/noms/issues/comments/275793404","html_url":"https://github.com/attic-labs/noms/issues/3123#issuecomment-275793404","issue_url":"https://api.github.com/repos/attic-labs/noms/issues/3123","id":275793404,"user":{"login":"rafael-atticlabs","id":12848669,"avatar_url":"https://avatars.githubusercontent.com/u/12848669?v=3","gravatar_id":"","url":"https://api.github.com/users/rafael-atticlabs","html_url":"https://github.com/rafael-atticlabs","followers_url":"https://api.github.com/users/rafael-atticlabs/followers","following_url":"https://api.github.com/users/rafael-atticlabs/following{/other_user}","gists_url":"https://api.github.com/users/rafael-atticlabs/gists{/gist_id}","starred_url":"https://api.github.com/users/rafael-atticlabs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rafael-atticlabs/subscriptions","organizations_url":"https://api.github.com/users/rafael-atticlabs/orgs","repos_url":"https://api.github.com/users/rafael-atticlabs/repos","events_url":"https://api.github.com/users/rafael-atticlabs/events{/privacy}","received_events_url":"https://api.github.com/users/rafael-atticlabs/received_events","type":"User","site_admin":false},"created_at":"2017-01-27T22:34:12Z","updated_at":"2017-01-27T22:34:12Z","body":"FYI, @arv "}}
{"actor":{"display_login":"rafael-atticlabs"}}