{"action":"closed","issue":{"url":"https://api.github.com/repos/xavierleroy/camlzip/issues/1","repository_url":"https://api.github.com/repos/xavierleroy/camlzip","labels_url":"https://api.github.com/repos/xavierleroy/camlzip/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/xavierleroy/camlzip/issues/1/comments","events_url":"https://api.github.com/repos/xavierleroy/camlzip/issues/1/events","html_url":"https://github.com/xavierleroy/camlzip/issues/1","id":201268242,"number":1,"title":"Random read_entry \"decompression error\" exception with zlib 1.2.9+","user":{"login":"einars","id":38091,"avatar_url":"https://avatars.githubusercontent.com/u/38091?v=3","gravatar_id":"","url":"https://api.github.com/users/einars","html_url":"https://github.com/einars","followers_url":"https://api.github.com/users/einars/followers","following_url":"https://api.github.com/users/einars/following{/other_user}","gists_url":"https://api.github.com/users/einars/gists{/gist_id}","starred_url":"https://api.github.com/users/einars/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/einars/subscriptions","organizations_url":"https://api.github.com/users/einars/orgs","repos_url":"https://api.github.com/users/einars/repos","events_url":"https://api.github.com/users/einars/events{/privacy}","received_events_url":"https://api.github.com/users/einars/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2017-01-17T12:38:14Z","updated_at":"2017-01-23T09:44:20Z","closed_at":"2017-01-23T09:44:20Z","body":"Since the notorious zlib update, I've been having hard-to-pin-down somewhat reproducible issues when processing large zip files (700 mb) and using a lot of memory while processing. I can reproduce them all the time with my core+camlzip processor on of my huge file that I am not able to share publicly, but been unable to extract to a smaller, reproducible case (even adding a debug output may make the issue go away).\r\n\r\nThe problem manifests itself as suddenly being unable to unpack zip file — the Zlib.Error \"decompression error\" on random files gets thrown, whereas the zip file is itself perfectly fine.\r\n\r\nWhat I've been able to pinpoint so far is that it started occuring with a zlib commit https://github.com/madler/zlib/commit/b516b4bdd7c0c9f0858adfebf732089014f7b282 — Mark Adler added some sanity checks,\r\n\r\nThe exception gets thrown in camlzip_inflateEnd, as zlib returns error,\r\n\r\nthe reason for exception is that inflateStateCheck checks the stream structure, and inside there's a \"state\" substructure that has a reverse pointer to the stream. The new check verifies if these streams are actually equal (https://github.com/madler/zlib/commit/b516b4bdd7c0c9f0858adfebf732089014f7b282#diff-327188edf18799ffbb5a51cc69f797e8R113) — and suddenly, they are not anymore. \r\n\r\nHere's my zlib debug info,\r\n\r\n\r\n    # let lines = Zip.read_entry z entry |> String.split ~on:'\\n' in ...\r\n    inflatestatecheck failed\r\n    strm   0x7f7c7cd8f7b0\r\n    state  0x2a35070\r\n    state->strm 0x7f7c887b5100\r\n    state->mode 16203 (distext)\r\n    Uncaught exception:\r\n    \r\n    Zip.Error(\"weather.zip\", \"wlask.min\", \"decompression error\")\r\n    \r\n    Called from file \"src/exn.ml\", line 90, characters 6-10\r\n\r\nI suppose that probably the garbage collector or something sometimes moves things around and the structure turns invalid, or something — — — any ideas?\r\n\r\n(Up-to-date 64-bit archlinux, ocaml 4.04.0 and all via opam)"}}
{"actor":{"display_login":"xavierleroy"}}