{"push_id":2204228778,"size":2,"distinct_size":1,"ref":"refs/heads/dev","head":"aa0c09bdc93d8f9382ccfce49eb83ba5d4db84b4","before":"569e06b91e1cdbf1a6b275cf18a3d0667efbd1aa","commits":[{"sha":"5c2f2ebfdb8fc8e074269ad4f85ac79267cc36fe","author":{"name":"Yann Collet","email":"ce291ff8e77f3870bf0f4631ecb1c40ac0f8bbc6@fb.com"},"message":"zstdmt via compress_generic: reduce opportunity to free/create mtctx\n\n`zstreamtest --newapi` (and `--opaqueapi`) create and destroy way too many threads\nresulting in failure of tsan tests,\nand potentially connected to the qemu flaky tests.\n\nThis is because, at each test, the nb of threads can be changed (random).\n\nThe `--no-big-tests` directive reduce this choice to 1/2 threads,\nin order to limit memory usage, especially for qemu and 32-bits builds.\nUnfortunately, swapping between 1 and 2 threads is enough to constantly create/destroy new mtctx.\n\nThis patch takes advantage of the following property :\nvia compress_generic, no internal mtctx is needed for nbThreads < 2.\nAs a consequence, when nbThreads == 2, the currently active mtctx is necessarily good.\n\nThis dramatically reduces the nb of thread creations when invoking `zstreamtest --newapi --no-big-tests`\n(only when parent cctx itself is created, which is randomized to 1/256 tests).\n\nExpected outcome :\n- at a minimum : tsan tests shall now work continuously without exploding the thread counter\n- at best : flaky qemu tests on `zstreamtest --newapi --no-big-tests` may stop being flaky, due to less stress from constant thread creation/destruction\n\nReal world impact :\nminimal, I don't expect users to constantly change `nbThreads` between each invocation.\nIf `nbThreads` remains stable, existing implementation re-uses existing mtctx.\n\nAlso : `zstreamtest --newapi` but without `--no-big-tests` doesn't benefit as much,\nsince this test can select a random `nbThreads` value between 1 and 4.\nThe current patch only reduces opportunity to free/create mtctx (for example : 2->1->2 doesn't need a new mtctx)\nbut doesn't completely eliminate it, since `nbThreads` can still change between 2/3/4.\nA more complete solution could be to only use 2 out of 4 allocated threads, thus keeping the pool at a constant size.\nThis would require a larger change to `POOL_*` api though.","distinct":false,"url":"https://api.github.com/repos/facebook/zstd/commits/5c2f2ebfdb8fc8e074269ad4f85ac79267cc36fe"},{"sha":"aa0c09bdc93d8f9382ccfce49eb83ba5d4db84b4","author":{"name":"Yann Collet","email":"2a64e10dfd58391a45c6779589bcfc1d5d451569@users.noreply.github.com"},"message":"Merge pull request #957 from facebook/nbThreads\n\nzstdmt via compress_generic: reduce opportunity to free/create mtctx","distinct":true,"url":"https://api.github.com/repos/facebook/zstd/commits/aa0c09bdc93d8f9382ccfce49eb83ba5d4db84b4"}]}
{"actor":{"display_login":"Cyan4973"}}