{"push_id":1667056430,"size":4,"distinct_size":4,"ref":"refs/heads/master","head":"87e5f8a2cba697864fcfdc2486cb635da569ede2","before":"06d5667ea8b03c60a0ece256c7433f2eddf45bd0","commits":[{"sha":"c8c2695e68d5eb8ec41e1f4b6a5eb7cda2da8e1b","author":{"name":"Alexey Tourbin","email":"ef549fe41dc6847b7f799182f935d49a1d6bbd7a@gmail.com"},"message":"genpkglist.cc: reframed in terms of package groups\n\nA group is one or more headers, grouped by their source rpm, furthermore\nin the right output order (adjacent, sorted by rpm filename), compressed\nin a single chunk.  The overall genpkglist algorithm is to load the groups,\nsort them by source rpm, and finally to spew them out.\n\nThus this change eliminates the second pass: headers are loaded only\nonce and stay compressed in the memory.  Memory usage, even in the worst\ncase, is surprisingly low:\n\n$ time ./genpkglist --changelog-since 1 --bloat --cachedir ~/.cache /klad/Sisyphus/x86_64 classic\nUSER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND\nat       14005  175  1.0 139472 87848 pts/1    S+   03:00   0:01 ./genpkglist\n\nThat's because the headers are not original, they are already heavily\nstripped down (e.g. file digests are removed), except that they may be\nbloated.  But lz4 is good at compressing file lists and changelogs.\n\nNow, without --bloat or --no-scan option, this change results in double\ncompression: bloated headers are compressed before final grouping by\nsource rpm; they are then uncompressed, file lists are stripped,\nthen compressed again and spewed out.  This seems to work somewhat\nslower than just the second pass with no compression and with no\nheaders retained in the memory.\n\n$ time LD_LIBRARY_PATH=~tmp ~tmp/genpkglist  --cachedir ~/.cache /klad/Sisyphus/x86_64 classic\n2.566 total\n$ time ./genpkglist --cachedir ~/.cache /klad/Sisyphus/x86_64 classic\n2.802 total\n\nThe comparison is actually not fair, because ~tmp/genpkglist is\nfrom apt-repo-tools-0.6.0.19-alt1.1.1.x86_64.rpm and is linked with\nlibrpm-4.0.4.so.  And new librpm 4.13 is anything but faster.\n\nMaybe some day I'll revisit the issue.  For now, the notion of loading\nthe groups is more important.\n\nThe output is byte-to-byte idential to an older rpm40-based version:\n\n$ ./genpkglist --cachedir ~/.cache /klad/Sisyphus/x86_64 classic\n$ lz4 -d </klad/Sisyphus/x86_64/base/pkglist.classic.lz4 |md5sum\na03912a3bfc8f8cbbdce3edbe0b0ab6d  -\n\n$ LD_LIBRARY_PATH=~tmp ~tmp/genpkglist --cachedir ~/.cache /klad/Sisyphus/x86_64 classic\n$ md5sum /klad/Sisyphus/x86_64/base/pkglist.classic\na03912a3bfc8f8cbbdce3edbe0b0ab6d  /klad/Sisyphus/x86_64/base/pkglist.classic","distinct":true,"url":"https://api.github.com/repos/svpv/apt-repo-tools/commits/c8c2695e68d5eb8ec41e1f4b6a5eb7cda2da8e1b"},{"sha":"c85cbf25434a52b5596575f84b82b6e4dba11024","author":{"name":"Alexey Tourbin","email":"ef549fe41dc6847b7f799182f935d49a1d6bbd7a@gmail.com"},"message":"genpkglist.cc: restored full second pass\n\nThis is to amend the previous commit.  Second loading works\nindeed slightly faster than double compression.\n\nWith previous commit:\n\n$ time ./genpkglist --cachedir ~/.cache /klad/Sisyphus/x86_64 classic\n2.844 total\n\nWith this commit:\n\n$ time ./genpkglist --cachedir ~/.cache /klad/Sisyphus/x86_64 classic\n2.546 total","distinct":true,"url":"https://api.github.com/repos/svpv/apt-repo-tools/commits/c85cbf25434a52b5596575f84b82b6e4dba11024"},{"sha":"ac43d2dc095a68c9da211635e6fccc896cbd44de","author":{"name":"Alexey Tourbin","email":"ef549fe41dc6847b7f799182f935d49a1d6bbd7a@gmail.com"},"message":"genpkglist.cc: implemented --prev-stdin option\n\nUnlike gensrclist, genpkglist cannot use a simple merge-like strategy\non stdin and dirEntries.  That's because genpkglist first groups (sorts)\npackages by src.rpm before spewing them out.  Nonetheless, an optimal\nstrategy which preserves grouping is still possible: two adjacent\npackages exposed on stdin can from a group (pertaining to the same\nsrc.rpm package) if they also exist and are furthermore adjacent among\nthe dirEntries.\n\nNote that only bloated pkglists can be reused as a cache.  In a\nstripped-down pkglist, headers are interdependent with each other\n(i.e. each header depends upon the list of useful files).\n\nThis change produces a byte-to-byte identical output.  In the\nfollowing example, I reuse p8 pkglist as a cache for Sisyphus pkglist.\nThe only restriction is that the same date should be used with the\n--changelog-since option.  (In the postprocess() routine, file lists\nare stripped, but changelogs are copied as is.  Another possibility\nwould be to pass genbasedir something like --changelog-since=tomorrow,\nwhich should keep only the last changelog entry.)\n\n$ ./genpkglist --bloat --changelog-since $(date -d 2009-01-01 +%s) /klad/p8/x86_64 classic\n\n$ ./genpkglist              --changelog-since $(date -d 2009-01-01 +%s) /klad/Sisyphus/x86_64 classic\n$ md5sum /klad/Sisyphus/x86_64/base/pkglist.classic.lz4\n88c367135d3952e74329a75c8e457b43  /klad/Sisyphus/x86_64/base/pkglist.classic.lz4\n\n$ lz4 -d </klad/p8/x86_64/base/pkglist.classic.lz4 |\n  ./genpkglist --prev-stdin --changelog-since $(date -d 2009-01-01 +%s) /klad/Sisyphus/x86_64 classic\n$ md5sum /klad/Sisyphus/x86_64/base/pkglist.classic.lz4\n88c367135d3952e74329a75c8e457b43  /klad/Sisyphus/x86_64/base/pkglist.classic.lz4\n\nThe following example further ensures that the caching actually works.\n\n$ strace -e trace=open ./genpkglist              --changelog-since $(date -d 2009-01-01 +%s) /klad/Sisyphus/x86_64 classic |&\n  grep 'open(\".*\\.rpm\"' |wc -l\n42284\n\n$ lz4 -d </klad/p8/x86_64/base/pkglist.classic.lz4 |\n  strace -e trace=open ./genpkglist --prev-stdin --changelog-since $(date -d 2009-01-01 +%s) /klad/Sisyphus/x86_64 classic |&\n  grep 'open(\".*\\.rpm\"' |wc -l\n18478","distinct":true,"url":"https://api.github.com/repos/svpv/apt-repo-tools/commits/ac43d2dc095a68c9da211635e6fccc896cbd44de"},{"sha":"87e5f8a2cba697864fcfdc2486cb635da569ede2","author":{"name":"Alexey Tourbin","email":"ef549fe41dc6847b7f799182f935d49a1d6bbd7a@gmail.com"},"message":"genpkglist.cc: implemented --bloater option\n\nWith the --bloater option, two files will be written: a stripped-down\npkglist.classic and a bloated pkglist.classic+bloat.  Note that the\nlatter can be used as an input to genpkglist with the --prev-stdin\noption.  This draws much closer the possibility of reusing previous\noutput as a cache.","distinct":true,"url":"https://api.github.com/repos/svpv/apt-repo-tools/commits/87e5f8a2cba697864fcfdc2486cb635da569ede2"}]}
{"actor":{"display_login":"svpv"}}