{"action":"created","issue":{"url":"https://api.github.com/repos/greenplum-db/gpdb/issues/3982","repository_url":"https://api.github.com/repos/greenplum-db/gpdb","labels_url":"https://api.github.com/repos/greenplum-db/gpdb/issues/3982/labels{/name}","comments_url":"https://api.github.com/repos/greenplum-db/gpdb/issues/3982/comments","events_url":"https://api.github.com/repos/greenplum-db/gpdb/issues/3982/events","html_url":"https://github.com/greenplum-db/gpdb/pull/3982","id":277094861,"number":3982,"title":"pg_compression: add Zstandard","user":{"login":"leskin-in","id":19490926,"avatar_url":"https://avatars3.githubusercontent.com/u/19490926?v=4","gravatar_id":"","url":"https://api.github.com/users/leskin-in","html_url":"https://github.com/leskin-in","followers_url":"https://api.github.com/users/leskin-in/followers","following_url":"https://api.github.com/users/leskin-in/following{/other_user}","gists_url":"https://api.github.com/users/leskin-in/gists{/gist_id}","starred_url":"https://api.github.com/users/leskin-in/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leskin-in/subscriptions","organizations_url":"https://api.github.com/users/leskin-in/orgs","repos_url":"https://api.github.com/users/leskin-in/repos","events_url":"https://api.github.com/users/leskin-in/events{/privacy}","received_events_url":"https://api.github.com/users/leskin-in/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2017-11-27T17:04:51Z","updated_at":"2017-12-01T19:46:09Z","closed_at":"2017-12-01T19:41:06Z","author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/greenplum-db/gpdb/pulls/3982","html_url":"https://github.com/greenplum-db/gpdb/pull/3982","diff_url":"https://github.com/greenplum-db/gpdb/pull/3982.diff","patch_url":"https://github.com/greenplum-db/gpdb/pull/3982.patch"},"body":"As described in [this issue](https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fgreenplum-db%2Fgpdb%2Fissues%2F3155&sa=D&sntz=1&usg=AFQjCNHitXVIaj8vyBb2H32TkPAqUSSC2w) and in [GPDB mailing list topic](https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/XoYcF97qK-Y), there is a need of some new compression algorithm in GPDB.\r\n\r\nWe (@leskin-in @kapustor, @acmnu) have added integration with the [Zstandard](https://github.com/facebook/zstd) library.\r\n\r\nA simple test conducted on CentOS 7 one-core machine shows that `zstd` is approximately 4 times faster than `zlib`, while compressed table size is almost the same for them both. We used a compression level 9 and a raw (uncompressed) table size was 248Mb. Here is a script:\r\n```\r\npsql -d postgres -c 'CREATE TABLE sales_zstd (txn_id int, qty text) WITH (appendonly=true, compresstype=zstd, compresslevel=9) DISTRIBUTED BY (txn_id);'\r\npsql -d postgres -c 'CREATE TABLE sales_zlib (txn_id int, qty text) WITH (appendonly=true, compresstype=zlib, compresslevel=9) DISTRIBUTED BY (txn_id);'\r\npsql -d postgres -c 'CREATE TABLE sales_none (txn_id int, qty text) WITH (appendonly=true, compresstype=none) DISTRIBUTED BY (txn_id);'\r\n\r\ntime psql -d postgres -c 'insert into sales_zstd (select generate_series(1,10000000) as txn_id, generate_series(1,10000000)::text as qty);'\r\n# real\t0m19.479s\r\ntime psql -d postgres -c 'insert into sales_zlib (select generate_series(1,10000000) as txn_id, generate_series(1,10000000)::text as qty);'\r\n# real\t1m37.831s\r\ntime psql -d postgres -c 'insert into sales_none (select generate_series(1,10000000) as txn_id, generate_series(1,10000000)::text as qty);'\r\n# real\t0m18.173s\r\n\r\npsql -d postgres -c 'SELECT * FROM pg_size_pretty(pg_total_relation_size('sales_zstd'));'\r\n# 44 MB\r\npsql -d postgres -c 'SELECT * FROM pg_size_pretty(pg_total_relation_size('sales_zlib'));'\r\n# 52 MB\r\npsql -d postgres -c 'SELECT * FROM pg_size_pretty(pg_total_relation_size('sales_none'));'\r\n# 248 MB\r\n```\r\n\r\nHowever, there are a few issues with this solution:\r\n1. Zstandard supports 19 (safe, see http://facebook.github.io/zstd/zstd_manual.html#Chapter1) compression levels. As far as I understand, the current version of `src/backend/utils/datumstream.c` does not allow to use compression levels >= 10 for some reason\r\n2. It is unclear how to disable compilation and linking of the file `zstd_compression.c` if the flag `--with-zstd` was not passed to `configure` script (at the moment, `zstd` is disabled by default). The Make command for that is in `src/backend/catalog/Makefile`. This is a reason why the CI tests cannot be passed\r\n\r\nWe will be glad to receive feedback from the community. As for those issues, we are going to fix them in a few days."},"comment":{"url":"https://api.github.com/repos/greenplum-db/gpdb/issues/comments/348596451","html_url":"https://github.com/greenplum-db/gpdb/pull/3982#issuecomment-348596451","issue_url":"https://api.github.com/repos/greenplum-db/gpdb/issues/3982","id":348596451,"user":{"login":"ashwinstar","id":11396815,"avatar_url":"https://avatars2.githubusercontent.com/u/11396815?v=4","gravatar_id":"","url":"https://api.github.com/users/ashwinstar","html_url":"https://github.com/ashwinstar","followers_url":"https://api.github.com/users/ashwinstar/followers","following_url":"https://api.github.com/users/ashwinstar/following{/other_user}","gists_url":"https://api.github.com/users/ashwinstar/gists{/gist_id}","starred_url":"https://api.github.com/users/ashwinstar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ashwinstar/subscriptions","organizations_url":"https://api.github.com/users/ashwinstar/orgs","repos_url":"https://api.github.com/users/ashwinstar/repos","events_url":"https://api.github.com/users/ashwinstar/events{/privacy}","received_events_url":"https://api.github.com/users/ashwinstar/received_events","type":"User","site_admin":false},"created_at":"2017-12-01T19:46:09Z","updated_at":"2017-12-01T19:46:09Z","author_association":"MEMBER","body":"Don't see tests added for CO tables. Also, for CO tables good to evaluate if having zstd as block compression instead of current RLE + zlib is better."}}
{"actor":{"display_login":"ashwinstar"}}