{"action":"closed","issue":{"url":"https://api.github.com/repos/BVLC/caffe/issues/1533","repository_url":"https://api.github.com/repos/BVLC/caffe","labels_url":"https://api.github.com/repos/BVLC/caffe/issues/1533/labels{/name}","comments_url":"https://api.github.com/repos/BVLC/caffe/issues/1533/comments","events_url":"https://api.github.com/repos/BVLC/caffe/issues/1533/events","html_url":"https://github.com/BVLC/caffe/issues/1533","id":51081295,"number":1533,"title":"Compress the training data with Google Snappy for higher IO throughput","user":{"login":"futurely","id":9004594,"avatar_url":"https://avatars0.githubusercontent.com/u/9004594?v=3","gravatar_id":"","url":"https://api.github.com/users/futurely","html_url":"https://github.com/futurely","followers_url":"https://api.github.com/users/futurely/followers","following_url":"https://api.github.com/users/futurely/following{/other_user}","gists_url":"https://api.github.com/users/futurely/gists{/gist_id}","starred_url":"https://api.github.com/users/futurely/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/futurely/subscriptions","organizations_url":"https://api.github.com/users/futurely/orgs","repos_url":"https://api.github.com/users/futurely/repos","events_url":"https://api.github.com/users/futurely/events{/privacy}","received_events_url":"https://api.github.com/users/futurely/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2014-12-05T09:59:20Z","updated_at":"2017-03-23T07:27:04Z","closed_at":"2017-03-23T07:27:04Z","body":"When [computation is faster than data IO](https://github.com/BVLC/caffe/pull/1148), the device utitlization  is reduced. To reduce the throughput gap, the data is usually compressed in many distributed computation or storage framework such as [Hadoop](https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/W265aa64a4f21_43ee_b236_c42a1c875961/page/Hadoop%20compression%20formats), [HBase](http://blog.erdemagaoglu.com/post/4605524309/lzo-vs-snappy-vs-lzf-vs-zlib-a-comparison-of), [Hive](http://hortonworks.com/blog/orcfile-in-hdp-2-better-compression-better-performance/) and [Kafka](http://nehanarkhede.com/2013/03/28/compression-in-kafka-gzip-or-snappy/). Among many compressing libraries available for this purpose, [Google Snappy](https://github.com/google/snappy) is a very widely used one. It supports [many languages](https://code.google.com/p/snappy/) and strikes a [good balance between compression ratio and speed](https://github.com/ning/jvm-compressor-benchmark/wiki#results).\n"}}
{"actor":{"display_login":"shelhamer"}}