{"action":"created","issue":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/2554","repository_url":"https://api.github.com/repos/zfsonlinux/zfs","labels_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/2554/labels{/name}","comments_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/2554/comments","events_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/2554/events","html_url":"https://github.com/zfsonlinux/zfs/issues/2554","id":39040211,"number":2554,"title":"btrfs bedup equivalent","user":{"login":"bjquinn","id":8106438,"avatar_url":"https://avatars3.githubusercontent.com/u/8106438?v=4","gravatar_id":"","url":"https://api.github.com/users/bjquinn","html_url":"https://github.com/bjquinn","followers_url":"https://api.github.com/users/bjquinn/followers","following_url":"https://api.github.com/users/bjquinn/following{/other_user}","gists_url":"https://api.github.com/users/bjquinn/gists{/gist_id}","starred_url":"https://api.github.com/users/bjquinn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bjquinn/subscriptions","organizations_url":"https://api.github.com/users/bjquinn/orgs","repos_url":"https://api.github.com/users/bjquinn/repos","events_url":"https://api.github.com/users/bjquinn/events{/privacy}","received_events_url":"https://api.github.com/users/bjquinn/received_events","type":"User","site_admin":false},"labels":[{"id":40358,"url":"https://api.github.com/repos/zfsonlinux/zfs/labels/Feature","name":"Feature","color":"02d7e1","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2014-07-29T21:57:14Z","updated_at":"2017-07-28T14:37:32Z","closed_at":"2016-10-01T01:34:55Z","body":"ZFS dedup is painfully RAM intensive.  I'm wondering if there are some compromises to be made that might allow for some deduplication functionality without all of the overhead.  Btrfs has bedup which simply does a scan of the filesystem looking for files with identical hashes, then compares actual files for the hash matches, and if the files are truly identical, it submits the file to be deduplicated.  Not sure exactly how the internals of this works, but I assume it's a bit like how cp --reflink works.\n\nAnyway, this doesn't have that kind of RAM overhead, and uses a sqlite table to keep up with files it's already hashed.  It's file-level only, not block-level.  But it seems like this would be very useful for backups.  Yes, snapshots and clones are helpful, but not when someone renames a file, or worse, a large tree of files -- if someone simply renames \"PHOTO 2014\" to \"PHOTOS 2014\" you're going to store an entire additional copy of that entire folder tree, which in my case is sometimes 100s of GB of incompressible data, which is quite annoying.\n\nIs this something that's possible?  I'd be happy to do some work on this, although I'd need some direction.  I think I'd be reasonably capable on the userland side of the tool, but I wouldn't know where to start in terms of what hooks I'd need into ZoL or what changes might be required there.\n"},"comment":{"url":"https://api.github.com/repos/zfsonlinux/zfs/issues/comments/318669140","html_url":"https://github.com/zfsonlinux/zfs/issues/2554#issuecomment-318669140","issue_url":"https://api.github.com/repos/zfsonlinux/zfs/issues/2554","id":318669140,"user":{"login":"giox069","id":5477803,"avatar_url":"https://avatars0.githubusercontent.com/u/5477803?v=4","gravatar_id":"","url":"https://api.github.com/users/giox069","html_url":"https://github.com/giox069","followers_url":"https://api.github.com/users/giox069/followers","following_url":"https://api.github.com/users/giox069/following{/other_user}","gists_url":"https://api.github.com/users/giox069/gists{/gist_id}","starred_url":"https://api.github.com/users/giox069/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/giox069/subscriptions","organizations_url":"https://api.github.com/users/giox069/orgs","repos_url":"https://api.github.com/users/giox069/repos","events_url":"https://api.github.com/users/giox069/events{/privacy}","received_events_url":"https://api.github.com/users/giox069/received_events","type":"User","site_admin":false},"created_at":"2017-07-28T14:37:32Z","updated_at":"2017-07-28T14:37:32Z","body":"Just to add a note: NTFS, starting from windows server 2012, has offline deduplication. It runs automatically when the system is idle, and it's suspended when there is system load.\r\nThis saved a lot of space on some general purpose fileservers, where my customers keeps many office files, images, emails, attachments. After 5-6 years of work, we find that 30-40% of used space is duplicated, and automatic NTFS background deduplication saved us that space.\r\nI tried to do the same with zfs online deduplication, but the ram usage grew up to more than 16GB. And not only ram was a problem, but mount time! It took a lot of time to remount a deduplicated disk after a reboot.\r\nI think that offline deduplication for zfs will be a very interesting feature.\r\nFor a deduplicating backup system, as asked on the 1st post, which avoids big transfers after a simple directory rename, look for (borg backup)[https://github.com/borgbackup]."}}
{"actor":{"display_login":"giox069"}}