I've been using an Ext3/4+lvm2+mdadm file system setup for about 5 years on a file server. This has grown (and moved processors a couple of times) and I've moved disks between systems (exporting/importing volume groups) and moved physical volumes onto new disks, so I'm fairly comfortable with it, but recently started looking at alternate file systems as the check and resync times for mdadm are becoming excessive and newer systems seem to be using checksums and dynamic healing to avoid this.

My setup uses RAID arrays build from partitions rather than entire disks, to allow mixing of disk sizes; for example, I have previously mixed 4 disks, 2 x 3TB, 1 x 2Tb and 1 x 1Tb to create a 9Tb array giving 6Tb of usable space. Assuming the 4 disks are formatted with 1Tb partitions, the mdadm/lvm2 commands are:

# mdadm /dev/md0 --create -lraid5 -n3 /dev/sda1 /dev/sdb1 /dev/sdc1
# mdadm /dev/md1 --create -lraid5 -n3 /dev/sda2 /dev/sdb2 /dev/sdc2
# mdadm /dev/md2 --create -lraid5 -n3 /dev/sda3 /dev/sdb3 /dev/sdd1
# pvcreate /dev/md0 /dev/md1 /dev/md2
# vgcreate grp /dev/md0 /dev/md1 /dev/md2
# lvcreate -l100%FREE --name vol grp

For ZFS, ignoring the log and any caches, the command would be:

# zpool create puddle raidz sda1 sdb2 sdc1 raidz sda2 sdb2 sdc2 raidz sda3 sdb3 sdd1

However, with BTRFS it seems the best that can be done is to create 3 raid arrays:

# mkfs.btrfs -draid5 /dev/sda1 /dev/sdb1 /dev/sdc1
# mkfs.btrfs -draid5 /dev/sda2 /dev/sdb2 /dev/sdc2
# mkfs.btrfs -draid5 /dev/sda3 /dev/sdb3 /dev/sdd1

Unless I've missed something, there appears no way within BTRFS to merge these into a single file system (Also, many discussions imply the raid5 implementation is fairly new and may not be up to production standard!). Is there a way to configure BTRFS to use to get a single file system across 4 disparately sized disks, with redundancy for the loss of any single disk and getting 2/3 of the space available for storage?

Or is my choice restricted to ZFS or my existing ext/lvm/mdadm stack?

Update: While waiting for BTRFS RAID5 to be declared stable, I went with creating a BTRFS RAID1 setup. Subsequently, BTRFS RAID5/6 was declared as very broken and only suitable for experimental setup, so I moved to using ZFS RAIDZ2 (RAID6) arrays.

  • I would not be considering using BTRFS at this time in any production environment. BTRFS still has aways to go to get to this point. – mdpc Mar 5 '15 at 2:14
  • I'm with mdpc on this. Note also you can add a write-intent bitmap to md devices so that if e.g. a power failure leaves the RAID device in an inconsistent state it's not necessary to resync the entire device, just those that might be out of date. Use mdadm --grow --bitmap=internal /dev/mdX to add this feature to an existing md device. – wurtel Mar 5 '15 at 8:05
  • That's something that surprised me. Although BTRFS is included in many (most?) distros as standard, it's not really considered "stable". Meanwhile, ZFS, which seems much more stable, mature and functional is still an add-on (although I suspect part of this is use of none-free code somewhere). I think I'll stick with ext4/lvm2/md for now (as it's really stable/tested/etc and I understand what it's doing) and just run the others on test systems. I'd still be interested to know if BTRFS can create the filesystem I'd like to use. – StarNamer Mar 5 '15 at 17:22
  • @Bratchley thank for the edit - comes from typing late and night, cutting and pasting and poor proofreading... – StarNamer Mar 5 '15 at 17:23
  • @StarNamer - ZFS is FOSS, but the licensing isn't compatible with GPL, so most distros don't include it. github.com/zfsonlinux – Peter May 25 '18 at 4:50

With your example disk sizes (2 x 3TB, 1 x 2TB, 1 x 1TB) this should work out well with btrfs raid5, with exactly 2/3 of the space being available for use, and with enough parity to lose any one drive. No partitioning is necessary, just give btrfs all of the devices directly.

You can use the btrfs space allocator website to experiment with different configurations to see how much space would be wasted.

In your example I would create the array by doing:

mkfs.btrfs -d raid5 -m raid5 /dev/sd[a-d]

Please do note however that while raid0/1/10 are fairly stable (they have been in the kernel for a long time now, and for example SuSE Linux Enterprise Server 12 officially supports these modes) the raid5/6 code only became semi-complete in linux 3.19, and raid5 is at this time still susceptible to the "write hole", so if you do try out btrfs raid5 then do make sure that you have good backups of any important data (which everyone should always try to do anyway, regardless of filesystem).

Personally I would wait a few more kernel versions before using btrfs raid5/6 for non-experimental setups.

To stay up to date with the status of raid5/6 the btrfs raid56 wiki page should usually be up to date.

  • Thanks for the information. After playing with the btrfs space allocator its clear that the information I was missing was that btrfs automatically stripes the data. AFAIK, md can only use equal size elements (in fact the current version warns if any element differs in size by more than 1%). If I understand correctly, btrfs automatically does the equivalent of mdadm /dev/md0 -C lraid5 -n4 /dev/sd{a,b,c,d}1;mdadm /dev/md1 -C -lraid5 -n3 /dev/sd{a,b,c}2;mdadm /dev/md2 -C -lmirror /dev/sd{a,b}3 plus create the volume. I obviously need to read up on it a bit more! (as well as waiting a while...) – StarNamer Apr 15 '15 at 9:40
  • 1
    A quick review of the btrfs wiki (btrfs.wiki.kernel.org/index.php/…) indicates that, once it is stable, btrfs will be able to reshape raid arrays, i.e. add and remove elements from a raid array without loss of data. This is a piece missing from ZFS so clearly makes it the better choice. – StarNamer Apr 15 '15 at 9:54
  • 1
    For the benefit of anyone reading this in 2020, the use of RAID 5 for metadata in btrfs is even more unstable/discouraged than for data, so the recommended command is mkfs.btrfs -d raid5 -m raid1 /dev/sd[a-d] instead which used RAID 1 for metadata. This will cost a little extra space, but makes it less likely you will lose the whole filesystem in case of errors. – ternaryOperator Sep 10 '20 at 3:08

Your Answer

