I've been tasked with selecting two new switches intended to be our new iSCSI SAN backbone.

Currently, we are using two Dell N4064 improvised for the task since that's what we had when the need arrived, but now we are going to replace them with two smaller, 24 port 10Gbit Ethernet switches.

I would ask if any 10Gbit Ethernet will perform adequately as an iSCSI backbone or if there are any specs I should be looking for or avoiding in my selection.

Thank you in advance for your input.

  • Clarification - this is a stack of 2x switches dedicated for redundant iscsi connections, and not for general traffic of which some is iscsi and some is "frontside" traffic ? – Criggie Feb 21 at 0:02
  • 1
    Power consumption (aka heat generated) can become an issue. That's true for any device in your server room, of course. But if your boss thinks that he's found a great deal on old powerful (read: power hungry) switches, tell him that he might need to upgrade the AC as well. – Klaws Feb 21 at 14:51
  • 1
    Criggie. This are to be used as two independent, non stacked, switches. They will be used just for the iSCSI traffic. – ÜM Mineiro Feb 21 at 17:21
  • Latency and bandwith are most important. Make sure there is no backplane bottleneck and the switches work cut-through. You also may want to consider DAC cables. And I would use rather more than less ports, if you can avoid upstream links it's a big plus. – eckes Feb 22 at 1:30

In a word: deep buffers, preferably per-port buffers

iSCSI does not tolerate frame drops. Delays are sort of ok, but a switch dropping a frame will create all manner of issues.

(Finding buffer size -- and type -- in manufacturer specs can be a challenge. They aren't always provided, and what is provided isn't always true.)

| improve this answer | |
  • I would actually argue the switch should do cut-through forwarding for low latency and avoid store+foreward and therefore should not buffer. – eckes Feb 22 at 1:27
  • 3
    If two or more machines try to send a frame at the same time, without a buffer, it gets dropped. Storage is about reliability, not low latency. In a SAN, it always takes measurable time to get the block you want; even an SSD is not instant. – Ricky Beam Feb 22 at 4:00

Make sure the switch is non-blocking. That is, the switch backplane can support all ports at line rate with no oversubscription. For a 24-port switch, the minimum backplane bandwidth is 24 x 10G, or 240Gb.

| improve this answer | |

Be sure to understand what your iSCSI implementation on the end systems has to say about LACP (Port Aggregation) and/or if it mandates path isolation, to have two completely independent network paths between any initiator and any target.

If either is true, then MLAG (Multichassis Link Aggregation) capabilities probably won't be required.

| improve this answer | |
  • Thanks. In our solution port aggregation is not allowed for iSCSI thought. – ÜM Mineiro Feb 20 at 17:39

In complement to Ron's answer, you need to look for Jumbo Frame. That is not mandatory, but it will significantly improve the bandwidth.

A good point too is the latency. It should ideally be below the µs.

You may also go for a full SFP+ switch, because depending on the equipment, you will have a mix of interfaces.

| improve this answer | |
  • 3
    "jumbo frames ... significantly improve the bandwidth" - that largely depends on your hardware. With decent offloading or hardware initiators the difference between standard and jumbo frames may be hardly noticable. – Zac67 Feb 20 at 15:04
  • I agree, in a scenario where I switched on jumbo frames it did only improve a few percents because the drivers have been doing receive aggregating. That's at least a feature of the Intel server adapters. – eckes Feb 22 at 4:08
  • We actually tested this on VMware ESXi, and jumbo frames added single percentage points in performance, very similar to this article (boche.net/blog/2011/01/24/…), so we decided to just to stick with 1500 due to the configuration demand that jumbo frames requires in larger environments. – Stuggi Feb 25 at 11:44

Your Answer

