text:
urn:lsid:ibm.com:blogs:entries-75889db3-24fb-4697-98cb-78945ded3943 seb's sanblog - Tags - performance A blog about storage networks and related technologies, news and hints. Author: Sebastian Thäle, IBM SAN PFE for EMEA in the ESCC (EMEA Storage Competence Center) in Kelsterbach, Germany. 0 30 15 2017-11-07T00:52:26-05:00 IBM Connections - Blogs urn:lsid:ibm.com:blogs:entry-6fa83763-c968-417f-a3eb-031700f675bb SAN Worst Practices 1: ISL R_RDY Mode seb_ 060000QVK2 active false seb_ 060000QVK2 active false Comment Entries application/atom+xml;type=entry Likes true 2017-04-10T06:12:36-04:00 2017-11-07T00:52:26-05:00
There is a function in FabricOS with a command name so unsuspicious and innocent you might want to enable it without even knowing what it does:

 portcfgislmode

Now doesn't that sound like: "If you plan to attach an ISL to this port, enable me! Enable me and I will prepare this callow port to be a real E-Port."?

 Well in some ears it seems to do so. But it's something completely different. This command enables the so called "ISL R_RDY Mode". Hmm... that still does not sound dangerous. So what is it?

Virtual Channels 101

We talked about virtual channels before and with "talked about" I mean I wrote about them here and here and briefly in some other articles. But for the sake of completeness let me explain what we need here.
 A normal classic ISL of a Brocade switch is partitioned in 8 so called virtual channels. In some documentation you'll find the term "virtual circuit", which is basically the same.

Unlike a multiplexed link an ISL still transports only one signal at a time. So what's partitioned is not really the ISL but its buffer credit management. That means you have a distinct buffer credit counter for each virtual channel, which is decreased every time a frame is transmitted for that particular virtual channel. The receiving switch recognizes for which virtual channel a frame is sent. To give buffer credits back as soon as it can receive more frames, the switch will send a VC_RDY. This is a ordered set (4 byte word) that contains the information for which virtual channel it carries a buffer credit.

In the classical ISL mentioned above we have basically 8 virtual channels and only 5 of them are needed for most purposes in a FCP SAN: Virtual channel 0 and 2-5. VCs 2-5 are the "data-VCs", they carry the user traffic - mainly the I/O between end-devices, but also control frames and administrative stuff between end-device and switch. Additionally switches talk with each other in an own service class "Class F" using virtual channel 0. If you change the zoning, the new zoning configuration will be distributed in the fabric as Class F traffic. The same for devices coming online or when the the principal distributes the current time.

Why Virtual Channels?

The advantage of virtual channels is, that if there is a bottleneck further down the way - for example due to a slow drain device - not all traffic is blocked, but only the traffic mapped to the same virtual channel like that slow drain device. The other devices still can talk freely. It's even more important to separate normal traffic from Class F traffic. You don't want your fabric to stay in an inconsistent state because a slow drain device consumed the buffer credits. And you don't want it because of another reason I will explain later.

Here you see the buffer credits of a classic ISL in the output of the root-level command portregshow. The VCs count from left to right beginning with VC0. For this article you can ignore the both credits for VC6 and VC7.

bbc registers
 =============
 0xcdc32800: bbc_trc 4 0 5 5 5 5 1 1

The VC0 (transporting Class F) has 4 buffer credits. The data-VCs 2-5 have 5 credits each.

You can increase the buffers of an E-Port (which you would see in the other switch's portregshow then) in 2 ways:

1) With portcfgeportcredits as described here. This would give you additional buffer credits on each data virtual channel. It would look like this:

bbc registers
 =============
 0xcdc32800: bbc_trc 4 0 20 20 20 20 1 1

All of the data virtual channels have 20 buffer credits now. The 4 buffers for Class F would stay the same.

2) With a long distance mode. By doing that, the data VCs collapse to only VC2. The rest of the VCs stay the same again.

bbc registers
 =============
 0xcdc32800: bbc_trc 4 0 300 0 0 0 1 1

You see: You might change buffer credits, but it will only affect the data VCs. Class F traffic stays with 4 buffers regardless of the distance. And that's a good thing. Because Class F is a beast.

Principal ISLs

Normal I/O can be routed over any ISL that is available as long as it is on one of the shortest paths according to the FSPF protocol (Fabric Shortest Path First). Class F traffic will usually not do that, it will go over principal ISLs. When the fabric is built, a spanning tree is built as well. It is rooted at the fabric's principal switch and reaches every switch in the fabric exactly one time (thus a spanning tree). You can see the involved ISLs in switchshow marked with "upstream" and "downstream" depending on the direction towards or away from the principal.

So why is Class F a beast?

Class F has the highest priority (together with the F_RJT and ACK traffic from class 2). Especially higher than any user traffic. But it is caged in VC0 with the fixed 4 buffers. So regardless of how long the distance is, it is always limited to these 4 buffers. It wants to bite the ones outside the cage, but it can't. In the fabric that means that it just can take some time until all the fabric information is distributed, but that's fine. The processes are designed in the way to cope with that.

But what if the beast breaks out of the cage?

ISL R_RDY mode is intended to be used on long distance links going over legacy SAN extenders or gateways that cannot cope with VC_RDYs or only transport the plain frames without any ordered set per se. If you use ISL R_RDY mode, VC_RDYs cannot be used and so there are no virtual channels. All virtual channels will collapse to one big channel and it will get all the buffer credits. So in portregshow it looks like this:

bbc registers
 =============
 0xcdc32800: bbc_trc 200 0 0 0 0 0 0 0

The cage is broken, the beast is free. The highly prioritized Class F traffic is not limited to 4 buffers anymore but can consume all of them freely. And it's strange how much Class F traffic there suddenly is. Especially in situations where something changes in the fabric, user traffic is effectively blocked and in extreme cases, the user traffic waits for such an amount of time, that it's dropped due to timeout. Additionally a lot of back pressure spreads in other parts of the fabric and causes performance problems even for apparently unrelated devices.

 Conclusion

If it's not absolutely necessary to use ISL R_RDY mode, you should NEVER enable it. It's a special purpose mode that should ONLY used for that special purpose of connecting two parts of a fabric with legacy SAN extenders or gateways. You don't have that? Then don't use ISL R_RDY mode. You even have ISL R_RDY mode enabled on local ISLs? Switch it off! You really have a reason for having ISL R_RDY mode? In my eyes the time is now to re-think your SAN architecture and figure out what changes have to be made to be able to live without ISL R_RDY mode.

Because you don't want that beast out there!

Short link: http://ibm.biz/ISLRRDY
 There is a function in FabricOS with a command name so unsuspicious and innocent you might want to enable it without even knowing what it does: portcfgislmode Now doesn't that sound like: "If you plan to attach an ISL to this port,... 1 2 5138 urn:lsid:ibm.com:blogs:entries-75889db3-24fb-4697-98cb-78945ded3943 seb's sanblog 2017-11-07T00:52:26-05:00 urn:lsid:ibm.com:blogs:entry-a39d477e-c264-4e61-ba35-ac41523f441d Performance buffers for Brocade ISLs seb_ 060000QVK2 active false seb_ 060000QVK2 active false Comment Entries application/atom+xml;type=entry Likes true 2014-10-23T13:03:44-04:00 2014-12-01T03:18:52-05:00
 Sometimes we notice that an ISL is actually a bottleneck in a fabric. Not a congestion bottleneck where the throughput demand is just too high for the ISL's bandwidth. This one could be solved by putting another cable between the both switches. But if you have a latency bottleneck your ISL won't be running at the maximum of the bandwidth. The contrary is the case: it lacks the buffer credits to ensure a proper utilization. If you see a latency bottleneck on an ISL it's often back pressure from a slow drain device attached to the adjacent switch. But every now and then I get a case where it's just the ISL. Sometimes in one direction, sometimes in both. Even with lengths were you don't think about using long distance settings at all.

 But in the past we did exactly that!

 When we encountered a situation like that, the first step was always to get rid of everything that reduces buffer credits for the real traffic flows, like an active QoS setting without having QoS zones. If the problem was still there the only way to give it more buffers was to configure a long distance mode then. We solved performance bottlenecks on ISLs by setting up a let's say 50m ISL in a 10km long distance mode (LE). I described this also 2 years ago in the article How to NOT connect an SVC in a core-edge Brocade fabric. While this indeed gives you more buffers, it comes with a drawback.

 Long distance and Virtual Channels

 On normal ISLs we have Virtual Channels. They work in a way that the buffer credit management of the ISL is logically partitioned into 8 channels. When we talk about normal Class 3 open systems traffic it's used this way:

 <colgroup> <col style="width:24px;" /> <col style="width:132px;" /> </colgroup>

  VC	  Used for

  0	  Class F

  1	  not used

  2	  Class 3 data

  3	  Class 3 data

  4	  Class 3 data

  5	  Class 3 data

  6	  Multicast

  7	  Multicast/Broadcast

 VC 0 is used for inter-switch communication. For example if a new zoning is distributed to all switches. The VCs 6 and 7 are not really of interest most of the time. We have to focus on VCs 2, 3, 4, and 5. (Mind the Oxford comma!) If you have a slow drain device that is reached using Virtual Channel 2 in your fabric, then at least the traffic of the other 3 data VCs is unaffected. With a long distance mode like LE you lose that advantage.

 Buffer distribution on Virtual Channels on a normal ISL:

 <colgroup> <col style="width:49px;" /> <col span="8" style="width:23px;" /> </colgroup>

  VC	  0	  1	  2	  3	  4	  5	  6	  7

  credits	  4	  0	  5	  5	  5	  5	  1	  1

 Buffer distribution on Virtual Channels on a LE configured ISL:

 <colgroup> <col style="width:49px;" /> <col span="8" style="width:23px;" /> </colgroup>

  VC	  0	  1	  2	  3	  4	  5	  6	  7

  credits	  4	  0	  80	  0	  0	  0	  1	  1

 While you have more buffers in total now, only the first data VC has them assigned. There is no partition of data traffic anymore and the result is the risk of Head of Line Blocking (HoLB). A latency bottleneck (for example due to back pressure from a slow drain device) will always impact ALL the user data going over that ISL! That's a high price for those additional buffers.

 The solution

 With FabricOS v7.2x Brocade introduced a new command:

 portcfgeportcredits

 It allows you to assign a freely configurable number of credits between 5 and 40 for that ISL. You might ask:

 But LE mode gives me 80 on 16Gbps!?!

 Yes, but look at the distribution:

 <colgroup> <col style="width:49px;" /> <col span="8" style="width:23px;" /> </colgroup>

  VC	  0	  1	  2	  3	  4	  5	  6	  7

  credits	  4	  0	  40	  40	  40	  40	  1	  1

 Not the whole data part of the link will share the 40 buffers. Each data VC gets its own 40 buffers and they are still handled independently! No Head of Line Blocking! And remember: This is not meant for long distance connection and it still comes for free! It works on 8G switches, too, as long as they are running at least v 7.2x.

 To give 40 buffers to each data VC on an ISL at port 1 you would enter:

 portcfgeportcredits --enable 1 40

 With the --disable parameter you switch back to normal mode and with --show you can see the current configuration of a port.

 And please keep an eye on the number of remaining buffers in portbuffershow :o)

 So from now on if you need just some more buffers on your ISLs to keep everything running smoothly:

 Use portcfgeportcredits!
 Sometimes we notice that an ISL is actually a bottleneck in a fabric. Not a congestion bottleneck where the throughput demand is just too high for the ISL's bandwidth. This one could be solved by putting another cable between the both switches. But if you... 3 11 18701 urn:lsid:ibm.com:blogs:entries-75889db3-24fb-4697-98cb-78945ded3943 seb's sanblog 2017-11-07T00:52:26-05:00 urn:lsid:ibm.com:blogs:entry-04c9dd45-0147-4a74-a311-1fc4c7919e66 Six excellent and free SAN ebooks without any registration or SPAM seb_ 060000QVK2 active false seb_ 060000QVK2 active false Comment Entries application/atom+xml;type=entry Likes true 2014-06-15T10:33:35-04:00 2014-06-15T15:27:13-04:00
 Well-made professional education is worth every cent, but in today's world controlled by CFOs everything costing money will be challenged sooner or later. And if you search for freebies you often end up with the first 3-4 sentences of an obviously good book about the topic and the prompt to register with your business information and email address. Weeks of business SPAM will follow then even if you unsubscribe again. Here are some good free books to get a good understanding of SAN switching and how it's implemented by the both big players Cisco and Brocade without the need to register for anything.

 Introduction to Storage Area Networks and System Networking

 Working at IBM I appreciate their Redbooks program. Experts from in- and outside IBM share their knowledge in for of these comprehensive ebooks. This one is a good introduction to SAN and how IBM does it. You learn how Fibre Channel works, the hardware, the software, the management, the use cases and the design considerations. And of course it covers the IBM products in that area, too.

 SNIA Dictionary

 Regular readers of my blog (are there any?) may know my opinion about the SNIA Dictionary, but for learning Storage Networking it's still a good source of definitions and explanations for many of the common terms and concepts. Get it directly from snia.org.

 Cisco MDS 9000 Family Switch Architecture

 This document is also known as "A Day in the Life of a Fibre Channel Frame" and I like it. Although it certainly saw some summers and winters since its release in 2006 but the general architecture is still the same. Of course everything is integrated and consolidated in the latest products, but if you ever understood how a frame is handled by a n older generation Cisco switch, it won't be a problem to work with, design for, or even troubleshoot the newest ones.

 Brocade Fabric OS Administrator's Guide

 While Brocade is certainly not revealing too much about the internals of their switches, the admin guide is still a good source of information about the Brocade features and implementations. Many SAN questions I'm asked in an average week could easily be answered by a glimpse into this guide. There is a new one for each new major codestream, so always look in the one for your installed FabricOS version. This is the link for FabricOS v7.2.

 The remaining two ebooks on my list are specifically for performance troubleshooting... ...my hobbyhorse somehow.

 Slow Drain Device Detection and Congestion Avoidance

 This one is from Cisco and it covers the different types of performance problems pretty well. If you read the one about Cisco architecture before (see above) you can get much more out of this piece as well. It has some good example, troubleshooting approaches and explanations for the counters you might see. A definite must-read.

 IBM Redpaper: Fabric Resiliency Best Practices

 This one is about Brocade switches and the IBM version of their "SAN Fabric Resiliency Best Practices". After explaining the fundamentals about SAN performance it shows you how performance troubleshooting is done on a Brocade fabric, especially by using built-in features like bottleneckmon.

 I'm sure there are many other good learning materials out there that don't exist for the sole reason to catch your contact addresses by registration. If you know some that should be on this list as well, please let me know. Thanks!
 Well-made professional education is worth every cent, but in today's world controlled by CFOs everything costing money will be challenged sooner or later. And if you search for freebies you often end up with the first 3-4 sentences of an obviously good... 2 0 12653 urn:lsid:ibm.com:blogs:entries-75889db3-24fb-4697-98cb-78945ded3943 seb's sanblog 2017-11-07T00:52:26-05:00 urn:lsid:ibm.com:blogs:entry-2130d3cf-1606-4131-91f8-38494a1e4512 Why congestion bottlenecks are dangerous, too seb_ 060000QVK2 active false seb_ 060000QVK2 active false Comment Entries application/atom+xml;type=entry Likes true 2013-09-10T04:17:45-04:00 2013-09-10T04:17:45-04:00
 Almost a year ago I wrote an article about congestion bottlenecks in Brocade switches. I said you should avoid them, because they mean that you probably have no redundancy because of too much workload or you don't use it properly. You can use the bottleneckmon to detect them. On the other hand I cared much more about latency bottlenecks, often caused by slow drain devices and their implications. And so I do today.

 Well...stop! Didn't you talk about congestion bottlenecks?

 Yes! Today I want to explain how a congestion bottleneck could cause the exact same symptoms on the devices like a latency bottleneck - and exactly the same performance degradations. This is how it happens. In the middle you see a SAN director with 2 portcards and 2 core cards. While the devices are connected to the portcards, the core cards provide the backend connections between them. They are internally connected via the backplane. So for example host 1's way over there to the storage array A would traverse the portcard, then one of the both core cards and leaves the other portcard until it reaches storage array A. It could even be that two devices connected to the same portcard have to go over the core cards, because so called local switching is only done within an ASIC and a portcard could have more than one depending on the number of ports.

 Now please meet host 2. Host 2 is a wonderful modern server. One of the work horses of the datacenter. It's fully packed with virtual machines, but its many cores and memory, as well as its state-of-the-art HBA, provide enough horsepower to cope with the workload. This baby is more than capable to do the work and it's in no way a slow drain device. It's zoned and mapped to the storage arrays A, B, C and D and it uses them heavily, mostly for read operations. The green tiny bars are read requests and as you see in the next picture it sends them to all of them, all of the time.

 Of course the other hosts send requests, too, but let's focus on our diligent host 2. Yes, the pictures are too simplistic, but I'm sure you'll get the point. On the next one you see the first responses flowing back to host 2. Communicating with several storage arrays the link towards host 2 is used heavily, but host 2 is processing the incoming frames quickly and gives buffer credits back to the switch in proper time. So far so good.

 But the more and longer the link utilization is very high, the more likely the following will happen if you enabled bottleneckmon with alerting:

 2013/09/07-12:07:11, [AN-1004], 7002, SLOT 7 | FID 128, WARNING, FAB1DOM5, Slot 2, port 14 is a congestion bottleneck. 99.67 percent of last 300 seconds were affected by this condition.

 If you didn't enable bottleneckmon, the congestion bottleneck would still be there... you just wouldn't know it.

 The crux is: you will hardly find a congestion bottleneck that just flows with high link utilization and no negative effects. The probability is much higher for the following scenario:

 Although there are enough buffer credits for this highly utilized link, frames are piling up towards it, because there is just too much workload and the link is busy sending frames. There is no slow drain device and to stay with the bathing metaphor: the drain works very well and transports as much water as its physically able to do. But there is so much more water in the tub that it could not go through the drain at the same time. And in addition imagine you have not only one water tap (in our case storage arrays) but four of them. They fill the tub quicker than the drain can empty it. As a result the internal buffers for all the hops through the SAN director fill up (that's basically the tub) and finally the director needs to do something against that: It will slow down the sending of buffer credits to the devices. Not only devices that want to send frames directly to host 2, but due to back pressure also the ones that send frames into that rough direction (using the same internal connections for example). And finally you'll end up in something like this:

 The SAN director just behaves like a slow drain device itself!

 Frames pile up inside the storage arrays and other end devices impaired by the slow drain behavior. If their RAS package is good, they will yell about credit starvation and probably even drop frames within their FC adaptors. In extreme situations these frame drops could happen in the director, too. At least you would see then something that would point you to a performance problem. Because otherwise - if you would have substantial delay in the traffic but all the frames get finally transferred to the next internal or external hop within 500ms ASIC hold time - you would only see the congestion bottleneck. And without bottleneckmon you wouldn't see anything at all then. The switch would look clean. Nothing in porterrshow or porstatsshow. Both show only external port counters anyway. As a SAN administrator you would not suspect anything in the director to cause this.

 And still it would be there. A big performance problem caused by a device communicating with too much other devices. Not a slow drain device but still causing a slow drain in the SAN.

 So how to solve it?

 It's basically what I wrote a year ago plus points 3. and 4. from How to deal with slow drain devices. You just have to ensure - from a architectural design point of view - that all components of the SAN are able to cope with the workload at any given time. It's both that easy and that complex. But the first step towards resolving such a situation is to detect it properly and to keep in mind what could happen.

 Almost a year ago I wrote an article about congestion bottlenecks in Brocade switches. I said you should avoid them, because they mean that you probably have no redundancy because of too much workload or you don't use it properly. You can... 1 1 14439 urn:lsid:ibm.com:blogs:entries-75889db3-24fb-4697-98cb-78945ded3943 seb's sanblog 2017-11-07T00:52:26-05:00 urn:lsid:ibm.com:blogs:entry-763d118a-aaee-4b47-b582-bd68ad6695ba Can IBM FlashSystem bring scale out systems back into the analytics game? seb_ 060000QVK2 active false seb_ 060000QVK2 active false Comment Entries application/atom+xml;type=entry Likes true 2013-05-05T09:27:56-04:00 2013-05-05T09:29:18-04:00
There are some good videos out there on the STG Europe Youtube channel about the infrastructures able to cope with analytics workloads. Distinguished Engineer John Easton discusses the requirements for these kind of workloads in the video "IBM Big Data with John Easton" below:

He points out that it is more efficient to use large memory systems with high computing power like Power Systems or System z instead of multiple parallel working System x nodes. The reason for that is the high I/O demand contrary the high wait times that result out of the usage of disk based storage systems to share the data between the nodes during processing. Especially for real-time analytics he recommends to have all the computation within the same box.

The same preference of a scale up approach of high powered systems versus scale out infrastructures is explained by Paul Prieto, Technical Strategist for Business Analytics in the video "Choosing the right platform for Cognos Analytics":

Can flash make a difference?

With I/O performance being the main reason for avoiding a scale out strategy, there is of course the question: What if I the I/O performance could be drastically enhanced? Before IBM acquired Texas Memory Systems in 2012, their RamSan systems were rarely used to accelerate scale out infrastructures as far as I know. The main use case was to boost the few big boxes running highly productive applications but waiting for their I/O due to inadequate I/O latencies provided by traditional disk storage systems. With their I/O latencies within the range of two-digit to lower three-digit microseconds and their capability to sustain several hundred thousands of IOPS they were used as a Tier 0 storage for only the most demanding and business-critical workloads.

With the integration of the now called IBM FlashSystem into the IBM storage portfolio another use case emerged and since then played a growing role in these deployments: IBM FlashSystem behind IBM SAN Volume Controller.

The pair "FlashSystem plus SVC" represents in fact two approaches:

Using SVC to virtualize the all-flash FlashSystem and enrich its raw I/O performance with the features you expect from a today's virtualized storage solution like seamless migrations, remote copy, thin provisioning, snap-shots (FlashCopy) and many more.

Using FlashSystem to boost existing SVC-virtualized storage environments by using it for Easy Tier as well as for pure flash-based volumes.

Especially the second way combined with the wide range of supported host systems, HBAs, and operating systems now makes it interesting for a former no-go: Running applications with really high I/O demand like analytics on scale out commodity systems while relying on an impressive I/O performance available outside in the SAN. But of course - as always - it's not that simple. Yes, there will still be scenarios where such a scale out approach is just not applicable. Especially then it might make much sense to speed up the storage even for the scale up purpose-built business analytics systems. However for many - for example SMB - companies it'd make perfect sense to run their analytics on flash-accelerated clusters of x86 based commodity hardware...

...if they do it right.

So how to do it right?

Well, this blog is not intended to explain reference architectures or architectural best practices for analytics. But I want to add the SAN point of view. (I guess you already wondered when this will start - given the usual topics of "seb's sanblog") And from my perspective as a SAN troubleshooter I can at least tell you what should be taken into consideration to not let it fail from the beginning. There are two major points: The general architecture and the hardening of the SAN. The proper architecture (for example by keeping the FlashSystem and SVC attached to the core) is the base, but a hand full of issues could have an unacceptable impact on the performance. Many of them I already covered in earlier blog posts and some of them will be the topics of future ones.

The main goal is to prevent the SVC ports from being blocked. Ever. May it be back pressure due to slow drain devices, sub-optimal cabling patterns, "unlucky" longdistance settings, enabled but unused QoS, too few buffers set for the F-ports, sheer overload of links, and many others.

With disk-based storage we talked about good average latencies of around 3ms. As the combination of FlashSystem plus SVC now works with a tenth of that and lower, the storage network's performance really start to make a difference. Usually we talk about single-digit microseconds one-way from device to device in a well-designed SAN. But the issues described above could increase this into the range of hundreds of milliseconds. Then of course it will hardly be possible to provide real-time business analytics. Therefore it is important to harden the SAN with the possibilities you have today, like - speaking of Brocade fabrics - Fabric Watch, bottleneckmon, Advanced Performance Monitoring, port fencing, traffic isolation zones, and so on. Brocade's "Fabric Resiliency Best Practices" are a good first step in this direction.

Conclusion

I think it's still possible to create a scale out infrastructure for business analytics even - and especially - with SAN based storage, as long as it's optimally prepared and using IBM FlashSystem solutions to overcome the mechanically caused latencies of disk storage. But it's crucial to ensure that these benefits are not rendered void by avoidable performance problems.

IBM Experts are more then willing to support you in this challenge. ;-)

 There are some good videos out there on the STG Europe Youtube channel about the infrastructures able to cope with analytics workloads. Distinguished Engineer John Easton discusses the requirements for these kind of workloads in the video "IBM Big Data with... 1 0 10598 urn:lsid:ibm.com:blogs:entries-75889db3-24fb-4697-98cb-78945ded3943 seb's sanblog 2017-11-07T00:52:26-05:00 urn:lsid:ibm.com:blogs:entry-715b844c-34b6-4cf2-aa4e-2707f34f52de How to fight back pressure against an appliance in a core edge fabric seb_ 060000QVK2 active false Comment Entries application/atom+xml;type=entry Likes true 2012-11-24T20:45:47-05:00 2012-12-02T10:16:39-05:00
A slow drain device often has a huge impact on the performance of many other devices in a SAN environment. That happens, because they block resources in a fabric other devices use as well. The main example for such a resource are ISLs, particularly the Virtual Channel(s) within those ISLs that are used to reach the slow drain device. But as soon as you have an appliance in the SAN, this could turn into such a blocked resource as well.

Disclaimer: There are several definitions and types of appliances. Within this article an appliance is a device "in the middle" between the hosts and the storages with a specific task such as a compression, encryption, virtualization or deduplication appliance. While I had the SAN Volume Controller (SVC) in mind while I wrote this, it applies to many other products matching this definition. The common thing is that the performance they can provide is to some degree dependent on their destination device's performance.

Fortunately many of the fabrics I saw over the recent years were designed using a core-edge approach. If the device is in the communication path of many of the devices in a SAN it's best practice to attach it directly to the core. But a slow drain device can still block it. This is how it happens:

In this sketch the appliance sends data towards a slow drain device. It will not be able to process the incoming frames quickly enough - they pile up in its HBA's ingress buffers (1). As the appliance is still sending frames but the edge switch cannot send them further to the slow drain device, they also pile up in the ingress buffer of the ISL port of the edge switch (2). Now this could already impair the performance of the other host connected to the same edge switch like the slow drain device - if the frames towards it use the same VC. Some microseconds later the same might happen to the frames from the appliance entering the core (3). They pile up there as well and as soon as that happens, this so called back pressure reaches the appliance itself then. As there are no VCs on the F-to-N-port connection used to attach the appliance to the core, the chance is high that the appliance cannot send any frames out to the SAN anymore - no matter to which destination (4).

That means?

Well, that means you just turned your appliance into a slow drain device itself! The performance of the whole environment is heavily impaired now:

In step (5) the frames from the other hosts towards the appliance pile up in the core as well and then the back pressure spreads further to the hosts connected to the edge switches as well (6).

Worst case, hmm?

After the ASIC hold time is reached (usually 500ms) the switches will begin to drop frames to free up buffers again. But as all switches have the same ASIC hold time, you'll end up in the situation that while the edge switch reach these 500ms first, the core switch will start to drop the frames likewise before the buffer credit replenishment information (VC_RDY) from the edge switches arrive. So not only the frames from the communication with the initial slow drain device will be dropped, but most of the others down the path as well. As as the appliance itself turned into a slow drain device, the same might happen to the frames piled up because of that, too.

So what to do against it?

The first thing is: give the F-ports of the appliance as much buffers as possible. Prio 1 should be that it should be able to send its frames out into the fabric, so the chances are higher that when the frames of the open I/O against the slow drain device are out there, there could be still some buffer credits left to send stuff to other devices. For clustered appliances like the SVC it's even more important, because they use these ports for their cluster-internal communication as well. Blocked ports could result in cluster segmentation then (SVC: single nodes rebooting due to "Lease expiry"). To assign more buffers to the switch port (= more buffer credits for the port of the appliance), use

portcfgfportbuffers --enable [slot/]port buffers

Update: Please keep in mind, that adding more buffers to an F-Port is of course disruptive for the link!

To check how much buffers are available, you can check

portbuffershow

But in many cases this is not enough. Some time ago, Brocade released Fabric Resiliency Best Practices with some good advises. In my opinion every SAN admin with Brocade gear should have read it. It recommends:

Use Fabric Watch to get alarms for frame timeouts. (Erwin von Londen wrote a good article about that.

Use Port Fencing to isolate slow drain devices. (Read Erwin's post about that, too.)

Configure and use the Edge Hold Time.

Configure bottleneckmon to get alarms for latency and congestion bottlenecks.

While Fabric Watch is used more and more and especially in the FICON world - but also for open systems - I see some of our customers using port fencing, I hardly see anyone utilizing the Edge Hold Time feature. For a situation as described above it could really improve the situation for the appliance and the other hosts dramatically. It can be set to any value between 100ms and 500ms. It was introduced in FOS v6.3.1b. So if you expect hosts connected to an edge switch to behave slow draining in certain situations, in my opinion the Edge Hold Time of that switch should be set as low as possible. Of course it's always depending on your environment and how likely it is to be impaired by a slow drain device, but 100ms is a long time in a SAN. If you also have some legacy devices connected to these edge switches, check if a decreased hold time could be a problem for them.

It can be enabled and configured using the "configure" command, where it can be found in:

configure Not all options will be available on an enabled switch. To disable the switch, use the "switchDisable" command. Configure... Fabric parameters (yes, y, no, n): [no] yes Configure edge hold time (yes, y, no, n): [yes] Edge hold time: (100..500) [100]

You don't need to disable the switch to change the Edge Hold Time and as one of the fabric parameters it will be included in a configupload.

As it seems to be used very seldom in the field I would like to get some feedback if you actually used it. Please give me a hint if and in which situation it helped you. Thanks!

But don't forget: The most important thing is to get rid of the slow draining behavior!

 A slow drain device often has a huge impact on the performance of many other devices in a SAN environment. That happens, because they block resources in a fabric other devices use as well. The main example for such a resource are ISLs, particularly the Virtual... 0 4 17672 urn:lsid:ibm.com:blogs:entries-75889db3-24fb-4697-98cb-78945ded3943 seb's sanblog 2017-11-07T00:52:26-05:00 urn:lsid:ibm.com:blogs:entry-d533bb37-2f8c-40a4-b418-da46ef5daa91 How to handle a congestion bottleneck seb_ 060000QVK2 active false Comment Entries application/atom+xml;type=entry Likes true 2012-11-02T06:33:13-04:00 2012-11-02T06:34:55-04:00
I check the referrers of this blog from time to time to get to know where my readers are coming from. For many of them I cannot actually see it, because often "bridge"-pages are used - for example by the social networking sites. But a fair amount come from searches in google and other search engines. Some search queries there seem to repeat very often. Maybe I will write more articles about the others - because hey, this seems to be the stuff you're coming for :o) - but this time it's about congestion bottlenecks.

Congestion bottlenecks - beside of latency bottlenecks - are one of the two things the Brocade bottleneckmon can detect. The default setup will alert you - if you enabled bottleneckmon with alerting - for all situations where 80% of all seconds within a 5 minutes interval had 95% link utilization. That is a big number! Of course you can also modify the setup to be more aggressive or to spare you some messages in an environment that is usually "under fire"...

But I encourage you to take it seriously!

In my opinion, a healthy SAN should NEVER have congestion bottlenecks. With "healthy" I mean of course the time of normal operation. Not when you have an incident at the moment and there is no redundancy in some parts because for example the second fabric has a problem or one out of two ISLs between two switches had to be disabled... I wrote an  article last year about that and I think it fits well within the topic.

Rule of thumb: Link utilization should be up to 50% only.

And of course it should not be only 50% because you configured too few buffers! The setup of the link should always allow it to transport up to 100% of the workload that's physically possible. Otherwise you will have no real redundancy again!

But how to handle them now that I have them?

So you see these [AN-1004] messages in the error log and you know the port. What now? This is more about your SAN concept than defects or software features. The congestion bottleneck happens because the utilization of a link approaches its physical capabilities. Here are some ideas:

Start a real performance monitoring. Have a look at what the Advanced Performance Monitoring license can do for you. (Ask your IBM SAN support for a free 60days Universal Temporary License)

For example use Top Talkers to find out where the heavy traffic comes from.

Simple: Use more links - if you have enough resources. For ISLs - if you have free ports on both switches and the possibility to connect them over your patch panels or if needed over the long distance between two sites, just add ISLs to spread the load.

If you use a Traffic Isolation setup, check if it's done correctly and does not cost you too much bandwidth.

Check if you can run the ISL on a higher line rate ("speed"). More line-rate means you can actually transfer more data at the same time. But please keep in mind, that higher line rates require better cable quality. If you have only 100m OM2 cables between two switches, increasing the line rate from 4Gb to 8Gb or even 16Gb will most probably result in problems for the link!

For congestion bottlenecks on end-device links you should check the multipath driver first. Is the load spread over the available paths? If you have a plain active/passive failover multipather it might be okay to have high load on the ports in use. But if you use a round-robin load balancing, check if you can add additional paths (more HBA-ports) with common sense. Keep in mind that "more paths --> longer failover times" and many devices have a maximum limit for paths!

Server virtualization may allow you to move workloads to more suitable regions of your SAN environment to relieve the ones under pressure.

And often forgotten:

In many cases the congestion bottlenecks will be observed only at specific times. Usually the devices in your SAN don't have the same workload all the times. There is time when people sleep, there is time when people come to work and switch on their VDI'ed PCs, there is time when the backups run and there is time when big batch jobs run. A proper planning and scheduling is mandatory in today's data centers! Don't let the big workloads run at the same time. Spread them accross the course of the 24 hours you have. The same is true for the course of the week, the months, the quarter, the year.

The fewest environments are totally under-sized for the average mix of workload - but the demand of the components of this mix over time is the heart of your storage environment's performance!

If you need help to better manage your workloads, I'm sure your local IBM Sales rep or IBM business partner can bring you in contact with the right performance expert to work these things out for your special situation.

 I check the referrers of this blog from time to time to get to know where my readers are coming from. For many of them I cannot actually see it, because often "bridge"-pages are used - for example by the social networking sites. But a fair amount... 0 0 16979 urn:lsid:ibm.com:blogs:entries-75889db3-24fb-4697-98cb-78945ded3943 seb's sanblog 2017-11-07T00:52:26-05:00 urn:lsid:ibm.com:blogs:entry-87e29e93-4e5c-4288-8c01-a673222fdf1b How to determine the average frame size seb_ 060000QVK2 active false Comment Entries application/atom+xml;type=entry Likes true 2012-04-24T18:49:59-04:00 2012-11-02T12:11:08-04:00
I was asked where to look at in a switch to find the average frame size for a port. The safest way would be to use an external monitoring tool like a VirtualWisdom or a tracer as described in my LD mode article but if you don't own something like that you can get a good guess from the switches themselves. You just have to calculate it out of the number of frames and the number of bytes transferred.

Cisco MDS
 For Cisco it's easy. Just look into the "show interface" for the specific port and you'll find the both numbers in the statistics section for each interface:

 1887012 frames input, 1300631486 bytes ... 542470 frames output, 482780325 bytes

So we can just calculate the average frame sizes for both directions:

1300631486 bytes / 1887012 frames = 689 bytes per frame
 482780325 bytes / 542470 frames = 890 bytes per frame

Brocade

For Brocade switches you can get the information out of the portstatsshow command:

stat_wtx 35481072 4-byte words transmitted stat_wrx 70173758 4-byte words received stat_ftx 1111087 Frames transmitted stat_frx 1177665 Frames received

Here we don't have the plain bytes but 4-byte words. Don't worry - fillwords don't count into this number, so it's still valid for our calculation. We just have to multiply it by four to use it:

(35481072 * 4) bytes / 1111087 frames = 128 bytes per frame
 (70173758 * 4) bytes / 1177665 frames = 238 bytes per frame

It's really that easy?

Basically yes. With this average frame size you can find out the multiplier for the buffer credits settings. So if you have an average frame size of 520 and a link of 30 km, just calculate:

2112 (the max frame size) / 520 = 4

So you would set up the link for 120 km instead of 30 km to reserve a sufficient amount of buffers. That's it.

A last catch

If you read my article about bottleneckmon you probably already know that we work with 32 bit counters here. While they cover a few hours for the frames they wrap much quicker for the 4-byte words. So to be able to calculate an average frame size over several hours or days, 32 bit counters are not enough. Actually there are 64 bit counters for these values in the switches - although they are not part of a supportsave. The command portstats64show provides them. The first thing to keep in mind: While in the latest FabOS versions a statsclear resets these counters as well, you had to reset them with portstatsclear in the older versions.

The 64 bit counters are actually two 32 bit counters and the lower one ("bottom_int") is the 32 bit counter we used all the time in portstatsshow. But each time it wraps, it increases the upper one ("top_int") by 1. So after a while you might see an portstats64show output like this:

stat64_wtx 0 top_int : 4-byte words transmitted 2308091032 bottom_int : 4-byte words transmitted stat64_wrx 39 top_int : 4-byte words received 1398223743 bottom_int : 4-byte words received stat64_ftx 0 top_int : Frames transmitted 9567522 bottom_int : Frames transmitted stat64_frx 0 top_int : Frames received 745125912 bottom_int : Frames received

For the received frames it's then:

(2^32 * 39 + 1398223743) * 4 bytes / 745125912 frames = 907 bytes per frame.

Much manual computing, hmm?

Of course you could write a script for that or prepare a spreadsheet but my recommendation is still to start with a multiplier of 3 for normal open systems traffic and check with the command portbuffershow how many buffers are still available. And if you still have some, use them - but keep them in mind if you connect additional long distance ISLs or devices you want to give additional buffers as well.

Update Nov. 2nd 2012:

I was made aware that there is an easier and much more convenient way to use portstats64show: Just use the -long option.

Without it:

pfe_ODD_B40_25:root> portstats64show 26 stat64_wtx 7 top_int : 4-byte words transmitted 485794041 bottom_int : 4-byte words transmitted stat64_wrx 13 top_int : 4-byte words received 2521709207 bottom_int : 4-byte words received

With it:

pfe_ODD_B40_25:root> portstats64show 26 -long stat64_wtx 30557972957 4-byte words transmitted stat64_wrx 58371265974 4-byte words received

Much better, isn't it? Thanks to Martin Lonkwitz!

 I was asked where to look at in a switch to find the average frame size for a port. The safest way would be to use an external monitoring tool like a VirtualWisdom or a tracer as described in my LD mode article but if you don't own something like that you can... 0 3 21340 urn:lsid:ibm.com:blogs:entries-75889db3-24fb-4697-98cb-78945ded3943 seb's sanblog 2017-11-07T00:52:26-05:00 urn:lsid:ibm.com:blogs:entry-e38912da-26d7-48bc-a6b7-4a3019064772 How to NOT connect an SVC in a core-edge Brocade fabric seb_ 060000QVK2 active false Comment Entries application/atom+xml;type=entry Likes true 2012-04-04T05:22:42-04:00 2012-05-09T11:14:26-04:00
In one of my previous posts I wrote about "Why inter-node traffic across ISLs should be avoided". There is an additional "bad practice" that could lead to performance problems in the host-to-SVC traffic.

Let's imagine a core-edge fabric. A powerful switch (or director) in its center is the core. The SVC and its backend storage subsystems are directly connected to it. Beside of that there are also the ISLs to the edge switches where the hosts are connected to. As there is an SVC in the fabric, all host traffic usually goes to the SVC and the SVC is the only host of all other storages. From time to time I see a cabling like the one below. The devices are connected in a common pattern. For example SVC ports are always on port 0, 4, 8, ... or for a director for example on port 0 and 16 on each card... Something like that. The reason behind that is often to spread the workload over several cards/ASICs to minimize impact in case of a hardware failure. But there's a risk in doing so.

Index Port Address Media Speed State Proto ============================================== 0 0 190000 id 8G Online FC F-Port 50:05:07:68:01:40:a2:18 1 1 190100 id 8G Online FC F-Port 20:14:00:a0:b8:11:4f:1e 2 2 190200 id 8G Online FC F-Port 20:16:00:80:e5:17:cc:9e 3 3 190300 id 8G Online FC E-Port 10:00:00:05:1e:0f:75:be "fcsw2_102" (downstream) 4 4 190400 id 8G Online FC F-Port 50:05:07:68:01:40:06:36 5 5 190500 id 8G Online FC F-Port 20:04:00:a0:b8:0f:bf:6f 6 6 190600 id 8G Online FC F-Port 20:16:00:a0:b8:11:37:a2 7 7 190700 id 8G Online FC E-Port 10:00:00:05:1e:34:78:38 "fcsw2_92" (downstream) 8 8 190800 id 8G Online FC F-Port 50:05:07:68:01:40:05:d3 ...

The SAN perspective

In the situation described above, all host traffic is passing the ISLs from the edge switches to the core. ISLs are logically "partitioned" into so called virtual channels. Of course the ISL is still just one fibre and only one signal is passing it physically at the same time. The virtual channels are just portions of buffer credits dedicated and the decision which virtual channel a frame takes - and therefore which portion of the buffers credits it uses - is made by looking into the destination fibre channel address.

Technical deep dive

A normal non-QOS ISL has 4 virtual channels for data traffic. For an 8G link each one of them has 5 buffers. They can only work with these 5 buffers and there is no possibility to "borrow" some out of a common pool like for QoS links. With the command "portregshow" you can see the buffer credits assigned to the virtual channels (I added the first line):

 VC 0 1 2 3 4 5 6 7 0xe6692400: bbc_trc 4 0 5 5 5 5 1 1

Only VCs 2-5 are used for data traffic. This makes 20 usable buffers which normally should be enough for a normal multimode connection between two switches in the same room with only some metres cable length. Basically the switch uses the last two bits of the second byte of the destination address. That looks so:

 Bits 00 -> frame uses VC 2 (which is the first virtual channel for data)

 Bits 01 -> frame uses VC 3

 Bits 10 -> frame uses VC 4

 Bits 11 -> frame uses VC 5

So where's the problem now?

In our imaginary core-edge fabric where for example all SVC ports are connected to ports 0 (bin 00), 4 (bin 100), 8 (bin 1000), 12 (bin 1100) , ... all host I/O towards SVC would use the same virtual channel. As this is the only traffic that passes the ISLs from edges to cores, only a quarter of the buffers are actually used! 5 buffers are very heavy in use and 15 are idling around never to be filled. And 5 buffers are actually pretty few for an edge switch full of hosts wants to speak with the core switch where the SVC is connected. The result would be credit starvation and congestion on a virtual channel level.

How to solve that?

There are 3 possibilities:

1.) You could re-cable your SAN in a manner that all VCs are used. But beside of the risk of physical problems and problems introduced by maintenance actions the devices have to learn about the new addresses of the SVC ports. For many operating systems this still means reboots or reconfigurations. It could involve a lot of work and risk for outages.

2.) You could just change the addresses with the portaddress command. This command is usually used in the virtual fabric environment and if you can use it depends on installed firmware and used platform. While it avoids the physical actions, it still has the disadvantages for the hosts because of changed addresses.

3.) The best and least interrupting possibility might be to set the ISLs to LE mode. This is the long distance mode dedicated for links under 10km in length. It will not only put more buffers on the link (40 for user traffic in an 8G link compared with the 20 for a normal 8G E-Port) but will also collapse the 4 user traffic VCs to just one. It looks like this then:

 VC 0 1 2 3 4 5 6 7 0xe6602400: bbc_trc 4 0 40 0 0 0 1 1

So all buffers and therefore also all buffer credits will be used by the hosts and nothing idles. There will of course be a short interruption while changing the ISL to LE mode but beside of that nothing changes for the hosts, because all the addresses stay the same. This is clearly the way to go in the situation described above.

Just something strange for the end: Some switches are delivered from manufacturing with an alternative addressing pattern. For example port 1 of domain 3 won't have the address 030100 then but something like 030d00. In that case the problem can happen similarly but on other ports. But using LE-mode would solve it in the pretty same way.

Please keep in mind that the whole article relates to a very special (although very common) SAN layout in an SVC-centered environment. This is clearly not a standard action plan for all performance problems but it could help if you have a customer in a situation like this. For any questions, feel free to contact me.

Additionally, please be aware that this is not an SVC problem by itself but will happen with every central storage connected to a switch using a pattern as described above and being used by hosts connected to another switch over an ISL!

Update from May 9th:

I was made aware that readers of this article queried their vendors, maintenance providers or business partners with the idea to just set all their ISLs to LE-mode regardless if the condition as described above is actually met. Because of that, I would like to state more clearly: Using LE-mode as a general approach for your ISLs can cause severe problems!

If the SVC ports are not connected in a way that only one Virtual Channel would be used, it actually makes sense to have ISLs with more than one VC. Virtual Channels are a good feature to prevent that a latency bottleneck due to back pressure impairs the traffic of all devices using the same ISL. If devices on the edge switches communicate with other devices connected to other ports of the core (or other edges) as well, the impact of using LE-mode would be even more extreme in the case of slow drain devices.

I made some drawings to illustrate this. The first one shows 1 normal ISL between the edge and the core. You can see the 4 VCs used for data traffic. (I left out the other VCs for better visibility):

Here host 1 and 2 make traffic against the SVC (green), host 3 against an additional disk subsystem (purple) and host 4 against a tape drive (orange). Based on the ports these devices are connected to, other VCs are used for that traffic.

If you would use an LE-port instead, it would look like this:

Now all 4 data traffic VCs collapsed to a single one. As long as everything runs smoothly, you won't see an impact.

Buf if for example one of the devices connected to the core is slow draining, following will happen most probably:

In the picture above the purple disk is a slow drain device. Due to back pressure the whole ISL will be a latency bottleneck, because all data traffic shares the same VC in LE-mode. The back pressure goes further towards the edge switch and all 4 hosts of our example are affected now although only host 3 communicates with the slow drain device!

With a normal E-port it looks like this:

Now only VC4 is affected while VC2, 3 and 5 are running smoothly, because they have their own, unaffected buffer management. Therefore only host 3 will face a performance problem while the hosts 1, 2 and 4 are running fine.

You see: Using LE-mode for the purpose described in my original article does only make sense if these special conditions are really met. In all other cases it can impair the SAN performance tremendously!

 In one of my previous posts I wrote about "Why inter-node traffic across ISLs should be avoided" . There is an additional "bad practice" that could lead to performance problems in the host-to-SVC traffic. Let's imagine a core-edge fabric. A... 1 18 31753 urn:lsid:ibm.com:blogs:entries-75889db3-24fb-4697-98cb-78945ded3943 seb's sanblog 2017-11-07T00:52:26-05:00 urn:lsid:ibm.com:blogs:entry-1ad82a90-23a1-40a1-90cc-4e04888b6cd2 Defining SAN performance-related terms seb_ 060000QVK2 active false Comment Entries application/atom+xml;type=entry Likes true 2012-02-12T03:28:24-05:00 2012-03-19T11:43:29-04:00
I claim that in 2012 performance problems will keep their place amongst the most frequent and most impacting problems in the SAN. In many of the cases the client's users really notice a performance impact and so the admin calls for support. Other support cases are opened because of performance related messages like the ones from Brocade's bottleneckmon or Cisco's slowdrain policy for the Port Monitor. Beside of that there are also cases that look not really like performance problems from the start but turn out to occur because of the same reasons like them. "I/O abort" messages in the device log, link resets, messages about frame drops, failing remote copy links, failing backup jobs or even worse failing recoveries - these could all be "performance problems in disguise".

When I analyze the data then and find out that a slow drain device or congestion is the real reason for the problem I write my findings down and try to give the client some hints about possible next steps. For example by mentioning my earlier blog article about How to deal with slow drain devices.

Do you know what's mean about it?

Often clients never heard of slow drain devices before. Longtime storage administrators are confronted with a term that sounds like a support guy made it up to fingerpoint to another vendor's product. Of course I usually explain what it is, what it means for the fabric and for the connected devices. But to be honest, I would be sceptical, too. I would go to the next search engine and query "slow drain device". The first finds are from this blog and from the  Brocade community pages and there are some questions about that topic. Considering the substance of posts in public forums, I would check Brocade's own SAN glossary. Guess what? Not a word about slow drain devices - Which is no surprise as it's from 2008. I would check wikipedia. Nothing. My fellow blogger Archie Hendryx mentioned that it's missing in the SNIA dictionary, too. And he's right: Nothing!

So why is that so?

Why are the terms "HTML" and "export" explained in the dictionary of the Storage Networking Industry Association but there is not a single appearance of the term "slow drain device" on the complete SNIA website (according to their in-built search function)? Well I don't know but of course we can change that. The SNIA dictionary makers are asking for contribution, so if you have a term that has a meaning in the storage industry, feel free to send them a definition for the next release. I thought about doing that as well for some of the SAN performance-related terms I didn't find in the dictionary. Below you'll find some definitions that I wrote. But I'm not inerrable and therefore I would like to have an open discussion about them. Let me know what you think about them. Let me know if your understanding of a term (used in the area of SAN performance of course) differs from mine. Let me know if my wording hurts the ears of native English speakers. Let me know if you have a better definition. Let me know if there are important terms missing. And let me know if you think that a term is not really so generally used or important that it should appear in the SNIA dictionary - side by side to sophisticated terms like Tebibyte :o).

My definitions:

slow drain device - a device that cannot cope with the incoming traffic in a timely manner.
Slow drain devices can't free up their internal frame buffers and therefore don't allow the connected port to regain their buffer credits quickly enough.

congestion - a situation where the workload for a link exceeds its actual usable bandwidth.
Congestion happens due to overutilization or oversubscription.

buffer credit starvation - a situation where a transmitting port runs out of buffer credits and therefore isn't allowed to send frames.
The frames will be stored within the sending device, blocking buffers and eventually have to be dropped if they can't be sent for a certain time (usually 500ms).

back pressure - a knock-on effect that spreads buffer credit starvation into a switched fabric starting from a slow drain device.
Because of this effect a slow drain device can affect apparently unrelated devices.

bottleneck - a link or component that is not able to transport all frames directed to or through it in a timely manner. (e.g. because of buffer credit starvation or congestion)
Bottlenecks increase the latency or even cause frame drops and upper-level error recovery.

Feel free to use the comment feature here or tweet your thoughts with hashtag #SANperfdef. If you add @Zyrober in the tweet, I'll even get a mail :o)

I updated the definitions with an additional sentence. Feel free to comment.

 I claim that in 2012 performance problems will keep their place amongst the most frequent and most impacting problems in the SAN. In many of the cases the client's users really notice a performance impact and so the admin calls for support. Other support cases... 1 4 26529 urn:lsid:ibm.com:blogs:entries-75889db3-24fb-4697-98cb-78945ded3943 seb's sanblog 2017-11-07T00:52:26-05:00 urn:lsid:ibm.com:blogs:entry-7d3222de-ab7d-43a4-9202-ef13e49f690b SAN Myths Uncovered: Quality of Service in Brocade FabricOS seb_ 060000QVK2 active false Comment Entries application/atom+xml;type=entry Likes true 2011-11-14T18:17:46-05:00 2011-11-15T10:45:09-05:00
When Brocade released FabricOS v6.0 in 2007 Quality of Service sounded like a great idea: It allows you to prioritize your traffic flow to the level of certain device pairs. There are 3 levels of priority:

High - Medium - Low

Inter Switch Links (ISLs) are logically partitioned into 8 so called Virtual Channels (VCs). Basically each of them has its own buffer management and the decision which virtual channel a frame should use is based on its destination address. If a particular end-to-end path is blocked or really slow, the impact on the communication over the other VCs is minimal. Thus only a subset of devices should be impaired during a bottleneck situation.

Quality of Service takes this one step further.

QoS-enabled ISLs consist of 16 VCs. There are slightly more buffers associated with a QoS ISL and these buffers are equally distributed over the data VCs. (There are some "reserved" VCs for fabric communication and special purposes). The amount of VCs makes the priority work - the most VCs (and therefore the most buffers) are dedicated to the high priority, the least for the low one. Medium lies in the middle obviously. So more important I/Os benefit from more resources than the not so important ones.

Sounds like a great idea!

Theoretically you can configure the traffic flow in terms of buffer credit assignment in your fabric very fine-grained. But that's in fact also the big crux: You have to configure it! That means you actually have to know which host's I/O to which target device should be which priority. Technically you create QoS-zones to categorize your connections. Low priority zones start with QOSL, high priority zones start with QOSH. Zones without such a prefix are considered as medium priority.

But how to categorize?

That's the tricky part. The company's departments relying on IT (virtually all) have to bring in their needs into the discussion. Maybe there are already different SLAs for different tiers of storage and an internal cost allocation in place. The I/O prioritization could go along with that and of course it has to be taken into account to effectively meet the pre-defined SLAs. If you have to start from the scratch, it's more a project for weeks and months than a simple configuration. And there is much psychology in it. Beside of that you really have to know how QoS works in details to design a prioritization concept. For example if you only have 20 high priority zones and 50 with medium priority but only 3 low priority zones, the low ones could even perform better. In the four years since its release I saw only a couple of customers really attempting to implement it.

In addition you need to buy the Adaptive Networking license!

So why should I care?

If QoS is such a niche feature, why blogging about it? Usually a port is configured for QoS when it comes from the factory. You can see it in the output of the command "portcfgshow". A new switch will have QoS in the state "AE" which means auto-enabled - in other words "on". An 8Gig ISL will be logically partitioned into the 16 VCs as described above and the buffer credits will be assigned to the high, the low and the medium priority VCs. But that does not mean that you can actually benefit from the feature, because you most probably have no QoS-zones! And so all your I/O share only the resources allocated for the medium priority. A huge part of the available buffers are reserved for VCs you cannot use! So as a matter of fact you end up with less buffers than without QoS and in many cases this made the difference between smooth running environments and immense performance degradation.

Conclusion:

If you don't plan to design a detailed and well-balanced concept about the priorities in your SAN environments, I recommend to switch off QoS on the ports. I don't say QoS is bad! In fact with the Brocade HBA's possibility to integrate QoS even into the host connection - enabling different priorities for virtualized servers - you have the possibility to better cope with slow drain device behavior. But done wrong, QoS can have a very ugly impact on the SAN's performance!

Better know the features you use well - or they might turn against you...

Update:

As this was not clear enough in the text above and I got back a question about that, please be aware: Disabling QoS is disruptive for the link! In most FabricOS versions in combination with most switch models, the link will be taken offline and online again as soon as you disable it. In some combinations you'll get the message that it will turn effective with the next reset of the link. In that case you have to portdisable / portenable the port by yourself.

As this is a recoverable, temporary error your application most probably won't notice anything, but to be on the save side, you should do it in a controlled manner and - if really necessary in your environment - in times of little traffic or even a maintenance window. The command to disable it is:

portcfgqos --disable PORTNUMBER

 When Brocade released FabricOS v6.0 in 2007 Quality of Service sounded like a great idea: It allows you to prioritize your traffic flow to the level of certain device pairs. There are 3 levels of priority: High - Medium - Low Inter Switch Links (ISLs) are... 1 11 29155 urn:lsid:ibm.com:blogs:entries-75889db3-24fb-4697-98cb-78945ded3943 seb's sanblog 2017-11-07T00:52:26-05:00 urn:lsid:ibm.com:blogs:entry-7c3d1d4f-a97a-4418-924f-efac4e4e1fe9 Brocade bottleneckmon 101 seb_ 060000QVK2 active false Comment Entries application/atom+xml;type=entry Likes true 2011-10-31T20:14:16-04:00 2012-03-22T11:16:55-04:00
Performance problems are still the most malicious issues on my list. They come in many flavors and most of them have two things in common: 1) They are hardly SAN defects and 2) They need to be solved as quickly as possible, because they really have an impact. If just a switch crashed or an ISL dropped dead or even an ugly firmware bug blocks the communication of an entire fabric, it might ring all alarm bells. But that's something you (hopefully) have your redundancy for. Performance problems on the other hand can have a high impact on your applications across the whole data center without a concerning message in the logs, if your systems are not well prepared for it. Beside of the preparation steps I pointed out here there is a tool in Brocade's FabricOS especially for performance problems: The bottleneck monitor or short:

bottleneckmon

If a performance problem is escalated to the technical support the next thing most probably happening is that the support guy asks you to clear the counters, wait up to three hours while the problem is noticeable, and then gather a supportsave of each switch in both fabrics.

Why 3 hours?

A manual performance analysis is based on certain 32 bit counters in a supportsave. In a device that's able to route I/O of several gigabits per second, 32 bits aren't a huge range for counters and they will eventually wrap if you wait too long. But a wrapped counter is worthless, because you can't tell if and how often it wrapped. So all comparisons would be meaningless.

Beside the wait time the whole handling of the data collections including gathering and uploading them to the support takes precious time. And then the support has to process and analyze them. After all these hours of continously repeating telephone calls you get from management and internal and/or external customers, the support guy hopefully found the cause of your performance problem. And keeping point 1) from my first paragraph in mind, it's most probably not even the fault of a switch*). If he makes you aware to a slow drain device, you would now start to involve the admins and/or support for the particular device.

You definitely need a shortcut!

And this shortcut is the bottleneckmon. It's made to permanently check your SAN for performance problems. Configured correctly it will pinpoint the cause of performance problems - at least the bigger ones. The bottleneckmon was introduced with FabricOS v6.3x and some major limitations. But from v6.4x it eventually became a must-have by offering two useful features:

Congestion bottleneck detection

This just measures the link utilization. With the fabric watch license (pre-loaded on many of the IBM-branded switches and directors) you can do that already for a long time. But the bottleneckmon offers a bit more convenience and brings it in the proper context. The more important thing is:

Latency bottleneck detection

This feature shows you most of the medium to major situations of buffer credit starvation. If a port runs out of buffer credits, it's not allowed to send frames over the fibre. To make a long story short if you see a latency bottleneck reported against an F-Port you most probably found a slow drain device in your SAN. If it's reported against an ISL, there are two possible reasons:

There could be a slow drain device "down the road" - the slow drain device could be connected to the adjacent switch or to another one connected to it. Credit starvation typically pressures back to affect wide areas of the fabric.

The ISL could have too few buffers. Maybe the link is just too long. Or the average framesize is much smaller than expected. Or QoS is configured on the link but you don't have QoS-Zones prioritizing your I/O. This could have a huge negative impact! Another reason could be a mis-configured longdistance ISL.

Whatever it is, it is either the reason for your performance problem or at least contributing to it and should definitely be solved. Maybe this article can help you with that then.

With FabricOS v7.0 the bottleneckmon was improved again. While the core-policy which detects credit starvation situations was pretty much pre-defined before v7.0 you're now able to configure it in the minutest details. We are still testing that out more in detail - for the moment I recommend to use the defaults.

So how to use it?

At first: I highly recommend to update your switches to the latest supported v6.4x code if possible. It's much better there than in v6.3! If you look up bottleneckmon in the command reference, it offers plenty of parameters and sub-commands. But in fact for most environments and performance problems it's enough to just enable it and activate the alerting:

myswitch:admin> bottleneckmon --enable -alert

That's it. It will generate messages in your switch's error log if a congestion or a latency bottleneck was found. Pretty straightforward. If you are not sure you can check the status with:

myswitch:admin> bottleneckmon --status

And of course there is a show command which can be used with various filter options, but the easiest way is to just wait for the messages in the error log. They will tell you the type of bottleneck and of course the affected port.

And if there are messages now?

Well, there is still the chance, that there are actually situations of buffer credit starvation the default-configured bottleneckmon can't see. However as you read an introduction here, I assume you just open a case at the IBM support.

You'll Never Walk Alone! :o)

*)Depending on country-specific policies and maintenance contracts a performance analysis as described above could be a charged service in your region.

 Performance problems are still the most malicious issues on my list. They come in many flavors and most of them have two things in common: 1) They are hardly SAN defects and 2) They need to be solved as quickly as possible, because they really have an... 0 1 37357 urn:lsid:ibm.com:blogs:entries-75889db3-24fb-4697-98cb-78945ded3943 seb's sanblog 2017-11-07T00:52:26-05:00 urn:lsid:ibm.com:blogs:entry-a757f1f8-4a15-4651-8389-f3c627193c81 Things I like in Brocade FOS 7.0 - Part 2: The Frame Log seb_ 060000QVK2 active false Comment Entries application/atom+xml;type=entry Likes true 2011-09-23T19:40:57-04:00 2011-09-23T19:40:57-04:00
Time for another piece of my little series! This time I'd like to write about a new feature in v7.0x especially for administrators and support personnel: The Frame Log. Maybe it's a bit early to write about it, because it seems to be a feature "in development" at the moment, but I did wait for it so long I'm just not able to resist. I think and I hope Brocade will further develop it like the bottleneckmon - which I was very sceptical about in its first version when it was released in the v6.3 code. After seeing its functionality being extended on v6.4 and even more in v7.0, the bottleneckmon is an absolute must-have.

Hmm... maybe I should write an article about bottleneckmon, too :o)

Back to the Frame Log. So what's that?

Basically it is a list of frame discards. There are several reasons why a switch would have to drop a frame instead of delivering it to the destination device. One of them is a timeout. If a frame sticks in the ASIC (the "brain" behind the port) for half a second, the switch has to assume that something's going wrong and so the frame cannot be delivered in time anymore. Then it drops it. Till FabOS v7.0 it just increased a counter by one. Since later v6.2x versions it was at least logged against the TX port (the direction towards the reason for the drop) - in earlier versions the counter increased only for the origin port, which made no sense at all. But now we even have a log for it! A log to store all the frames the switch had to discard. While that sounds a bit like rummaging through the switch's trash bin, the Frame Log is very useful for troubleshooting though. It contains the exact time, the TX and the RX port (keep in mind the TX is the important one) and even information from the frame itself. In the summary view you see the fibrechannel addresses of the source device (SID) and of the destination device (DID).

For example to see the two most recent frame discards in summary mode, just type:

 B48P16G:admin> framelog --show -mode summary -n 2 ===================================================================== Fri Sep 23 16:07:13 CET 2011 ====================================================================== Log TX RX timestamp port port SID DID SFID DFID Type Count ====================================================================== Sep 29 16:02:08 7 5 0x040500 0x013300 1 1 timeout 1 Sep 29 16:04:51 7 1 0x030900 0x013000 1 1 timeout 1

In the so called "dump mode" you even see the first 64 bytes of each frame. Usually I have to bring an XGIG tracer onsite to catch such information and often it's not even possible to catch it then, because an XGIG can only trace what's going through the fibre. So you'll only see this frame if you trace a link it crosses before it is dropped. And even then you can't trigger (=stop) the tracer directly on this event, but you have to have it looking for a so called ABTS (abort sequence). If a frame is dropped the command will time out in the initiator and it will send this ABTS. Depending on what frame exactly was dropped in what direction, the ABTS could be on the link several minutes after the actual drop of the frame. Imagine a READ command being dropped. The error recovery will start after the SCSI timeout which could be e.g. 2 minutes. But 2 minutes is a long time in a FC trace. Chances are good that the tracer misses it then.

Not so with the Frame Log!

The frame log can tell you exactly which frame was dropped. If you try to find out if a particular I/O timeout in your host was caused by a timeout discard in the fabric, this is your way to go. If you see your storage array complaining about aborts for certain sequences, just look them up in the Frame Log. With this feature Brocade finally catches up with Cisco and their internal tracing capabilities - and Brocade does it way more comfortable for the admin. The logging of discarded frames is enabled by default and it works on all 8G and 16G platform switches without any additional license.

The big "BUTs"

As I mentioned at the beginning of this article there are still things for Brocade to work on to turn the Frame Log into a must-have tool like the bottleneckmon. The first catch is its volatility. In the current version it can only keep 50 frames per second on an ASIC base for 20 minutes in total. At the moment I personally think that's too short. But I'll wait for the first cases where I can use it before I forge an ultimate opinion about this limit.

The other - more concerning - constraint is that it only works for discards due to timeout at the moment. So if a frame is dropped because of one of all the other possible reasons, it won't be visible in the Frame Log in its current implementation. But that's exactly what I need! If the switch discards a frame because of a zone mismatch or because the destination switch was not reachable or because the target device was temporarily offline or whatever - I want to see that. If a server is misconfigured (uses wrong addresses) and so cannot reach its targets, you'd see the reason right there in the target log - no tracing needed! There are plenty of other situations that would be covered with such a functionality. So I honestly hope that there is a developer with a concept like this in his drawer or even already within its implementation. Allow me to assure you that there is at least one support guy waiting for it...

...desperately! :o)

The picture is from Zsuzsanna Kilian. Thank you!

 Time for another piece of my little series ! This time I'd like to write about a new feature in v7.0x especially for administrators and support personnel: The Frame Log. Maybe it's a bit early to write about it, because it seems to be a feature "in... 0 2 12360 urn:lsid:ibm.com:blogs:entries-75889db3-24fb-4697-98cb-78945ded3943 seb's sanblog 2017-11-07T00:52:26-05:00 urn:lsid:ibm.com:blogs:entry-226be0b8-74e3-4ca2-b817-16c6b3672835 How to deal with slow drain devices? seb_ 060000QVK2 active false Comment Entries application/atom+xml;type=entry Likes true 2011-07-09T20:14:18-04:00 2012-03-30T06:44:53-04:00
One of the ugliest things that can happen in a SAN is a big performance problem introduced by a slow drain device (or slow draining device). Why is it so ugly? Well, if a full fabric or a full data center drops down - due to a fire for example - it's definitely ugly, too. But such situations can be covered by redundancy (failover to another fabric, to another data center, etc), because the trigger is very clear. Whereas a performance degredation due to a slow drain device is not so obvious - at least not for the most hosts, operators or automatic failover mechanisms. Frames will be dropped randomly, paths fail but with the next TUR (Test Unit Ready) they seem to work again, just to fail again minutes later. Error recovery will hit the performance and the worst thing: If commonly used resources are affected - like ISLs - the performance of totally unrelated applications (running on different hosts, using different storage) is impaired.

So you have a slow drain device. If you have a Brocade SAN you might have found it by using the bottleneckmon or you noticed frame discards due to timeout on the TX side of a device port. If you have a Cisco SAN you probably used the creditmon or found dropped packets in the appropriate ASICs. Or maybe your SAN support told you where it is. Nevertheless, let's imagine the culprit of a fabric-wide congestion is already identified. But what now?

The following checklist should help you to think about why a certain device behaves like a slow drain device and what you can do about it. I don't claim this list to be exhaustive and some of the checks may sound obvious, but that's the fate of all checklists :o)

Check the firmware of the device:

Is this the latest supported HBA firmware?

Are the drivers / filesets up-to-date and matching?

Any newer multipath driver out there?

Check the release notes of all available firmware / driver version for keywords like "performance", "buffer credits", "credit management" and of course "slow drain" and "slow draining".

If you found a bugfix in a newer and supported version, testing it is worth a try.

If you found a bugfix in a newer but unsupported version, get in contact with the support of the connected devices to get it supported or info about when it will be supported.

Check the configuration:

Is it configured according to available best practices? (For IBM products, often a Redbook is available.)

Is the speedsetting of the host port lower than the storage and switches? Better have them at the same line rate.

Queue depth - better decrease it to have fewer concurrent I/O?

Load balanced over the available paths? Check you multipath policies!

Check the amount of buffers. Can this be modified? (direction depends on the type of the problem).

Check the workload:

Do you have a device with just too much workload? Virtualized host with too much VMs sharing the same resources? Better separate them.

Too much workload at the same time? Jobs starting concurrently? Better distribute them over time.

Multi-type virtualized traffic over the same HBA? One VM with tape access share a port with another one doing disk access? Sequential I/O and very small frame sizes on the same HBA? Maybe not the best choice.

Check the concept:

If the slow drain device is an initiator, does it communicate with too many targets? (Fan-out problem)

If the slow drain device is a target, is it queried by too many initiators? (Fan-in problem)

Is it possible to have more HBAs / FC adapters? On other busses maybe?

Is the device connected as an L-Port but capable to be an F-Port? Configure it as an F-Port, because the credit management of L-Ports tends to be more vulnerable for slow drain device behavior.

Check the logs for this device for any incoming physical errors. Of course, error recovery slows down frame processing.

Check the switch port for any physical error. If you have bit errors on the link, the switch may miss the R_RDY primitives (responsible for increasing the sender's buffer credit counter again after the recipent processed a frame and freed up a buffer).

Use granular zoning (Initiator-based zoning, better 1:1 zones) to have the least impact of RSCNs. (A device that has to check the nameserver again and again has less time to process frames.)

If all other fails: Look for "external" tools and workarounds:

Does the slow drain host get its storage from an SVC or Storwize V7000? Use throttling for this host. Other storages may have similar features.

Brocade features like Traffic Isolation Zones, QOS and Trunking can help to cushion the impact of slow drain devices.

Have a Brocade fabric with an Adaptive Networking license? Give Ingress Rate Limiting a try.

Last resort: Use port fencing or an automated script to kick marauding ports out of the SAN.

The list above is just a collection of things I already saw in problem cases. Having said this, it might be updated in the future if I encounter more reasons for slow drain device behavior. Of course I'm very interested in your opinion and more reasons or ways to deal with them!
 One of the ugliest things that can happen in a SAN is a big performance problem introduced by a slow drain device (or slow draining device). Why is it so ugly? Well, if a full fabric or a full data center drops down - due to a fire for example - it's... 3 3 40608 urn:lsid:ibm.com:blogs:entries-75889db3-24fb-4697-98cb-78945ded3943 seb's sanblog 2017-11-07T00:52:26-05:00 urn:lsid:ibm.com:blogs:entry-3edc2263-a2a2-427d-b19a-1b9d951055dd Why you want to choose the optimal SAN Extension for your Brocade SAN seb_ 060000QVK2 active false Comment Entries application/atom+xml;type=entry Likes true 2011-06-28T10:31:22-04:00 2011-08-02T05:46:48-04:00
First of all: the following blog is about some SAN extension considerations related to Brocade SAN Switches. The described problems may affect other vendors as well but will not be discussed here. It will also not cover all sub topics and consideration but describes a specific problem.

There are a lot of different SAN extensions out there in the field and Brocade supports a considerable proportion of them. You can see them in the Brocade Compatibility Matrix in the "Network Solutions" section. As offsite replication is one of the key items of a good DR solution, I see many environments spread over multiple locations. If the data centers are near enough to avoid slower WAN connections usually multiplexers like CWDM, TDM or DWDM solutions are used to bring several connections on one long distance link.

From a SAN perspective these multiplexers are transparent or non-transparent. Transparent in this context means that:

They don't appear as a device or switch in the fabric.

Everything that enters the multiplexer on one site will come out of the (de-)multiplexer on the other site in exactly the same way.

While the first point is true for most of the solutions, the second point is the crux. With "everything" I mean all the traffic. Not only the frames, but also the ordered sets. So it should be really the same. Bit by bit by bit exactly the same. If the multiplexing solution can only guarantee the transfer of the frames it is non-transparent.

So how could that be a problem?

In most cases the long distance connection is an ISL (Inter Switch Link). An ISL does not only transport "user frames" (SCSI over FC frames from actual I/O between an initiator and a target) but also a lot of control primitives (the ordered sets) and administrative communication to maintain the fabric and distribute configuration changes. In addition there are techniques like Virtual Channels or QOS (Quality of service) to minimize the influence of different I/O types and techniques to maintain the link in a good condition like fillwords for synchronization or Credit Recovery. All these techniques rely on a transparent connection between the switches. If you don't have a transparent multiplexer, you have to ensure that these techniques are disabled and of course you can't benefit from their advantages. Problems start when you try to use them but your multiplexer doesn't meet the prerequirements.

What can happen?

Two examples:
Credit Recovery - which allows the switches to exchange information about the used buffer-to-buffer credits and offers the possibility to react on credit loss - cannot work if IDLEs are used as a fillword. They would use several different fillwords (ARB-based ones) to talk about their states. If the multiplexer cuts all the fillwords and just inserts IDLEs at the other site (some TDMs do that) or if the link is configured to use IDLEs, it will start toggeling with most likely disastrous impact for the I/O in the whole fabric.
 Another problem appears less obvious. I mentioned Virtual Channels (VC) before. The ISL is logically split. Of course not the fibre itself - the frames still pass it one by one. But the buffer management establishes several VCs. Each of them has its own buffer-to-buffer credits. There are VCs solely used for administrative communication like the VC0 for Class_F (Fabric Class) traffic. Then there several VCs dedicated to "user traffic". Which VC is used by a certain frame is determined by the destination address in its header. A modulo operation calculates the correct VC. The advantage of that is that a slow draining device does not completely block an ISL because no credits are sent back to enable the switch to send the next frame over to the other side. If you have VCs the credits are sent back as "VC_RDY"s. If your multiplexer doesn't suport that (along with ARB fillwords) because it's not transparent, you can't have VCs and "R_RDY"s will be used to send credits. The result: As you have only one big channel there, Class_F and "user frames" (Class_3 & Class_2) will share the same credits and the switches will prioritize Class_F. If you have much traffic anyway or many fabric state changes or even a slow draining device, things will start to become ugly: The both types of traffic will interfer, buffer credits drop to zero, traffic gets stalled, frames will be delayed and then dropped (after 500ms ASIC hold time). Error Recovery will generate more traffic and will have impact on the applications visible as timeouts. Multipath drivers will failover paths, bringing more traffic on other ISLs passing most probably the same multiplexer. => Huge performance degradation, lost paths, access losses, big trouble.

Conclusion

You see, using the wrong (or at least "non-optimal") equipment can lead to severe problems. It's even more provoking the used multiplexer in fact is transparent but the wrong settings are used in the switches. So if you see such problems or other similar issues and you use a multiplexer on the affected paths, check if your multiplexer is transparent (with the matrix linked above) and if you use the correct configuration (refer to the FabOS Admin Guide). And if you have a non-transparent multiplexer and no possibility to get a transparent one, don't hesitate to contact your IBM sales rep and ask him about consultation on how to deal with situations like this (e.g. with traffic shaping / tuning, etc).
 First of all: the following blog is about some SAN extension considerations related to Brocade SAN Switches. The described problems may affect other vendors as well but will not be discussed here. It will also not cover all sub topics and consideration but... 0 0 10049 urn:lsid:ibm.com:blogs:entries-75889db3-24fb-4697-98cb-78945ded3943 seb's sanblog 2017-11-07T00:52:26-05:00


text:
Have an opinion?
Merry Christmas to you, young lady :) <3
The day has been good and yes enjoyable :) trust yours has been too :)
yeah it was alright
enjoy the rest of your day
Thank you for the up vote and your kind words as well as wishes young lady <3 :)
no problem happy new years
Happy new year to you too young lady :) <3
yeah you too thanks :)
It's almost 9am, I haven't had time to enjoy it yet...
ooh then hopefully u've had a good day
Merry Christmas and yes I have enjoyed it
good have a rest of your good day
Thanks you too
You cannot undo this action. The opinion owner is going to be notified and earn 7 XPER points.

subset: commoncrawl


text:
Wedding Officiant in and around Cerrillos, Nm.
Life's Moments are those times in life that we decide to make a special acknowledgment of life altering occasions. Weddings are celebrations, weddings are events, yet most of all weddings are a new beginning, a minute in time to be cherished. Having the ideal Minister is very important.
For these moments, we provide thorough Wedding Minister solutions, Weddings officiating, and wedding event solutions. From the very conventional to modern services, our wedding celebrations as well as officiating solutions are created to show your desires, needs and also dreams. We can help you to craft an occasion, nonreligious or spiritual that will be a cherished memory for many years to come.
Pastoral services and also wedding event officiating in the Cerrillos, Nm area is our specialty, but we can travel if the requirement occurs. We offer a complete variety of Weddings plans; from elopement on the courthouse steps to one of the most intricate formal wedding you might want.
We also provide the finest in Justice of the Peace or secular weddings. These civil ceremonies provide their own form of celebration, whether it be with family members or just a couple and also a couple of witnesses, your wedding celebration is an unique moment to be cherished.


text:
Decided: Goldman v. Weinberger - The Supreme Court votes 5-4 to uphold a military provision requiring a uniform dress code which prohibits Orthodox Jews from wearing their religiously required yarmulkes.
According to the decision:
"Our review of military regulations challenged on First Amendment grounds is far more deferential than constitutional review of similar laws or regulations designed for civilian society. ...
[W]hen evaluating whether military needs justify a particular restriction on religiously-motivated conduct, courts must give great deference to the professional judgment of military authorities concerning the relative importance of a particular military interest."
Read More: Goldman v. Weinberger


text:
"Each of you should give what you have decided in your heart to give, not reluctantly or under compulsion, for God loves a cheerful giver." (2 Corinthians 9:7)
Giving (time, talents and resources) is central to the Christian life, it shows our Love for God's people, God's kingdom and our Love for Jesus Christ. When we act generously, we respond to Jesus' call to serve others, and we open our hearts to appreciate what God has provided. Giving sustains not only our mission and our congregation, but each one of us.
Thank you for helping First Presbyterian Church of Mountain View fulfill its mission and vision. Through your offerings, we are able to support the work of Christ in Mountain View, Silicon Valley, and around the world.
To donate to First Presbyterian Church of Mountain View using a credit card:
We welcome you to contact the church treasurer if you have questions or need more information about giving at FPCMV.


text:
Coming Out of the Shadows – Albinism in Fiji. Trailer
I'm very proud to present the first official trailer for the short documentary, 'Coming Out of the Shadows – Albinism in Fiji'.
I went to Fiji in August last year to shoot a conference in Suva – The First Fiji Albinism Awareness Symposium. It was organised by Sydney dermatologist Margot Whitfeld, who has done a lot of work and research in Fiji. Her sister Anna Whitfeld is a friend of mine and it was through Anna I heard about Dr Whitfeld's work and the upcoming symposium.
With very little preparation time before flying to Suva, Anna and I decided it would be great to make a short documentary as well as the conference videos I was doing for Dr Whitfeld. It seemed such a fascinating story.
We weren't wrong there.
I'll let you know more about it as I progress through the edit.
I hope you enjoy the first trailer.
Director, camera, editor: Christine Nestel
Research: Anna Whitfeld
Drone footage in the opening sequence: Isaac Alongi and Alistair Ferrier (used with kind permission)
Produced by Backyard Films & Video
Leave a Reply