text:
As another man teased him for not finishing his meal, Joseph Jones chuckled as he leaned back in his chair.
"I ain't going to leave it; I'm going to eat it all, you know that," the 73-year-old said. "There won't be a drop of it left here soon, I promise you that."
Jones, who often lives at the Inner City Night Shelter in the 100 block of Arnold Street, was among thousands of homeless and working poor Savannahians to attend Savannah Feed the Hungry's Christmas Dinner on Saturday afternoon in Forsyth Park.
Click here to view photos from the event
It was the first time the event was held in the downtown park. The Rev. Carl Gilliard, the charity's CEO, hoped the location would help bring some recognition of the staggering number of homeless in the area.
"What I'm praying for today is that Savannah will see the total need for the homeless and the working poor, and that people will really get involved," Gilliard said. "We're expecting between 8,000 and 10,000 people here today. For me, you see people you grew up with and people who were above you in their careers - we see them out here. Then you see the people that are working poor, and they're really struggling."
But the event was about more than just providing a hot meal, clothing and toys to the people who so often go without.
Gilliard said every person who signed up for a meal and took a bag of produce with them will be monitored by the organization and offered opportunities to advance themselves.
"We don't want to just give them food, food, food. We want to empower them, and I'm very serious about that," Gilliard said. "So many times people come here to Forsyth Park and they just give them food, give them food.
"We've got to be able to make them be uncomfortable so they'll move out of being career homeless and they'll be empowered."
Savannah Feed the Hungry is offering the people it serves medical services, counseling and the opportunity to explore education.
"That's what today means for us," Gilliard said. "Feed the Hungry is going to go in a whole different direction. We can't just give out bags (of food) anymore; we've got to be able to lift some people out of this situation."
For those who have struggled through homelessness or poverty it means a lot, said Miriam Lewis.
"A lot of people out here don't have a lot; they don't have homes or opportunities," the 66-year-old said. "It means a lot to us that someone would provide us with this meal today and that we know that there are still good folks out there that care."
Gilliard challenged Savannah to get involved with helping those in need, especially after the holiday season, when charities see a huge drop in donations and volunteers.
"You can see the need, and you can see we've got to do more than we've done," he said. "The test will be in January and through the rest of the year what the total community will do."
To volunteer or donate to Savannah Feed the Hungry, call 912-349-0774 or go to savannahfeedthehungry.com.


text:
Images tagged "transport"
0 responses to Images tagged "transport"
the first step is done. They will eliminate current education/certification program. No ILT, no labs, no serious certification…just WBT with simple tests. Someone might say it's my Brocade instructor point of view. But I think it's Broadcom's lack of expertise in end user products. They simply don't know how to sell such complicated products, teach to manage them and support.
I'm afraid you may have hit the nail on its head. I certainly hope not but we'll just have to wait and see. Broadcom simply has a history of engineering OEM relationships and have never seen a customer environment before.
I like your style of writing and summary of FOS from 8.0.0 to 8.0.1!
I'd enjoy reading an update of FOS 8.0.1 to 8.1, now that it is out.
Hi Nobody,
Thanks. I'll see what I can do.
what about this latest news… ARRIS to Acquire Brocade's Ruckus Wireless and ICX Business
prepared for brocade certs… but now its wasting money…
Hi Lucas,
Indeed. You just may hope that Arris takes over any qualification and certification you may already have obtained but I do question the current market value as Arris certifications seem to be non-existent. This would indeed mean that any certification you may hold on the Ruckus/ICX side of Brocade's business is void and yes, most likely has been a waste of time and money.
I have lost access to this Config section paid yesterday just after clicking a link to a topic from Troubleshoouting section and following a free registration
It is ugly!!
(trying to post from a longly opened browser window)
good review/look at it, assuming the IP business stays whole Great point on the Certs :-/
Great info Erwin. Cheers, Paddy
No worries matey…
"I wouldn't be surprised if an anouncement is made by Broadcom and Qlogic for a similar merger. With the acquisition of Brocade they not only leveled the playing field but leapfrogged Avago here by several miles."
This seems a rather strange statement since Avago purchased Broadcom last year so they are one and the same company.
Indeed but the problem is they are still different entities. Engineering efforts are not always shared. We'll see how it pans out. If Broadcom also decides to buy Qlogic because of the Brocade technology (Brocade sold the HBA business to Qlogic) there will no more competition left in that area. Good this is that FC HBA's can then finally be seen as a mass-commodity and prices can drop sharply as it has done with Ethernet NICS before. The HBA ASICS can then simply be OEM'ed to whoever wants to bring it to market including all the current networking and storage vendors. An exciting time to see how this goes in the near future.
Pingback: tim_txcrd_z What is it? What does it represent? | Storage & Beyond
Pingback: tim_txcrd_z What is it? What does it represent? | Storage & Beyond
Thanks Erwin, these stale entries indeed prevented me to perform an upgrade.
No problem. Glad the post helped.
Can old fashioned zoning and TDZ/PZ coexist in the same fabric?
Hello marksman,
Yes, that is no problem.
Good stuff, Erwin.
Can you elaborate on this section a bit?
"…simply hooking these [4G/8G devices] up to a new 16G or 32G switch will most certainly provide you headaches with performance (or rather the lack thereof) and wide-spread fabric congestion and latency issues. At some stage, sooner rather then later, it will cause significant problems."
Is the implication that if you mix these in with newer 16G/32G N_Port devices that you'll run into trouble? Or is this more a statement about possible issues with cable type/quality? Just looking for a little clarification.
I used to support fabrics from back in the 1G days until just after the 16G stuff came out, but have been out of the game for awhile. I still work on fibre target devices though, so I like to try to keep up. Thanks.
Hi Jay,
The problem is two fold. First the change in physical cabling requirements. 16G and 32G speeds simply require OM3 or better OM4 cabling. No questions asked. If you want to drive a Ferrari to go 250mph you need a good road. Same goes for these high speed links.
Secondly the chance of credit back-pressure when mapping 4G devices to each other which traverse ISL's or switch back-end links who also need to push frames from brand new kit will most certainly happen. You will run into scenarios like I described here http://22.214.171.124/2015/04/cross-fabric-collateral-damage/
The basic rule of thumb I always use is to have a maximum on 1 speed generation difference between any piece of equipment. Remember that speed increases are exponential in the FC world and so will the amount of performance problems if mixing old and new equipment.
Hope this helps.
Pingback: Brocade FOS version 8 and 32G hardware | Storage & Beyond
At large scale, I would hope that outside orchestration would be solving the complexity problem.
actually I have several customers that are pretty excited about it.
Does the entire fabric have to be running FOS 7.4.x before you can use TDZ? I have some old brocade 4020 in ibm blade chassis', that are stuck at 6.2.2.
Yes, the entire fabric need to have that installed. It has to do with the way the CAM tables are programmed in the ASIC and that code is simply not available in older code levels.
Very very useful article. Never found in Brocade a document where the initiation of a port were better explained. Thanks a lot
Thanks Tuco. Appreciate the feedback.
F1 has been lost at sea 4 years. It'd be a better sport if they raced Hyundai Getz around the Woolies car park with big spoilers 🙂
excerpt from Brocade San Fabric Resiliency best practice: "Once set by the configure command, the EHT value will be maintained across firmware upgrades, power cycles and HA Fail-Over operations. This is true for all versions of FOS."
So the example you described above "Now what happens if the storage admin decides to upgrade the core-switches to FOS 7.1.0x.??? The EHT will drop right away to 220ms since that's the new default on that FOS version." will never occur according to Brocade.
But I see a different challenge, which I have faced. People who installed new hardware in their infrastructure were not aware of eht and used default 220ms (partly on the core switches and partly on the edge), and they still have a lot of old switches with eht set to 500ms at the edge. As far as I understnad we can not change eht on the fly. only by disabling the switch. so what should these guys do now?!
Hi Erwin. And one more question: do you know if Golden eye 2 supports Edge hold time?
I'm not 100% sure but I think it only works on Condor ASICs. I'll try and get some clarification.
thank you. And I also wonder if blade chassis switches in AG mode will support this feature.
Good info. Erwin. Thanks !
But issue is not stopping only on this ( APM ) level. If there no timebase configured for some mapsrule(s), then also it failing.
Example error during upgrade from 7.3.x to 7.4.x:
The following item(s) need to be addressed before downloading the specified firmware:
Invalid MAPS rule(s) in FID 128:
There's no timebase for the above MAPS rule(s). Please use "mapsrule –config -timebase " command to correct the same.
i do agreee with Erwin on that. Better focus on correct routing configuration than think that Traffic isolation will really protect better our bandwidth.
We have been using SAN routing since year (2008) to interconnect our Datacenter through MAN link but we wanted to segregate the Tape and Disk traffic to different Path.
We start using Zone Traffic Isolation for EVA replication and we pushed Brocade to accept the Fabric Level Traffic Isolation for tape traffic as we had too many source and target (Lanfree agent to Tape devices) to do that zone by zone (even forced them to write it down in the Admin guide).
Now that we move to 7840 with a mix of FCIP and IP extension we will drop this very complex feature that was better for the sake of management than for real protection of our prod environment.
I am currently using peer zoning in some of my development environment fabrics and it is a really great addition and one that should have been implemented a long time ago.
The one major problem with peer zoning in its current format or at least in Brocades implementation of it, is that it will not allow the use of aliases. Which I'm sure you'll agree, increases the administrative burden that it was meant to ease and reduces user friendliness .
I can only hope this gets resolved in the very near future before I will consider the migration of zones from traditional to peer in my production environment.
Hi Stephen,
I agree that the use of aliases would have been a nice addition. You have to remember though that peer-zoning is simply piggy-backing on TDZ actually being the underlying idea. You don't need any administration for that on the switches as it is the arrays who will add and remove zone-members as needed. Peer-zoning is in my view a simple intermediate step until all the array vendors start supporting TDZ.
Hello Stephen,
A short followup on your comment above. As you may have seen Brocade added the alias naming convention to the peer zoning functionality in FOS 8.1.0. You just have to wait a short time now till the OEM's have qualified this version.
Hope this helps.
Hi Erwin. Some of our switches show value of "0" – switch.edgeHoldTime:0. what does it mean? they are models 5100 FOS 7.1.1c and 5470 FOS 6.4.2b4.
Thats simply means they use the default of what the FOS-code is at that time. There have been some changes in that value over time. It's best to set it manually. Be aware that on pre-condor3 ASIC's the value is set per ASIC and not per port.
How many frames exactly in the buffer that the switch can hold?
I must say you have a peculiar name.
My name is Erwin, I think I'm a nice guy and would like to speak on a first name basis so if you would provide your name that progresses these kind of conversations much better.
First of all your question is incorrect on two parts. You did not mention which switch and secondly it is not the switch that has the buffers but each ASIC in a switch or blade. This then lines out the number of buffers in a switch but these are not shared in a single large pool but more per port-group. As you question is unclear I'm unable to answer it.
Hope this helps.
Hi Erwin
Thank you for your time to reply my questions. You may refer me as Shafiee, sorry for the silly question. i'm an amateur in SAN with only +2 yrs experience previously i'm a Unix Guy 🙂
In our current environment we have HP Brocade B-Series:
0. We have DCX, DCX 8510-8 & 5300.
1. Currently running on FOS v7.1.1a.
2. We have 4 Virtual fabric in total & are separated:
2.1 VF Trunking & ICL
2.2 VF External Storage – XP having external storage from EVA
2.3 VF Production – Server & Storage
2.4 VF 128 – Default
3.Fabric Watch enabled.
So each front end & back-end ASIC has buffer credits, if so how much? Just curious to know.
Recently we had performance issue so i came around your post "One rotten apple spoils the bunch" Pt 1 – 4, been reading it for a couple of days. FYI the issue is currently ongoing at the moment.
From my observation:
1. disc c3 value kept increasing on affected device connected after running portstatsclear & run porterrshow on port in question.
2. From BNA i can see there's a perf bottleneck congestion alert.
3. When running errdump -r on the switch, I came across this two important message.
2015/11/24-13:39:28, [C2-1014], 646086, SLOT 7 | CHASSIS, WARNING, XXXXCORE001, Link Reset on Port S8,P-1(57) vc_no=15 crd(s)lost=3 auto trigger.
2015/11/24-13:39:28, [C2-1012], 646085, SLOT 7 | CHASSIS, WARNING, XXXXCORE001, S8,P-1(57): Link Timeout on internal port ftx=336197441 tov=2000 (>1000) vc_no=15 crd(s)lost=3 complete_loss:1.
/Shafiee Van Weiringen
Attaching some screenshots:
FID1 – Is the VF Trunking & ICL
I think you should open a support-request with your vendor as it might be something triggered by one or more defects.
They should be able to help you here.
This may be slightly off topic. In another case the extended distance trunks(invovled DWDM) were formed successfully. However, if is being noticed that it is utilizing almost one link of the trunk as against distributing it dynamically(and in-order) between the links(2 extended ISL links).We are sure though that it will re-direct in case the first one fails. but why not distributing it already in the first place… Please advise….Thanks Bacil.
Hi Bacil,
As you are aware traffic is dispersed over ISL when Exchange based routing is enabled. When an ISL consists of trunks for FSPF this is one link so from an official Fibre-Channel standards point of view there is not much that can be done and it is up to the implementation of the vendor.
As for you question as of FOS 6.2.(something) Brocade changed the switching algorithm in the ASIC driver in a way that a threshold mechanism was introduced regarding load balancing. The challenge has always been to keep frames in order for each FC exchange. If you just have a single link (non-trunked ISL) there is not really a problem as FSPF will take care of that as the entire exchange is mapped onto a single ISL. If you have two or more physical links on a single ISL you need to make sure to prevent Out-of-Order frames as much as you can. The easiest way to do this is to keep frames using the same physical link as it is therefore impossible to get frames out-of-order. On 1, 2 and 4G speeds the lentgh of each individual frame was long enough for the receiving side to keep things in order as it would not overlap if the cable length difference was short enough. On 8G and now 16G this may not be 100% the case and preventative measures where taken by the Brocade FOS engineering team.
The driver now makes sure to map all frame onto the same physical link until a certain utilisation in the form of bandwidth or latency has been reached and then starts using the other links in that trunk on a per exchange basis. This way the chances of getting Out-of-order frames in the same exchange is next to nothing.
I got this question a lot in the past and it is just a cosmetic thing when you see one link utilised. In reality there is no difference in the total amount of data being pushed through the entire trunk. It should just be seen on the whole trunk and not the individual members.
Hope this explains it a bit.
Thanks for the detailed explanations.
No problem.
Great post. I supported fabrics for Sun/Oracle for 12+ years and remember learning about FDMI so long ago and being excited about the possibilities it could bring to administration and troubleshooting. Alas, it never seemed to really catch on and I'm pretty sure Q still does not include it in their Solaris driver and Emulex's implementation (for Solaris) just gives some of the basic static info.
Indeed, so much more administrative and troubleshooting information could be used from just having this feature enabled. It is indeed a shame that Oracle does not instruct Emulex and Qlogic to include the options in their drivers. The code is all on a FC3 level which both vendors already have on numerous platforms so I'm quite curious why Solaris is being kept out the zone…
Hi, We want to use MAPS features in new FOS. However, few of our switches still have FOS below 7.2(actually v6.4.XxX) and due to hardware limitations on those switches(48Ks), we can't have new FOS for MAPS, and we still have to live with them for some more time. My question is , can we have in a same fabric , MAPS on all and two three switches with still Fabric Watch. Okay, while converting Network Advisor will complain for them, but is there any other drawback apart from that. I mean since MAPS or FW are configured on switch basis, so at the most we will be unable to manage them with MAPS in Network Advisor. Apart from that what other issues you see in this scenario…
I also found this –
"Fabric capability is based on the least capable switch participating in the fabric. If a fabric has products participating that are operating with an older version of Brocade FOS, the limits of the fabric must not exceed the maximum limits of that older version of Brocade FOS."
But then again, I thought it will not be managed by MAPS, that's it. We will manage it via CLI and Fabric Watch alerts anyway we will receive via email alerting set on those individual switches.
So, what are your thoughts on "mixed" non-supported environment like this ?
Hello Bacil,
If I were you I would shift my focus on replacing these 48K's to newer equipment as they are simply not longer supported as of January which leaves you no option to open any support-cases with any of the OEM vendors nor Brocade.
As for the use of FW and MAPS you basically can not use any fabric wide feature of MAPS or FW. You can obviously still use the features and functions that monitor and action an individual port or switch. There is no restriction using these in a mixed environment.
Once again, please get rid of these 48K's especially when they are connected in mixed fabric you basically render your entire environment unsupported. You certainly don't want that. Believe me I have experiences in that where a customer had even 24K and 12K directors installed not too long ago. His environment went down hard without us being able to do anything about it. This became a very costly exersize as all equipment had to be replaced in addition to numerous down-time occurrences.
Hope this answers your question.
Hi, We want to use MAPS features in new FOS. However, few of our switches still have FOS below 7.2 and due to hardware limitations on those switches, we can't have new FOS for MAPS, and we still have to live with them for some more time. My question is , can we have in a same fabric , MAPS on all and two three switches with still Fabric Watch. Okay, while converting Network Advisor will complain for them, but is there any other drawback apart from that. I mean since MAPS or FW are configured on switch basis, so at the most we will be unable to manage them with MAPS in Network Advisor. Apart from that what other issues you see in this scenario…
I also found this –
"Fabric capability is based on the least capable switch participating in the fabric. If a fabric has products participating that are operating with an older version of Brocade FOS, the limits of the fabric must not exceed the maximum limits of that older version of Brocade FOS."
But then again, I thought it will not be managed by MAPS, that's it. We will manage it via CLI and Fabric Watch alerts anyway we will receive via email alerting set on those individual switches.
Good post, but with one erroneous entry. The "firmwarecleaninstall" will not let you update from "stoneage" FOS to 7.4. From the command reference guide:
"Use this command to initiate a clean reinstall of the firmware in cases where the loaded firmware does
not function correctly, the normal firmware download fails, or to recover from a rolling reboot situation."
It is only used in case your 7.4 firmware is all messed up; not to update lower level FOS versions.
Hello Spencer,
You are correct. I thought I was clear in my description but apparently there is room for misinterpretation. Obviously when a feature is introduced in a certain revision I assumed it was clear that this is not available in preceding versions.
So to be clear a switch has to run at least FOS 7.4 before being able to use this option. Future releases will then be able to utilise this in case things have gone horribly wrong for whatever reason.
I still have an RFE out to be able to boot a switch over tftp right out of the bootprom and recover from there. Haven't heard about it for a long time. Will send them a note around this..
Suppose that there is need for checking HBA queue depth
Hi Yury,
That may well be the case. We haven't been given much info on this system when I wrote this article. max_execution_throttle was not set so I don't know what the default is on this system.
Enhanced Group Management license is needed for some BNA management tasks. It allows you to perform operations on a group of devices. For example, recently I was able to upgrade firmware on all switches in the fabric at once except one with the lack of the EGM license. I suppose that massive config upload and download requires the EGM as well.
Hi Yury,
I'm sure you're right and there may be other tasks/feature that require this license however I haven't been able to track down a detailed list. Anyway on all 16G platforms it is included as well as all director class switches from FOS 7.x and above. Most of the time the Enterprise bundle is purchased anyway which also includes this so in the end most of us will not have any issues with that.
Hi Erwin! Two questions:
1) Why don't u multiply 346392029 by 4 as well? This counter is for 4-byte words.
2) Why u multiply 55*4294967295 by 4? I think u shouldn't as these values are for frames
You are correct. I don;t know why I did write it that way as my (hds internal) post has the correct one. 🙂
I modified the formula and it should be OK now.
Thanx Erwin! Great post in the area of SAN audit
Thanks Erwin, a good read, as always. However, this is not the only problem. See DEFECT000491910. We suspect that we've hit this one recently.
Hello Alexei,
This may indeed be a problem. As always, maintaining code-levels can prevent these issues.
We've qualified 7.3.1a for a while (as of this writing) and this particular defect is closed with code-change on that level.
Thanks for your comments. Highly appreciate3d as always.
hello Erwin,
I have downloaded some supportsaves (from two Brocade switches), in order to troubleshoot a LUN disconnect.
Do you know if there is a Tool that makes sense of all list of logs in it?
basically I am looking for the Log showing any port status change within the last week.
in which Log should I look for?
Hello Igal,
I wish there was. The tools that are there are a collection of scripts and tools but these are all Brocade or OEM restricted. The log you're after is displayed on the screen if you type in "fabriclog" in the CLI. This provides you with a list of all port-state-machine changes and is basically a circular log which means it has a finite number of entries (that depends on type of the switch and the FOS code revision) but in general contains enough entries to go back a fair bit in time. That is unless you have a very active fabric where lots of hosts and link issues are ongoing.
Hope this helps.
yes, that helped, thanks!
Unfortunately, the log was full –
Number of entries: 1024
Max number of entries: 1024
so I am not sure I see the latest entries (the last is from Dec 2014):
Switch 0; Wed Dec 10 08:42:19 2014 GMT
08:42:19.745618 SCN Port Offline;g=0x17a D2,P0 D2,P0 21 NA
08:42:19.745636 *Removing all nodes from port D2,P0 D2,P0 21 NA
08:42:19.813833 SCN Port Offline;g=0x17c D2,P0 D2,P0 20 NA
You can resize this log. fabriclog -r should do that. The limit depends a bit on the platform and FOS code level.
Looks like a very improved version the 7.4.
Not yet tested…I will add comments when done.
thanks for the post Erwin.
I'm seeing mass errors across my Brocade encryption switches currently all ports are set to 0 idle/ildle. The problem we are seeing is when a cfgenable is run some of the esx hosts see their storage connections become unstable. To recover the issue the port is disabled and the device recovers on the other connections to the SAN. Currently the storage is all HDS (vsp and hus) the major issue appears to be with the HUS as connections stay alive to the VSP.
We are planning to change the HDs devices to portcfgfillword 2 and the ESX hosts to 3. Is this a red herring or are we following the right path. any advice appreciated.
Hi Tony,
In order to rule out incorrect fillwords having an effect you should set them according to manufacturers guidelines. You didn't really specify WHICH errors you see as a bad-ordered-set is not necessarily a hard error but it can have a negative effect which then results in these hard errors.
Let me know how it goes.
hey Erwin the only error we are seeing is "er_bad_os 730762736 Invalid ordered set" the stat was cleared on Thursday last week and is happening on 20 ports on the switch and seem to be about 1 billion per port/day. We are working to fix them and should have them done by the end of the week.
Yes, that is the expected counter to ramp up when an invalid fill-word is set. If you set the correct fillword as I described these will stop to accumulate.
Thank you for such a good summary of all the development in this area in the last 3-4 years!
The only thing that is missing is … just a little bit of spell-checking 🙂
NPIV with Brocade 6240 and FlexFabric not working
Guys i have to migrate from B5300 to 6240 switch and when a working NPIV port was moved to 6240 i am not getting any wwn logged into behind the flex fabric.
1. Port itselft is online and logged in
2. NPIV is on, NPIV PP Limit =126, NPIV FLOGI Logout ..
3. FOS 7.3.1 on new switch, fill word not required with this version of fos and is depreciated.
4.portFlags: 0x24b03 PRESENT ACTIVE F_PORT G_PORT U_PORT LOGICAL_ONLINE LOGIN NOELP LED ACCEPT FLOGI ( Notice missing NPIV)
5. NPIV is ON on the port.
I think the Flex Fabric is configured to manual log in redistribution.
Please help.
Hello Saiyed,
Can you clear the portlog and bounce the port once (portdisable/portenable) and show the portlogdump for that port. This might provide some clues. Be aware this might not provide all details for that login process but it would give me something to look at.
Secondly be aware that if a base device logs out the NPIV address will also be removed. You can change this behaviour with portcfgflogilogout –enable -all command. I think that is your best bet for now.
If that doesn't work I would really need to look at the portlogdump or, if even that doesn't provide a clue, a FC trace. Your support-provider may be able to help you with that.
I have reconnected the NPIV node and removed it couple of times, right now its not on this fabric, the log is during this actity.
So you are saying enable the setting below.
NPIV PP Limit: 126
NPIV FLOGI Logout: OFF ( Turn this on)
Hello Saiyed,
Did you check with HP if anything needs to be done here. From a Brocade perspective it seems to be OK and there is not much you can do. If an HBA (or in your case teh HP FlexFabric switch) logs into the fabric it needs to set word 1, bit 31 of the common service parameters in the FLOGI request to 1. That triggers the switch port to allow FDISC ELS request in order to address multiple ALPA's behind the port-ID.
This may sound a bit technical and unfortunately there is no way to see this in a switch log and you would need a FC analyser to see if this bit is turned on during the FLOGI request from the HP FlexFabric. The picture below shows the bit that needs to be 1 if NPIV is used. By default it is off.
however, it should be turned off by default. but if it's there, you can always start using it!
very good document, solved my issue with Storage Provision, where only SMI-S agent of BNA used for software to switch communication and JVM restarted all times, increasing heap size to 6GB stabilized it
hi all, i have FOS 7.3.0c with brocade 6520 and i have tried all the mentioned settings & i am still having java issue, i have tried, JAVA 7.60, 7.71, 7.67 & 8.25 with errors such as "bad request details" i have opened a case with Brocade.
More specific error below.
Missing Application-Name manifest attribute for: http://192.xx.x3.xx/wt-app.jar
log4j:WARN No appenders could be found for logger (com.brocade.wtcommon.access.AccessController).
log4j:WARN Please initialize the log4j system properly.
Hello Saied,
According to the release notes the webtools app on FOS 7.3.0 should be compatible with JRE 1.7.0 U60. Have you tried to clear the JRE cache? I've seen these sorts of incompatibilties pretty often and in many cases being on the correct JRE level with the correct setting and a clear cache often resolves the issue.
I'm curious what Brocade finds out. If you get any feedback from them please let me know.
Update: issue is not resolved.
1. reenabled the swtich after disable
2. reran configure, please see the the output and advise if any thing should change.
Fabric parameters (yes, y, no, n): [no] no
Virtual Channel parameters (yes, y, no, n): [no] no
F-Port login parameters (yes, y, no, n): [no] no
D-Port Parameters (yes, y, no, n): [no] no
RDP Polling Cycle(hours)[0 = Disable Polling]: (0..24) [0] 0
Zoning Operation parameters (yes, y, no, n): [no] no
RSCN Transmission Mode (yes, y, no, n): [no] no
Arbitrated Loop parameters (yes, y, no, n): [no] no
System services (yes, y, no, n): [no] yes
Disable RLS probing (on, off): [on] on
Portlog events enable (yes, y, no, n): [no] no
ssl attributes (yes, y, no, n): [no] no
rpcd attributes (yes, y, no, n): [no] no
cfgload attributes (yes, y, no, n): [no] no
webtools attributes (yes, y, no, n): [no] yes
Basic User Enabled (yes, y, no, n): [no] no
Perform License Checking and Warning (yes, y, no, n): [yes] yes
Allow Fabric Event Collection (yes, y, no, n): [yes] yes
Login Session Timeout (in secs): (60..432000) [7200]
Found the following:
Looks like the HTTP/HTTPS daemon in the switch had encountered errors and not pumping the data properly.
As a workaround try to restart the switch http/https daemons this can be done by executing the following commands from CLI
execute "configure" from CLI
select "http attributes"
disable HTTP/HTTPS
exit out of configure.
execute "configure" and enable the HTTP/HTTPS services.
PS: This just restarts the HTTP/S daemons and does not affect the switch functions in anyway.
Note: I did not find the same setting on FOS 7.3.0c and 6520
may it has changed or burried in some other config.
4. I changed the switchname to remove _ and keep it at simpler name just in case it was affecting java.
I have sucess finally without a lot of changes.
1. Uninstalled all Java from my desktop
2. Reboot
3. connected to the switch without any java, it prompted for Java 6.25 ( Strange), i allowed it to download it, then relaunched the connection, broswer complained about out of data Java and asking for 1.7+ ( it should be noted that when connecting ie it downloaded 6.25 but firefox redirected me to update the java @ 8.31-current).
I ignored and updated from my privious download to 7.60 ( supported by brocade for this switch).
4. Reconnected to the swich, this time it complained about out of date but i said proceed, meanwhile browser add o n popp up were going on about the enable add on with some sort of SE add on ( SSV Helvper & SSV2 plugin helper), i allowed them.
Then after the 1.7.60 install completed, the webtools loaded.
Been fighting this for two days, i really wish Brocade & Java fix this type of issues its giving them bad name.
Also i am not happy that Brocade does not allow access to thier KB support site, even thought i have mybrocade log in and we own several brocade product because we buy it from 3rd party-Not a good policy in my openion
Hello Saiyed,
Good to see it's finally working. As you can imagine the issue is mainly with Java and Oracle's release and security policies that seem to keep changing ever month. The java engine and plugins change in such a rapid pace it's almost undoable to keep up-to-date with it. As you mentioned the Brocade app requests a certain level of the JRE but as soon the browser requests that version you first run into a browser check which flags that version as being insecure or incompatible (for whatever reason) and either blocks it entirely or it redirects you to Oracle's Java webpage which, more or less, gives you no other option than to download and install the latest version.
I always liked the webtools app but not for management purposes. I learned the CLI pretty quick and found that it's not only much quicker but also more reliable adn your not depending on third party versions and related conflict you may run into.
As for Brocade's portal I cannot comment on that as I may have different privileges than you and I'm not in a position to make changes to their policies.
To be honest, I had no intend to down play anything. I do strongly believe though that things do not happen over night. In your article you more or less elude to his, maybe without realizing:
"Companies like E-bay, Spotify are happily harnessing the power and flexibility of Docker. Also well-know search engines like Yandex and Baidu are using Docker for a multitude of functions."
The reason companies like e-bay etc are successful in deploying these types of technologies is because they have a completely different application stack then most enterprises. You want to ask yourself next why it is that virtualization has been so successful, and in my opinion that is mainly because it offered a couple of things :
1) higher utilization of resources
2) mobility
3) availability
And it did this without any disruption to the workload, no changes to the application or even OS required besides the installation of some drivers. That is the big difference with what you are describing and what companies like e-bay are doing.
Now don't get me wrong, I do believe the world is changing. Containers but more importantly distributed application architectures are the way of the future, but knowing how many enterprises operate and how little control they (unfortunately) have over a vast majority of their application stack I also realize that it is not realistic to expect this to change drastically within lets say 5-6 years, at least not for the portion they do not own internally. Again, yes the world of IT is changing… but hasn't it always been?
Hi Duncan,
Thanks for your response. I agree the success of VMware was partly contributed by your arguments but it has been mainly skyrocketing due to the lack of competition in that niche market for the entire 2000/2010 decade. That market has not really changed from a technological perspective but additional choice in hypervisor technologies from a multitude of vendors are fishing in that same pond now. Given the fact that currently the maturity of the VMware hypervisor, its contributing technologies and eco-system make it an obvious candidate on each short-list.
The companies mentioned in Docker's "use-cases" have the advantage of being able to target very specific applications which makes adoption easier. The same thing we see here in Australia where Openstack is very well represented in the education sector. The major threat for VMware is however the fact that vendors like Oracle and SAP are in a very comfortable position when it comes to shifting their products into containerized formats and do not longer need to take OS interoperability into account. That means much shorter dev-cycles and therefore much improved code resiliency. As soon as that happens the majority of main-stream business applications are available for current customers who run this on hardware or VM's and that fact, combined with an even less resource hungry virtualisation stack like Docker and AWS Lambda, will certainly eat big-time into VMware's sales-number. As I said, when push comes-to-shove in the end it's about the $$ and if a CIO can save by squeezing even more juice out of his/hers hardware they most certainly will.
As for time-lines it's always hard to predict. I don't have insight into roadmaps from SAP, Oracle and other major companies in this space. I would not be surprised though that they have been exploring these kind technologies for quite a while now and all of a sudden whack products on the market which fit in this space. Customers who have been exploring the same technology will most certainly adopt very quick.
Again, thanks for your feedback.
Personally I am not sure the overhead difference is that big, we are primarily talking disk space here for now and with advanced techniques like VMfork and smart linked cloning technology even that will diminish. For many enterprise companies there is no benefit in having 2 or 3 different operational strategies, it will simply not be cost effective and will be too complex. On top of that there is the whole security aspect, companies like Google run their containers in a VM for a reason.
Anyway, this is an interesting discussion… but with your background I do wonder how you feel how this will impact the world of storage. Especially with this new highly distributed application architectures the need for expensive shared storage systems start to disappear for a large portion of the stack. How will Docker, but more so the distributed architecture, eat your lunch?
🙂 Replying whilst eating lunch… haha..
I can't comment on what Google is running or not. Under NDA here.
Its fortunate enough I work for a company who's bare-bone roots lie in large-scale compute infrastructures (mainframe compatible kit) and a huge background in storage but the main benefit is that we (as HDS) are not solely tied to compute infrastructures and related materials. As a subsidiary of Hitachi we have the benefit we tie in with a massive amount of diverse, technology related, sister companies which has opened up a huge playground of opportunities, City smart-grids, health-care, logistics, manufacturing technologies and much more in a single house which will all be tied in with the Hitachi logo stamped onto it. The onflow of these market sectors will ensure that the compute platform such as Hitachi UCP will be cemented to support these.
My diet may shift to a reduced lunch but breakfast and dinner will be much more …. interesting. 🙂
You may want to check out https://community.hds.com/community/innovation-center
Some sneak previews are there or will be published soon.
I agree with your analysis.
With all respect due, there is nothing visionary here, it's pretty much common sense.
Innovate, monetize, lather rinse and repeat. Or die.
What's new?
Hello Massimo,
Thanks for agreeing. I do however tend to disagree on your second statement though. Innovation comes from vision. You see opportunities in new markets or have the ability to create something much better than something that is already out there. Henry Ford did that with cars, Steve Jobs with phones, Bill Gates with DOS (the latter based on new markets, not an improvement in quality of products. :-))
Again thank you for your comment.
I was referring to the vision required to underline this pattern, which is common sense.
Obviously the vendor will need to have a vision to keep innovating.
Looks like there is no way to predict the compression ratio and average frame size before the compression enabled, am I correct? In that case, we should do a best effort estimation first and then run the configuration for a while and collect the actual data (compression ration and average frame size) and do the fine tune, right?
Hello Kelvin,
Yes, that is correct. I've been involved an fair few cases where we had to start off with the assumption of no compression at all and work our ways towards a more realistic number. Especially in FCIP cases where WAN links are involved this may be the best method to go. It will also allow you to reduce your leased bandwidth to that number and save some money.
Hi Erwin, thank you very much indeed.
Hi Erwin,
Where did u get the official standards for the chart above regarding the fill modes? Do you have any links?
These are based upon the official FC standards. You may see vendors advising differently especially when it comes to mode 3 and 2.
I've got 3 HDS G1000 with 8gb ports .. what is the best practice for fillword please ?
Hi Stef,
As per T11 fibre-channel standard the switch-ports on a 8G switch (DCX/DCX-4s or any Goldeneye2/Condor2 based switch) connected to any HDS array should be set to mode 2.
Hope this helps.
Erwin, you say "based on the QOS zone mapping and if contention occurs the priorities of the frames being sent over the ISL is set to a 60%/30%/10% ratio. " – is 60/30/10 ratio something what is hardcoded and not customizable? When we asked Brocade about how QOS internally works they never said about such ratio.
I thought this ratio is valuable always and QOSH zones traffic have priority on ISLs just because VCs 10-14 have dedicated 5 buffer credits each (when QOSL zones have only two of them and QOSM can borrow credits). Now I'm bit confused.
Hello Roman,
This is not new and has been in FOS for a long time. I'm curious why Brocade is so secretive on this. It is mentioned in their training material. Be aware this algorithm only kicks in when contention occurs and is fixed in the ASIC code. Although you have virtual channels the link itself remains a serial line and only one frame can be sent after the other. If you have multiple frames arriving at the switch from different originators which are mapped to traverse a particular ISL. The order in which they will be sent is depending on arrival however if each of these ingress buffers of the originators is filled up the order on which the frames will be sent onto this ISL is in a 60/30/10 ratio. So you will get 6 frames if the destination is in a QoSH zone, then 3 frames when it is in a QoSM zone and 1 frame on a QoSL zone. This is to prevent locking a higher QoS level any lower one and there will still be frames traversed on lower QoS values. If you have a very busy system that would be mapped on a QoSH zone it could potentially prevent all other traffic in QoSM and QoSL from being moved over an ISL. By FOS applying this ratio there is still a relatively fair queuing mechanism and shouldn't cause issues on host. Obviously when you run in these situations it would be wise to start re-designing your infrastructure and increase the capacity on the ISL's to accommodate for more traffic.
Hope this explains the situation.
Kind regards,
Thanks Erwin. Now it's more clear than it was before 🙂
Hi Erwin,
Did you have tried or know how to make work brocade web tools from mac os X ?
Hello Adrian,
I do not have access to a Mac so unfortunately I have to say No on your question. Apple nor Brocade has sponsored me one. 🙂
hello Erwin! Could I ask you to get back on this once more. You tack about RSCN as the trigger for path offline. If so why it is online till real I/O or SCSI TUR will occur? This is also stated in HDLM documentation – "Without path health checking, an error cannot be detected unless an I/O
operation is performed, because the system only checks the status of a path
when an I/O operation is performed. With path health checking, however, the
system can check the status of all online paths at regular intervals regardless
of whether I/Os operations are being performed."
Hello Yury,
When things go wrong in a fabric there is no guarantee that the RSCN will reach all ports in the fabric. The TUR or the real IO request is then a fallback method for MPIO to take the path offline. If the RSCN has reached the HBA or storage port you can safely assume that path will be taken offline immediately.
Hope this helps,
One situation when RSCN will not reach the port is that link itself from the HBA port to switch is broken. Could you tell me about another situations? Suppose in normal environment RSCN should work fine for fabric events that the particular port is registered for (SCR Value).
Hi Yury,
Indeed. There has been a defect in FOS which inadvertently modified the SCR value in the nameserver registration. This caused RSCN's not to be sent to the device in question. Obviously the same thing can happen when a device has a firmware/driver bug and not properly handles the SCR registration or does not act properly upon a reception of an RSCN. These are however corner cases.
Thanx, Erwin
In our environment (RHEL 5.10 with Native MPIO) we have the situation when host gets RSCN (when storage port is offline), but does not block path to that storage port immediately.
It waits for dev_loss_tmo seconds (or other scsi timeout value based on scsi error handler) instead.
Do you want to say that something definitely wrong in our configuration and if RSCN has been got MPIO has to block path to that storage port immediately (without waiting for any scsi eh timeouts)?
Hello Roman,
No, your environment is behaving as expected. There are a number of reasons why a path is not market as failed immediately because an RSCN may be sent due to a number of reasons, a failed link is only one of them. There are two timers who control this. One you already mentioned and the other is fast_io_fail_tmo. Please see the documentation on when and how to use these.
Hope this helps.
Kind regards,
Thanks Erwin for reply.
I understand that RSCN may be sent due to a number of reasons.
But let's take a look storage port offline case – shouldn't MPIO block failed paths immediately right after it gets RSCN about storage port is offline?
🙂 There is no RSCN for a "Port-Offline". The RSCN can contain a few options one of which is "Changed Name Server object". This is the RSCN if any of the ports where the HBA is zoned to does change its state in any way. Doesn't necessarily have to be "Offline". It is then up to the recipient of the RSCN to determine what has changed. This might take a while. There is also an RSCN which is "REMOVED OBJECT" but I'm not 100% if that one is used currently. It depends on the RSCN event qualifier being used by FOS. That in turn depends on the SCN registration the recipient has sent to the fabric controller upon NS registration.
Thanks again for clarification 🙂
Just trying to identify what level (fabric, HBA driver or MPIO) is responsible for latency problem we have in case of storage port failure.
Hello Siddarth,
Thank you for your comment.
Thank Gavin for the response. As I mentioned above I'm not 100% sure how the calculation is done. I think its a bit too over-simplified to say you get 16000 IOPS out of a subsystem and divide that number by the sum of spindles in the system. I also mentioned in my post that by utilizing every surface of a spindle does not improve performance perse. It will most likely decrease if not handled correctly due to increased delay caused by the head-switch time which incurs an additional settling time.
If you can elaborate a bit more that would be very appreciated. As far as I can determine now per your comments, the proclaimed number of 400 IOPS per spindle is only substantiated in an overall 40 spindle subsystem.
"Obviously there are some additional smarts which is handled by the drive
firmware and buffer-space which can adjust and re-order the read- and
write commands to optimize some of this."
Some or all of this? TCQ and analogous NCQ changes all of this. Tracks will switch and get that nearby read, never mind it came on to the queue much later.. as you know. Maybe a guess of the firmware that Richie and crew came up with is to move the hotter blocks to faster tracks. I seriously doubt they improved on TCQ/NCQ algorithms. But who knows.
Seems Gavin's response below gives some clues on the usage. It doesn't seem to be NCQ/TCQ but I'm not 100% sure how the calculation is done.
Hello Erwin! Thanx a lot for this post. Just one question regarding CS_CTL values with Default mode. As I see ranges 1-8, 9-16 and 17-24 are not correspond to the number of VCs. How switch will interpret this? As kind of priorities? Thus will it allow the bigger percentage of High Priority VCs usage for frames with value 24 than for those with value 17?
Hello Erwin! Thanx a lot for this post. Just one question regarding CS_CTL values with Default mode. As I see ranges 1-8, 9-16 and 17-24 are not correspond to the number of VCs. How switch will interpret this?
Each of these values are mapped to the LOW, MED and HIGH VC groups in a round-robin fashion as far as I've been able to determine. Be aware that these values are defined in the RFC's I mentioned. I wouldn't be surprised if the QoS functionality from the CS_CTL field are extended to FCIP configurations where these DSCP values can be used to propagate these QoS priorities in the IP WAN networks.
I extended my question with assumption of some kind of priority. This will win with CS_CTL not only from the fact that it works with the particular LUN, but have a much higher level of granularity in the distribution of traffic.
As opposed to the IP layer capabilities where there are more options to choose from obviously. On the Brocade side we're restricted to 3 QoS levels hence why they grouped these values into 3 sections. There is no further prioritization within these groups.
Hello Erwin,
thanks for the guide.
One question regarding the system.cpuload setting. Is changing it to 10 still true for FOS v7.x or other fixed versions referenced in TSB 068 and 111?
What I got from Brocade is that in any version containing the fix modifying the cpuload setting is not needed. Do you have more information about that?
Best regards
Hi Michael,
Indeed there has been improved logic in later code releases however I've still seen issues hitting the i2c bus (the same bus that is used when polling the sfp's). By setting these values you can still prevent a process from hogging that bus. No guarantees if Brocade will hold on to this setting and values. The problem is most apparent when two or more management applications are polling the switches for info. I've seen one environment where 5 management applications plus two scripts where constantly polling the switches. This customer had run into an issue and wanted to be notified on multiple levels via various methods. Obviously the way they configured and used it was well beyond what a switch CPU could handle and they ran into even more issues. By killing off all but one management tool and reducing the polling frequency they went back to a more normal state. That management application was then configured via snmp and syslog forwarding to notify these other apps.
Hope this helps.
maybe little off topic but anyway could you provide information of CS_CTL support on HDS side. Is it supported? If so then for which sequence in exchange – inbound, outbound?
Hello Yury,
CS_CTL is not used in HDS equipment (or any other arrays). I'll try and provide a short article on it as it seems it does have cause some confusion of what this field in the FC frame header actually does.
thanks Erwin! would be great
Hello Yury,
I added a new chapter covering QoS and includes a section of CS_CTL. Hope it is clear and helps.
Thank you
This is great material. The only thing I might do differently is matching up the switchnames to the domain IDs. I usually advise to give the switch a name that helps correlate it to the domain. That way they (and us support people) don't have to remember
SW01 = 225 (e1)
SW02 = 193 (c1)
SW03 = 226 (e2)
I might go with something like SW-225, SW-193 and SW-226. Granted, you can see this mapping in 'fabricshow' (and it could complicate domain ID changes), but I just like knowing at a glance which domain I'm dealing with and it might help the customer make sure they're sending in data from the right switch.
Thanks for all of these posts.
Hi Erwin van Londen,
Well, i have an issue regarding Brocade FOS logging into the IE. Java 7 u 67 blocks and issues a Failed Certificate.
I tried all the stuff which has been talking around all the conversation held above by editing in Notepad++ and i cannot downgrade java as well due to environment standards, which should only run in JRE 7 u 67. But it doesn't help me out.
Is there any other solution with you that can help me out from this issue
": Brocade Network Advisor – Java memory size. – http://t.co/uuQF1Elfmj" < Great hint!
@landonnoll People still use Fibre Channel?
Hi Erwin,
Another solution could be TI zones… Maybe in your next post?
Hi Erwin, thanks for the excellent reading, even better than always.
Just as a side note about the D-Port functionality. Very good stuff, but still needs some … "tuning". 1) I've seen D-Port reporting "PASSED" while some very bad counters increasing at one of the ends of the wire. 2) Occasional disabling of the D-Port might lead to the reset of the entire blade.
Hi Erwin,
I had this annoying problem running and older FOS (6.4.2b) and a new java (1.7.0_55-b13) on my Windows 8.1 x64 laptop.
The solution is as described (changing the line jdk.certpath.disabledAlgorithms=MD2, RSA keySize < 1024 to jdk.certpath.disabledAlgorithms=MD2, RSA keySize < 256) in the java.security file, however when running a 64bit system this file exists in TWO locations:
c:\program files\java\jre7\lib\security
c:\program files (x86)\java\jre7\lib\security
Be sure to make the corrections in both files that worked for me 🙂
While I am not certain why you were seeing a rule to alert on anything more than 0% Memory Usage, the default in BNA is to alert above a threshold of 75% memory utilization. I believe you will agree this is a reasonable default threshold value.
Hi Erwin,
I discovered as well some unexpected behavior with the conversion tool from FW to MAPS. I would advice cloning one of the default policy and then do the modifications from this customized policy:
mapspolicy –clone dflt_moderate_policy -name my_policy
You can see below the default log threshold for CPU (80%) and Memory(75%) with the default moderate policy:
mapspolicy –show my_policy
defCHASSISMEMORY_USAGE_75 RASLOG,SNMP,EMAIL CHASSIS(MEMORY_USAGE/NONE>=75)
defCHASSISCPU_80 RASLOG,SNMP,EMAIL CHASSIS(CPU/NONE>=80)
If you decide to build your customized policy from one of the default ones, please take care at the fencing thresholds. Fencing is activated for almost all error counters with the default policies. This is a Brocade best practice that I would advice as well. However a lot of customers are afraid of such behavior. This is at least worth to be aware of it.
Dropping the value from ' <1024 ' to ' < 256 ', by itself, did not work for us here on a 48K at FOS 6.4.1b with Java 1.7.0_51 on the host. But doing that AND changing the java control panel security tab from HIGH to Medium did the trick nicely!
Dear Erwin, could you please clarify what happens when frames tagged by a Cisco Equipment on one side are received by a Brocade device on the other side. To be more precise, I am referring about the interconnection between UCS (FEX) and a Brocade device. By default, the FEX will tag traffic in VSAN 1. It means that somehow this tag should be included in the extension header. In this case, what happens when the brocade switch receives it. He simply does not take it into account and accept the traffic?
I was under the impression that VSAN were working the same way than VLAN. With VLAN,when a port is in access mode or untagged more, the frame is sent without 802.1q header at all. However, with VSAN, I did some tests that made me think that it behaves differently. When we connect a FEX in VSAN 1 to a MDS switch, there is communication only if the VSAN 1 is configured on the other side as well. It means that the VSAN ID is included in the header. However when we connect it to a Brocade switch that is not able to "speak" VSAN, there is still communication. Any idea how?
Thanks in advance.
I use firefox and some nights at 12:00 exactly or 12:30 exactly, firefox stops working and does not load a page. The internet connection is perfectly fine though because it says excellent. I am connected to a home router and the signal strength is always excellent. I do not know why this happens at exactly 12:00 or 12:30 on my desktop time but it's a pain. I've tried "ping"ing and everything looks fine..
Hello Erwin, I have got a brocade 6510 connect to a PC with SUSE linux.When I try connect to switch after that I write login and password the webtool dosn't allow neither use the tool bar nor progress to other window.
Linux suse has a java.security file but I don't find "jdk.certpath.disabledAlgorithms=MD2, DSA, RSA keySize < 1024" sentence.
and then I can't apply it.
Hi Erwin, thanks, that's a good reading!
Clock sync is really essential when troubleshooting a complex issue.
I once had to help people finding the problem, where each component was supported by a different support partner: AIX host, HP disk array, EMC multipathing and CISCO SAN – what a nightmare environment to build and maintain! And BTW everything was hooked to NTP, except disk array SVP, but its time difference (like 57 minutes or something like this) was already carefully recorded. So the email thread when I've got it looked like this:
IBM: ERRPT shows FC errors and resets at 7PM. What do you see at this time?
EMC: yes we see this in the PowerPath logs.
HP: well we see some SCSI reserve conflicts, but we cant tell why they happen…
CISCO: we don't see anything at all!!!
Others: ??????
It was the first in my life troubleshooting exercise with CISCO and I was surprised to discover that they log everything using GMT, so SAN guys were looking at the much later point of time in fact… Also, at the end of the end, CISCO wasn't found guilty 😉
I am having the exact same problem that brjaiswal has. I've also tried the same things with the same results. In other words, I am not able to run the switch explorer after the install of java 7 update 51.
so let's pretend we are interested in excavators 🙂
Good tip to share. I've seen this burn customers before too who were trying to merge fabrics. Merges failing due to zoning "conflict" even though one side was already cleared with cfgclear. If that defzone is set to NoAccess, that hidden dummy cfg can prevent the fabrics from merging.
Can this be changed non-disruptively if you have zoning on and have an active zone configuration?
Hi Erwin,
Thank you for putting this together. I believe it is very helpful information.
THANKSSSSS – I don't know how to thank you, I couldn't find a solution almost anywhere on the net, this worked for me straight away.
Just a hint that Windows users need to modify that "java.security" using Notepad++ or similar app. Normal Notepad will NOT work.
I have tried all the tricks listed here and still i cannot get both versions of java to operate on my laptop.
I need Java 6 & 7, i have updated/downgrated to 7.25 without any luck, the newer swtiches work with Java 7 ( Brocade 7800)
but the older one with FOS 6.* 5300,300 dont.
Hi Erwin,
I have changed RSA keySize < 1024 to 256 but I am still not able to access brocade 5300 web tools. I have v7.2.0b on switch and java 7 update 51 in my laptop. I also tried to comment out "jdk.certpath.disabledAlgorithms=MD2, DSA, RSA keySize < 1024" but not working. Please suggest any solution.
It helped me.
The cache-clearing thing really works around the bug.
Thank you…
at the same time, our storage tools require no more than 1.7.40… what a mess!
What can say about AoE (ATA over Ethernet) developed by Coraid?
It looks like pure ethernet without any FC layers, instead of FCoE (which is not ethernet in reality 🙂 ).
You raise some excellent points, as usual. Obviously, I can't speak to all vendor implementations on the planet 🙂 because ultimately what it boils down to is whether or not vendors err on the side of caution. (Full disclosure to your readers I am a PM for storage for Cisco's Nexus switch product line).
The FCoE frame size is fixed as per the standard, so with respect to running a mix of larger and smaller frames on the same lossless Class of Service (COS), this would require the additional placement of lossless frames (RoCE?, iSCSI frames?) into that same COS. I know, though, that when FCoE is configured for lossless COS no other traffic type is allowed on that same COS.
You raise an excellent point about other types of lossless frame sizes, however. I've always been highly suspicious about the efficacy of putting iSCSI into a lossless environment, but your point about different sizes is particularly apt if people start playing around with 9k sizes.
Your point about needing to resend a PAUSE frame to prevent resumption is something that has always puzzled me – it's actually built into the standard spec. I always thought it would be more appropriate to have an UNPAUSE frame sent back instead, but I'm sure that was discussed before I became involved in the standard body. Next time I go I'll ask, and maybe I'll have a logical reason to share. (Whether we agree it's the correct thing to do or not is a different story 🙂
Realistically, however, the key factor here is one of distance, as often the HWM is hard-coded at the maximum allowable threshhold for potential disruption; it's not a dynamic setting. In other words, we take the worst-case scenario and use that conservative limit as the starting point.
Having said that, of course, the behavior you describe – while certainly possible – doesn't appear to be observed in practice. Do you have test results (or production environments) that have experienced this? I have to confess that over the past 6 years of doing FCoE (from pre-beta days) I've never actually seen this occur, personally. I would be very interested to see or hear anything about this in real environments.
Hey Erwin,
Great analogy. One quick addendum regarding your water basin analogy for Ethernet. Your water metaphor is appropriate, because there is a "high water mark" in the ingress buffers on ASICs capable for lossless traffic. This is calculated based upon a maximum distance by which the PAUSE frame can be sent back to the source in order for all the water in the pipe to be satisfactorily stored in memory without discarding frames.
This is why distance becomes somewhat of an issue, as you know. At the risk of piggy-backing on your well-written article, your readers may find some additional information useful as well: http://blogs.cisco.com/datacenter/storage-distance-by-protocol-part-iv-fibre-channel-over-ethernet-fcoe/
Again, really well done.
I highly recommend Port Fencing for ISL's. I once (2009) encountered a strange phenomenon although I guess can be explained by several contributing factors. We had a IBM AIX host with some 40TB storage assigned to it and it was capable of some heavy IO. So we had 4 end to end connections, 4 HBA's zoned to 4 USP-V ports. One of the paths went down and the application timed out.
The reason the path went down was because it encountered a flapping ISL enroute and the resultant IO errors caused the path down. But because this was not a direct offline sequence the RCA determined that the IBM SD driver sent messages to the HDLM driver which were not understood and as a result the path was taken offline a bit late, resulting in a app crash.
If I'm not mistaken such flapping ports also cause a drain in buffer credits. It would be nice if you could write a piece of Flapping ISL's, the effects and the cure.
so how is the EVA different, customer claims that he used to
1. Zone the host to the port so that the WWPNs are visible on the port.
2. Mask the LUNs using the visible WWPN
At least the claim is that the LUNs were visible right after, no reboots or flapping paths.
As one of my colleagues pointed out, is it because EVA has the concept of a LUN0 visible to all hosts. Some sort of an access device which allows every PLOGI to be registered??
I tend to agree with pretty much everything that has
been composed in "Storage & Beyond | by Erwin van Londen".
Thanks a lot for all the actual details.Regards,Damon
I must admit that have trouble understanding your question. Can you be more specific?
Hi Ron,
Use the recommended values of 220 to 250 on the edges and 500 in the core. If you can Condor2 based switches in the core (DCX/DCX4S) try to separate E-ports from F-ports onto different ASICs. If your edges contain TARGET ports (arrays, tape devices etc.) do not set a low value here.
The problem is that it's a very grey area with touches on performance and quality of links, drivers, firmware and hardware including the age of the equipment. You can have a perfectly well functioning HBA from 2002 which does a great job for the standards that were valid at that time but these days it could be a potential bottleneck.
Yes, the (0) is the default value which will incur the values a I depicted above.
Hope this helps.
i have some question which i have face that is !
Where all my available storage in the enterprise?
Why is it available?
Hi Erwin,
I just checked, EHT on both core and edge are showing as 0(default?? on this version). we are at 6.4.1 with an upgrade scheduled in two weeks. what would you recommend it changed it to?
Thanks for your help.
Be aware this box has been developed with some serious future proofing installed (as they did with the MDS95xx series). I wouldn't be surprised that a fully populated box with 40GE and 32G FC ports (albeit on a somewhat lower portcount) would also be non-blocking and hence you would need the switching bandwidth they've build into it now.
Something I don't get in their math:
First, 384 ports @ 16G gives 6.144 Tbps duplex or (okay, okay) 12.288 Tbps "marketing" if you count each direction separately.
Second, 256 Gbps per "fabric" with max 6 modules gives 1.536 Tbps total, which in my book of references means 4:1 chassis oversubscription when compared to the above.
None of it comes close to the "24-Tbps FC Switching Bandwidth" advertised in the product sheet.
What am I missing?
It's even more complex with different FOS versions, chipsets, if the "configure" command has been run or not. Quite some potential bombshells here.
It seems Brocade is prepping a whitepaper on it so we'll see. Meanwhile the entire frame-timeout scenario becomes even more difficult to troubleshoot.
Wow, I was not aware of that. I also found no statement in the release notes. But our switches clearly show the default of 220ms for the EHT. Thanks for the hint, Erwin!
Hello Raj,
The feature you mention are IP based technologies. Only VLAN's could be an option on the Ethernet layer.
Then again, do you want to burn such a massive number of VLAN's for initiator/target segregation? Network admins already think the current limitation is hampering their network growth. All in all it's a bad idea.
But that's just my opinion. 🙂
Hi Erwin,
You can use VLANS similar to that of FC Zones.
Use MACADDRESS of the Initiator for LUN masking on the target..
You can implement end-to-end encryption something similar to SSL/TLS. Exchange a session key on discovering the LUN by the initiator and use it for encryption/decryption for the life of that session..
What do you think?
Yes, Brocade publishes a separate guide for FabricWatch. Be aware this is a licensed feature. You can download this from the Brocade website if you registered for a MyBrocade account.
Is there a guide where or how to configure this Fabric Watch feature? I have dos B40 Switch and I would like to take advantage of this feature
thanks a lot
Thank you so much for sharing this storage 2013 and beyond. It is very important to us because we need to protect our things or files. And these kinds of tips are very informative. Keep on sharing!
Learn more about Storage Get Information Here
SNW is also quite well established in Europe – 2013 will be in Frankfurt – 29-30 October – mark the date if you're in storage in Europe. http://www.snweurope.com
Thanks Erwin. I read your post with interest. I would also add that without SNIA there would be no independent global storage education. This is particularly important in the regions that I deal with such as South Asia, China and India where there is always a huge demand for high quality education in storage technology.
one can use vlan tagging to separate the traffic at level2. The switches normally won't pass packets tagged for vlan A to vlan B
Hi Seb,
Indeed TUR is one option, a read-sector-0 another. Problem is that if only 0.5% of frames are affected you still have 99.5% chance of TUR/RS-0 to succeed. Secondly if one of these TUR/RS-0 are going to lala-land there will always be a re-try of that same command which also has a 99.5% of succeeding. As you said this is not a very reliable way to determine and end-to-end path status.
I'm working with some folks in T11 to get this addressed but it might take a while since some new frames need to be introduced. These protocol changes are always cumbersome and even if they are rectified we still need the vendors to implement it. Still might take a few years.
Good article! It's a good wake-up call for admins relying on an assumed "intelligence" in the multipather to save them not only from hard connectivity problems but also from "soft" problems like much longer latency or frame drops. Indeed there is an additional mechanism (often called "Heartbeat" or "Healthcheck") in many multipathers. It sends TURs (SCSI Test Unit Ready) repeatedly and could for example set the path to dead if the TUR fails. But in the field, if you have a performance problem, often only a small number of frames are really dropped – although this "small number" would be a pain, too, as it will eventually lead to error recovery with all its nasty timeouts. If the TUR was not affected, the multipather just doesn't recognize it. And even when it hits a TUR and the path is seen as "not reliable anymore" – just some successfully completed TURs later it will be used again although the performance problem is still there. So unfortunately TURs are not the solution of the problem.
In the mainframe world there are some new mechanisms to cope with the problem as Brocade's Dr. Guendert explains here: http://community.brocade.com/community/brocadeblogs/mainframe/blog/2012/09/14/the-ibm-zenterprise-ec12-brocade-dcx-8510-sportscar-tire-extraploation
Let's see what the open world guys come up with.
Hi Seb,
Thanks for your comments.
Indeed you need to take care on which classes and entities you enable this portfencing and as such instructions from support people such as yourself and I should be followed. The thing I wanted to emphasize is that administrators and operational management should strive to use this feature as a measure to prevent problems and not wait for such problems to happen after which significant downtime and possible recovery efforts have to be undertaken to get the infrastructure back on track again.
When it comes to portfencing I think it's worth to mention that you can configure it to trigger for C3 discards due to timeout on the TX side. If you do that only for F-Ports, you should be able to catch hard slow drain device ports. Don't do this for E-ports, because due to backpressure there could be a trail of dropped frames through the whole fabric and you would fence a lot of "affected-but-not-the-culprit"-ports. Plus: If you enable portfencing also against other "areas" like PE (protocol errors), set the correct fillword first. If you use fillword 0 (IDLE/IDLE) for an 8G connection, the link might come up (because the link initialization uses IDLE and is okay then), but then the switch interprets the incoming ARBff fillwords as protocol errors (because it expects IDLEs) and fences the port. You see that as "er_bad_os" in portstatsshow.
Cheers seb
No, I won't be there. My HDS T11 rep needs to schedule this but I first need to get some things out of the way from a technical perspective.
It should go into FC-FS but I don't know which release yet.
Are you attending one of the f2f meetings about this topic? In which "workstream" is this discussed?
I have something in the pipeline to address this. Will most likely be addressed at T11 in October or November.
Hi Seb,
Yes, there aren't many blogs around which cover all this.
There is more to come.
Interesting to read your blog as well.
Keep in touch
Interesting article. Good to see more blogs about SAN troubleshooting!
To the LR: In the situations where I traced it, the link reset usually happened after ED_TOV (2s) of running without a single buffer rather than the 500ms ASIC hold time. But maybe some devices start earlier with the recovery – both values are very long periods in the world of FC.
Cheers seb
Good article! The ugly thing with most of the multipathers is that they are not really built for problems like these. They can cope with links going down->be repaired->go up in perfect shape again. But if frames get dropped in the SAN due to slow drain devices, the multipather will notice a problem, but with a later TUR (Test Unit Ready) the path looks good for it again and will be further used. It would be better if the MPIO would track response times over the different paths and would move load towards better performing paths (avoiding the ones impacted by the slow drain device).
Cheers seb
Hi Roger,
Thanks for your comments. You are right that the majority of Ficon and FC Extension products came to Brocade through various acquisitions. As for calling this a technological advantage I'm not sure. Both Brocade and Cisco have obtained significant knowledge in both spaces so when looking at the sum of RFE/RFP tick-boxes I wouldn't know which ones would show an advantage. Acquisitions done over 6 years ago do not provide an advantage in my view. Most technology has been refreshed, standards have changed and engineers working at those companies might have been jumping borders to other avenues. If a company has no products, knowledge or R&D facilities in certain spaces (like Brocade back then had with Ficon and extension) then the fasted and easiest way is an acquisition. Buying McData was a good one, keeping the product-set on the sales inventory list wouldn't have been my option)
I do agree that Brocade most likely has an advantage with the variety of products which fit both enterprise and SMB markets but Cisco also has products which fit both spaces. I guess in the end it comes down to customer requirements and Brocade likely has more products which can fit specific scenarios.
As I mentioned Brocade has been around since the dawn of the FC time and hence they have a longer track-record in the FC space. This is most likely the reason why customers still look to Brocade in the FC arena and to Cisco in the networking space.
Again, thanks for your response.
Hi Erwin,
(i'm an IBM employee)
Good post thank you. As ex- Inrange/CNT/McData employee i have a slightly other view to this. Looking from the ESCON/FICON view into the Datacenter i see some clear advantages for Brocade (because of the heritage coming from Inrange/CNT think on the Inrange FC9000 director, CNT UMD director etc). Here Cisco is lacking behind. In the FC arena i see the most of my clients in Small- and Medium market jump on Brocade products because of the variety of there products.
Hi David, thanks for your comment.
As for AoE its a very lean and mean protocol. Coraid is one of the vendors using it. The problem is that you need a flat layer 2 Ethernet network since nothing is route-able and on large scale storage architectures it might be limited. Personally I never used it.
You might want to check out the Tech Field day website (http://techfieldday.com/2012/coraid-presents-storage-field-day-1/) where Coraid presented. They also had a presentation on the protocol differences. Be aware that presentations and Q&A are always biased 🙂
As with iSCSI, AoE also can have its place in the market but currently it looks like a very niche one to me.
Just curious… I stumbled across your article due to hearing about ATA-over-Ethernet (AoE) that's been built into the Linux kernel for a few years now.
What would your take be on AoE? I assume same security concerns– I'm not really sure I see an advantage of AoE over iSCSI.
Your take?
Hi Erwin,
The use of FCID forces host CNA to have to directly connect to FCF to acquire FCID allocated by the FCF (FCF maintains centralized SNS database), at least for current FC-BB-5 standard. Maybe FC-BB-6 has way to assign FCID w/o directly connecting CNA to FCF but that could add great deal of complexity, for example, FC-BB-6 introduces distributed FDF.
I maybe wrong but just my 2 cents.
To see the same output on a Cisco MDS/Nexus, use:
# show interface fc1/1 transceiver detail
I don't think the addressing itself is the problem but more the complexity it introduces.
Thanks for you comments.
Kind regards,
I think you are right on. The problem with FCoE is that it is building a networking protocol (FC) over another networking protocol (Ethernet). FC uses 24-bit FCID for addressing but Ethernet uses 48-bit MAC for addressing, that is unnecessary and can only complicate design and cause more inter-op issues. It is absolutely possible to design a light-weight transport protocol to directly transport SCSI command/data over Ethernet.
thanks for the article, now I understand why VMware had problems with it. I agree with Chris that it's the storage array's fault, but I also think that there is a simple solution:
The array should maintain a bit-flag "not_written_since_last_reassignment" per physical block, stating that the physical block has been unmapped before and not written to since then. As long as the flag is still nonzero (i.e. true), the array should return all-zero-blocks on any read-request. Using such flags there would be no need to physically zero unmapped blocks.
Thanks for a good write up on the SCSI Unmap issues.
I'm sorry to hear about your situation. When I was in the same boat 2 years ago I was in contact with a outplacement organisation who helped me getting things back on track w.r.t. resumes etc. Did your former employer provided you this as well?
Unfortunately I hear more and more stories of peoples lives getting wrecked because of this misused visa.
I hope you'll find something soon.
With you on this. In my case, the 457 was the ONLY way I could get into the country. Employers were not willing to wait for the RSMS, and I had just barely missed the age cutoff for the other migration options. (In a cruel twist of fate, because I came to Oz when I did, I also miss the revised points test announced last year, due to not having the right mix of ranges of years of employment in Australia and my home country.)
So now I am redundant in a backwater town in which there are no other companies that require someone with my skills. Here I am looking for work over the Christmas season, where the odds aren't as good as they would be even a month later. At this point, I will work anywhere willing to take up my sponsorship for a 457, but my ultimate goal is permanent residency.
As a second comment Hu Yoshida and Michael Heffernan have also posted some comments on their blogs on http://www.hds.com.
Check them out.
Hi Chris,
Indeed 42MB is not a bad number and due to this fact we don't have this much CPU overhead plus we can use a very good sequential algorithm to get this sorted whereas if you have much smaller segment sizes these could be scattered all over a diskpool which causes much more randomness and hence a significant performance impact. (Yes this 42MB didn't drop out of the sky for nothing since it has many more effectiveness all over the place w.r.t external storage, HDP, HDT etc….. We always told you we have smart folk on board. :-))
W.r.t. to the T10 UNMAP being secure or insecure or whether it should have this option that's up for debate. Be aware that pools can be shared between hypervisors and non-hypervisors as well as RAW apps like Oracle. Also how does a hypervisor determine when a certain LBA needs to be securely erased or not. If the OS does some memory swapping the hypervisor is really unable to determine if this is due to the memory swap process or if an application is deleting files. As I mentioned in one of my first blog posts the lack of data intelligence mapped onto the infrastructure is massive to such an extend that every piece of storage equipment out there has absolutely no clue of what's going on on a application levels (some exceptions do apply) . This is where the real integration should be. This means that the apps should determine whether an UNMAP operation should be wiped or not and not have the hypervisors interfere with these kind of operations.
Anyway, lets see what the future brings.
Thanks again for your reply.
This draws up a number of questions;
First, 42MB might not be such a bad number after all, as the number of block release an remappings would be significantly less than with, say, 16KB. Next, you would expect VMware would *not* be releasing blocks back to the free pool on a regular basis unless some kind of threshold was met. For example, as VMFS marks a block as free, if it kept (say) 10% blocks in a local queue and reused them, then there would be less need to push off the release to the array.
I think also it means that the implementation of UNMAP on the underlying array is flawed. T10 & VMware and the storage vendors should have implemented both a secure and unsecured UNMAP, one which wipes data, one which doesn't, rather than assuming everything is so critical that it needs to be wiped at release. Where arrays are only connected to VMware, then an unsecure UNMAP might be acceptable as new blocks would zeroed out on usage.
Perhaps we'll see some of these functions coming along as they mature.
SSD will indeed solve the performance factor but now they have to work on capacity as well as reliability.
I modified the font size. Looks like Goofle adjusts it to the smallest possible to be able to include the tables as they should look.
Very interesting and timely…. leads to the question of SSDs..
Only complaint is that the font is coming out really small. Not sure if its the browser or WordPress? Or maybe your eyes are better than mine…. #;-)
I agree, the only thing you need to with FC is create a zone. Voila plug and play. 🙂
Of course you can add a gazzilion functions and features but the bare essence is domain-id and zones and you're good to go.
FCoE is designed to run a flat L2 Ethernet layer which by definition is not rout-able. If you want to introduct FC routing on FCoE yiou have to do very weird tricks and I wonder how long a troubleshooting case would take in such an environment. 🙂 Veeeeeerrrryyyy lllooooonnnnngggg…..
Thanks for your support.
Nice to see someone giving a contrary view to all the FcoE bleeting, must do the same….
What I find most annoying are complaints that Fibre Channel is hard to setup. Its simply not true.
Amusingly the most complex thing I face in FC, is setting up FC routing… and guess what? FcOE cannot do that. #;-)
Have a listen to this podcast:
Really interesting stuff, but worrying about the implications of lack of interop….
Hi Edwin
Good article and solid points. There was an article published in the Register which is a nice complement to what you've written entitled "Brocade says FCOE may not happen". I think the title says it all (-;
Hello Erwin,
I agree. (short form).
Inventions not always brings a tool to establish "real" TCO.
What I call "panacea".
Because someone does it, someone else has to do it, a bit like some people wearing Christian Dior: the need was to get dressed, not to spend millions.
CPU Storage Sharing suffered the same. Sharing protocol because storage was expensive was an attempt to avoid cost… but when adding expert cost to make it run and maintain it (which no one does) plus down time due to human errors because too many protocols are shared… then sharing does not look that good anymore.
It is like pooling things.
Sometimes being "dynamic" is ok to do. Sometimes benefits are greater with static, because of management cost.
Great article Erwin.

subset: commoncrawl


text:
This is an archived article that was published on sltrib.com in 2013, and information in the article may be outdated. It is provided only for personal research purposes and may not be reprinted.
A bill that would limit the power of Utah's Medicaid Inspector General (OIG) to police health care providers has cleared its first legislative hurdle, after sparking contentious testimony over how to handle billing errors that drain taxpayer dollars.
HB315 largely clarifies the Legislature's intent in creating the OIG two years ago to crack down on the waste and abuse of Medicaid funds, after a string of legislative probes that found widespread "upcoding," or overcharging, by doctors and hospitals.
The bill directs the OIG to pursue its watchdog role without alienating the declining number of doctors willing to treat Medicaid patients to differentiate "between honest mistakes and intentional errors, or fraud ... for the purpose of entering into settlement negotiations."
But the hospital-backed measure would also absolve providers for billing errors stemming from conflicting Utah Department of Health policies. It would limit the look-back period for investigations to three years and require investigators to collect evidence "on a day, and at a time, that is minimally disruptive" to the health professional.
And it would amend the current requirement that health care workers and state employees report billing errors to the OIG. Instead, whistle-blowers would first report errors to their employer or employing hospital "unless there is a credible allegation of fraud." If, within 30 days, the mistakes aren't fixed, the worker can report them to the OIG.
Current Inspector General Lee Wyckoff opposed the bill at a Monday committee hearing, saying it "muddies" current law and would hamper his ability to recoup misspent taxpayer funds. "I haven't heard a single example here today of people testifying that our office is overstepping its bounds," he said. "We are not a smash-and-grab operation."
Utah Hospital Association lobbyist Dave Gessel disagreed, insisting hospitals have "had many problems with the OIG." Last spring, he said, small rural health centers received letters from Wyckoff's office demanding repayment of close to $1 million in overcharges.
"As the hospitals, on their time and money, dug into it, they found that over half the claims had nothing to do with problems identified," and as a result, the OIG pulled back the audit, he said.
Wyckoff challenged Gessel's claims as unfair and inaccurate.
The demand letters were prepared and mailed by federal investigators "without my review," he said. "I didn't know the letters had gone out until I got calls from [the hospital association], reviewed them and within a matter of days retracted that audit with me calling each and every CEO or CFO of those organizations and apologizing personally. For [Gessel] to bring that in front of this organization and exploit it as something our office screwed up on, I think, is a good example of how things are misrepresented."
This drew an admonishment against "attacking [peoples'] character" from the committee co-chairman.
Dave Davis, president of the Utah Retailer Merchants Association, meanwhile, rallied for Wyckoff, commending him for working with pharmacies to clarify misinterpretations of federal billing rules.
"There was an attempt to claw back money [erroneously billed]," he said, "but in working with the OIG, that was set right."
Still, Davis said he supports setting firmer policing "guidelines and fence posts" because "we don't know who will come after Lee Wyckoff."
Frank Pignanelli, a lobbyist for a company that supplies the OIG with auditing software, warned if the state frees providers from liability over policy misunderstandings, taxpayers will be left holding the bag.
"You're basically exempting the providers, because there will be overpayments," he said, "and the federal government's going to want their money back."
But the physician-stacked House Health and Human Services Committee unanimously endorsed the bill as a prophylactic against the past investigatory sins of others.
"Sometimes there's a Gestapo approach. The fraud units get a 10 percent cut on the money they receive, so there's emphasis on them to go set up these stings based on limited information," said surgeon and Fruit Heights Republican Rep. Stewart Barlow. He referred to a raid, long before the creation of Utah's OIG, where federal agents closed a Richfield doctor's office, seizing computers and records. "They weren't guilty of fraud. It was just a coding error."
Rep. Mike Kennedy, R-Alpine, a physician, said medical coding, or billing, "is a byzantine process, at best. ... I can guarantee every health professional you know, including those on this committee, have committed some form of abuse of this process, in the vast majority of cases unknowingly."
The battle over policing Medicaid
For a look at Utah's past efforts to attack Medicaid fraud, waste and abuse and the state's new Inspector General: http://bit.ly/Xe08ds


text:
A hearty "welcome home" is an appropriate greeting as you enter the gates of Disney's Old Key West Resort. This "home away from home" resort is designed for relaxation with its quiet, peaceful atmosphere. The 2 and 3 story bungalows are colored in soft, relaxing pastels of blue, pink and green, with tin roofs and gingerbread accents. This is reminiscent of the quaint villas of the turn of the century Key West. The lush landscaping with its flowering bushes, tropical plants and swaying palm trees are found amidst the waterways that wind throughout this large, spread out resort. In typical Disney style, the detail extends to shell and starfish imprints in the cement walkways and the "not so hidden" Mickey heads in the ornate white railings. Old Key West is a Disney Vacation Club resort, but vacant rooms are made available to the general public on a per night basis as with any other Disney resort.
The easy life is the theme of Old Key West's villas. Furnished with plush armchairs and bedding, the rooms invite you to kick back and relax. A beach motif is carried throughout the rooms with muted pastels and décor accents of seashells and seagulls.
The Old Key West Resort features large living areas, with beautiful views of the Lake Buena Vista golf course, neighboring woodlands, or the waterways. Four types of accommodations are available here: Deluxe Studios, One Bedroom Suites, Two Bedroom Suites and the Grand Villa. Deluxe Studio: The Deluxe Studio offers 2 queen-sized beds, nightstand, armoire, TV, table with 2 chairs and a decent sized closet, in a 376 square foot room. Full bathrooms have ceramic tiled floors and storage area in the vanity under the sink. The kitchenette area includes an under the counter refrigerator, microwave, coffeemaker and small sink. A start-up supply of coffee is provided in addition to coffee mugs, glasses, paper bowls and plates, and plastic cutlery. The picturesque view can be enjoyed from the private balcony or patio. Handicapped accessible rooms are available here.
Type of Room: Villa
Categories: Deluxe Studio, 1 Bedroom Villa, 2 Bedroom Villa, 3 Bedroom Grand Villa
Approximate # of Rooms: Deluxe Studio: 210, 1 Bedroom Villa: 210, 2 Bedroom Villa (including 2 bedroom lock-offs): 453, 3 Bedroom Grand Villa: 24
Maximum # people: Deluxe Studio: 4 people, 1 Bedroom Villa: 5 people, 2 Bedroom Villa: 9 people, 3 Bedroom Grand Villa: 12 people
# Handicap Accessible Rooms: Deluxe Studio: 20, 1 Bedroom Villa: 20, 2 Bedroom Villa: 51, 3 Bedroom Grand Villa: 3
Note: Handicap-specific room locations differ according to the disability being accommodated
Square Footage: Deluxe Studio: 376 sq. ft., 1 Bedroom Villa: 942 sq. ft., 2 Bedroom Villa: 1333 sq. ft., 3 Bedroom Grand Villa: 2202 sq. ft.
Basic Room Contents: All rooms come with an alarm clock with mp3 player, ceiling fan, coffee maker and coffee setup, table and chair, DVD player, hair dryer, safe, iron and ironing board, pack 'n play crib, telephone and television. Deluxe studios come with a kitchenette (mini fridge, microwave, wet bar and toaster). The bedroom villas come with a kitchen (full-size refrigerator, microwave, stove, oven, toaster, blender, dishwasher, plates, flatware, glasses, cookware and sink) as well as a washer and dryer.
Basic Room Layout:
Note: The only buildings with elevators are buildings 61-63.
Lounges and Pool Bars
Tip: Olivia's offers a To Go menu, complete with appetizers, soups, salads and savory entrees.
At Old Key West, the Sandcastle Pool is behind Olivia's. The focal points are the slide which is encased in a giant sand castle and sauna located inside the iconic lighthouse. A hot tub, kid's pool and playground can also be found in the pool area.
There are three other additional quiet pool areas sprawled throughout the resort. Pool towels are available at all locations.
A number of poolside activities are scheduled each afternoon. Ranging from arts and crafts to trivia, there is a spectrum of games and challenges for families to participate in while they enjoy the water area.
Flying Fish Gameroom: Open 7am-10pm
Electric Eel Gameroom: Open 7am-10pm
Jogging Trail: Canal and Golf Course Trail (1.5 mile)
Basketball, Volleyball and Tennis Courts: Open 8am-10pm. Complimentary volleyballs, tennis balls, shuffleboard equipment and tennis racquets may be checked out at Hank's Rent 'N Return.
Outdoor Campfire and Movie: Each night there is a complimentary showing of a Disney film. Located next to Family Tree, the feature film changes daily (consult recreation calendar given upon guest check-in for time, location and particular movie).
Guided Fishing Excursions: For a fee, an expert guide will host your group during a catch-and-release fishing trip in some of the lakes on Disney property. Available times are 7am, 10am, and 1:30pm. Reservations required (407) WDW-BASS or visit the Concierge desk.
Community Hall Activities: Open 9am-7pm. Stop in for video games, arts and crafts and free DVD rentals. For a fee, there are organized activities to participate in such as ceramics and crafts.
Barbeque Pavilion: Located near the pool next to building 19, the pool next to building 42 and the pool near building 55. Charcoal can be purchased at Conch Flats General Store, and utensils are available at Hank's Rent 'N Return.
Playground: Located near Sandcastle Pool, Turtle Pond Pool and Southpoint Pool
Fitness Center
Conch Flats General Store: Open 7am-11pm (Disney Vacation Club and resort souvenirs, luggage, sundries, snacks and limited grocery items)
Magic Kingdom: Bus (available 45 minutes before park opening until 2 hours after park closing)
Epcot: Bus (available 45 minutes before park opening until 2 hours after park closing)
Hollywood Studios: Bus (available 45 minutes before park opening until 2 hours after park closing)
Animal Kingdom: Bus (available 45 minutes before park opening until 2 hours after park closing)
Water Parks: Bus (available 30 minutes before park opening until 1 hour after park closing)
Disney Springs: Bus (available 45 minutes before park opening until 2 hours after park closing); Boat (available from 10am-11pm)
Note: The bus stop is located outside of the lobby and near the conference center with buses departing approximately every 20 minutes. The boat dock is located on the waterfront behind the lobby.
Guests who drive either their own vehicle or a rental car to Walt Disney World should know that a standard overnight self-parking fee of $25 per night will be applied to the guest hotel folio upon check-out. Disney Vacation Club Members using vacation points for their stay are not required to pay parking fees.


text:
Three local players join 2014 Mallards
Handel, Butler, Ricci coming to Warner Park
MADISON — Three former high school stars from the Madison area will play for the Mallards in the 2014 season.
Infielder Michael Handel, who played high school baseball at Sun Prairie and now is at the University of Minnesota, played for the Mallards in 2011 and 2013.
He'll be on a temporary contract to start the season.
Joining Handel will be infielder Tony Butler, the former Sun Prairie High School star who just completed his first season at the University of Virginia.
Butler has a full contract with the Mallards.
Former Madison LaFollette outfielder Nate Ricci will also return to town to play for the Mallards at Warner Park. Ricci has played the last two years at the University of Jacksonville.
Ricci has a temporary contract with the Mallards.


text:
You have no items in your shopping cart.
Nike Air Max 90s have been revered the world over by both athletes and sneakerheads alike since their inception 25 years ago. Complete with all the usual ground breaking features as standard, such as the iconic visible Air-Sole unit. JD's regal range of Nike Air Max 90s features vintage classics and the latest silhouettes.
Nike Air Max 90 released in 1990 and is considered to be the second flagship sneaker of the Air Max legacy. The most popular colorway is the 'Infrared', which was the original colorway in 1990. The Air Max 90 was designed by legendary Nike architect Tinker Hatfield.
Regular Price: £65.45
Special Price £54.95
Regular Price: £111.65
Special Price £59.17
Regular Price: £89.32
Special Price £60.88
Regular Price: £84.70
Special Price £61.82