text:
It is our unfortunate task today to report that Massimo Tamburini, 70-years-old, has died this weekend, succumbing to his battle with cancer. A maestro of design in the motorcycle industry, Tamburini gave motorcycling two of its greatest treasures: the Ducati 916 Superbike and the MV Agusta F4 – two of the most iconic sport bikes in modern time.
It almost doesn't do the man justice to list all of his accomplishments and creations, as surely some would fall through our words, but Tamburini is best known for his contributions to Cagiva, Ducati, MV Agusta, and Bimota — with the last two letters in "Bimota" standing for the first two letters in "Tamburini".


text:
I finally had my first session with the Fuse 18.
I weight 86 kg, rode it with a tt 135x40.5 and 4 line bar (hard bar settings) with 27m lines.
Wind was 15/16 knots,(people were using 12 and going upwind easily).
I had to completely under power the kite, and let the bar out most of the time.
For sure this is my high end for this kite. I can´t say i was overpowerd since I rode safely but not comfortably most of the time.
It was hard to hold my edge for higher jumps, so most jumps were done with kite sending close to zenith, hights and glides pretty normal for an 18.
There was no condition to try kiteloop jumps or downloop transitions in 15 knots (very disapointed)
Bar pressure was medium and the turning speed not fast.
I am not a good high end rider, I am much better when kite is at low to medium force since I like to work the kite for jumps and tricks.
I could have changed the bar to soft and moved the back line one point to see what happens, but the wind was close to 20s so I went with rebel 13 and jumped better than with 18.
In this first test the kite didn´t achive my expectations.
I hope this kite suits me when the wind is 8/ 11 knots , that´s when I need it.

subset: commoncrawl


text:
Here's the latest article from the Children's Books site at BellaOnline.com.
Afternoon on the Amazon - MTH6 - review
Afternoon on the Amazon Magic Tree House Book is one of many children adventure books by Mary Pope Osborne. Join Jack and Annie in the Amazon Rain Forest.
Please visit childrensbooks.bellaonline.com for even more great content about Children's Books.
To participate in free, fun online discussions, this site has a community forum all about Children's Books located here -
I hope to hear from you sometime soon, either in the forum or in response to this email message. I thrive on your feedback!
Have fun passing this message along to family and friends, because we all love free knowledge!
Launa Stout, Children's Books Editor
One of hundreds of sites at BellaOnline.com


text:
2015 Farm Share/CSA is back! Sign up before end of May to be part of this year's membership. For more info or to sign-up please see our Market and Farm Share page
Our Projects and Places
Here at the New Urban Farmers we have some great projects going on from aquaponics in geodesic domes to teaching the next generation of farmers. Please explore the wonderful projects and places we have.
Our Programs
The New Urban Farmers like to grow minds as much as we like to grow safe healthy food. We have many different programs where our urban kids get a chance to become farmers in their own world. We also offer great growing and learning oppottunities for adults - Check them out!
Get in the Soil
The New Urban Farmers have many ways for you to get involved. We love to meet the folks from our community and we would love for you to try and get involved any way you can.


text:
What to CookWeeknight Cooking
Salmon Pasta
- Jenny
The situation is as follows: I like salmon. I like pasta. I felt certain when I spotted Salmon Pasta that the two elements would find a marriage of great harmony, not unlike the peanut butter and chocolate nuptials that brought us so many decades of low-rent pleasure.
What I particularly like about this recipe, outside of its many-food-groups-in-a-bowl goodness, is that while it has three steps taking place in three separate pans, they are done more or less simultaneously, thus somehow lessening the time and work load of the entire process.
The first step takes all of three minutes: salt, pepper and lightly oil your (I hope extremely fresh wild) salmon and toss it on a lined baking sheet as requested by mariaraynal, and pop it in your pre-warmed oven. I would like to pause here briefly to tell you all about a very exciting thing that has occurred in my life –- I went to Surfas, the restaurant supply store in Culver City, on Sunday and purchased a huge flat of pre-cut parchment paper. The first piece was used for my salmon. Oh, joy!
Okay so while your salmon is cooking (you won't let it overcook right? Thank you.) go ahead and get the water boiling, then add the sturdy pasta that was requested of you. She likes penne. So do I.
I talked to my husband on the phone for the entirety of the next step, which is chopping up a cup of shallots. Don't skimp. In fact, if anything, I would like to see two cups of shallots for this recipe, which calls for a pound of pasta (even though I did not throw the entire pound into my final dish). The shallots are the love of this marriage –- the pasta and the salmon being perhaps its commitment and virtue, respectively –- and you really want to taste them in this dish. Ditto for the pepper flakes. Be generous of hand. Use the best anchovies you can get your hands on, as I have lectured before.
Mariaraynal asks you to reserve your entire pot of pasta water in the event you need to moisten the pasta shallot mess you will soon have in your large frying pan. I think this is excessive; I used a tablespoon. So just save a cup, ok?
The incipient vegetarian –- at present a pescatarian, she says -– came home from the gym and rested a bowl precariously on her knees. She never had to be told to keep eating. "I never really thought about salmon and pasta mixed," she said, a fan of both. Now we all know better.
By mariaraynal
Serves 4 to 6
- 1 pound short, sturdy pasta (I like farfalle, penne or fusilli)
- 1 1/2 pounds fillet of wild salmon
- Freshly cracked black pepper, sea salt and crushed red pepper, to taste
- 2 tablespoons olive oil, plus extra for salmon
- 1 cup shallots, sliced thinly
- 4 cloves garlic, roughly chopped
- 5 anchovy fillets, minced
- 1/2 cup dry white wine
- Pasta water, to taste
- Zest of one lemon
1. Put a pot of water over high heat on the stovetop for pasta. When the water come to boil, add a handful of salt and cook the pasta for about 8 minutes, or until al dente. Drain and save pasta water.
2. Meanwhile, preheat the oven to 400 degrees. Place the salmon fillet on a lined baking tray and season well with sea salt and freshly cracked black pepper. Drizzle lightly with olive oil. Bake for approximately 25 minutes, or until the salmon flakes easily with fork.
3. In a large skillet, heat the olive oil over medium heat and add the shallots, stirring occasionally, until translucent. Add the garlic and anchovies and continue to stir. Season with black pepper, crushed red pepper, and, if desired, salt.
4. When the shallots and garlic are soft and lightly caramelized, add the drained, cooked pasta to the skillet and toss with the shallot mixture until well combined. Deglaze with the wine and reduce for several minutes.
5. Remove the salmon from oven and flake into large pieces. Place in the skillet and stir gently. If needed, add some pasta water and check the seasoning. Zest the lemon over the pasta and serve.
By day, Jennifer Steinhauer, aka Jenny, is the Los Angeles Bureau Chief for The New York Times. By night, she is an obsessive cook.


text:
Automatic CNC Programming Made Easy
There are no scheduled upcoming events.
ATS is the leading manufacturing CAD/CAM/CAE and Shop Floor Monitoring solutions provider for manufacturing companies in the Midwest and Southeast United States. We provide consulting services, software, and support to automate mechanical industrial manufacturing companies.3171C Beaver Vu Drive, Beavercreek, OH 45434 | 937-427-3015 | www.ATScnc.com.


text:
i would love to work there!
Elemental LED talks DIY Answered
Elemental LED hosts office DIY Challenge, names Instructables as their source for DIY inspiration and help
Elemental LED recently posted an article about their office DIY Challenge to see who can create the best LED projects. You might remember Elemental LED as the sponsors to our summer LED Contest.
The video talks about Elemental's place in DIY culture and why sharing creations on Instructables is important. In the video you can see some of the projects they created during their DIY Challenge party, they also mention how Instructables is their place to come and get inspiration and guidance for their creations. The video includes footage filmed at the Instructables lab, and appearances by our very own Bilal and Carley!
via Elemental LED
Hey, why don't you do an in-house contest at HQ, and get it judged by member vote?
We actually are! We're going to run an Autodesk Halloween contest and threw the idea around of having members vote :D
Awesome - make sure *everybody* enters, even Dworley!


text:
Miscellaneous Sales
Groupon is currently offering a Tea Forte Sampler Collection Deal.
$28.99 for Tea Forte Tea Chest Sampler Collection ($45 List Price)
just thought I'd throw it out there since I came across it…
$10 off a Teavana purchase of $50 or more online with code: SPRING
I think you have to print out the coupon to use it in store. Not sure how to share that with y'all. :/
Thru 4/28:
American Tea Room 15% off and free sample when you use the code "Kabuki" at checkout. Kabuki is their new green tea with cherry blossoms…
Mmmm, their ChocoLate …
Teavana has free shipping on all orders until tomorrow (May1/13)
Its too late :(
I couldn't resist. Free shipping, that's unheard of for this great country of ours.
Oooh. You almost got me when I realised you meant they had free shipping to CANADA! :O But I haven't had Teavana, and don't want to risk 2oz on unknowns… Especially when I live near a Teavana… :P (I'm just afraid to shop in store)
The good thing about having a Teavana nearby is being able to go in and get a cup of tea to go so you can try it out. I've only ever done it with herbals (and of course they are more forgiving to differences in steep time/temp) but it was a great way to try out a tea there before committing to a full 2oz. And shockingly enough I wasn't upsold in either of the two Teavanas I went to recently.
Lots of reasons. Mostly the horribly pushy obnoxious sales people that hound you your entire visit asking you tons of questions and telling you about every product your gaze pauses on. I have had one or two that were the exception to that rule, but I think they train their employees in the bombardment technique which sets me on edge.
The Teavana by me has teenage employees who ignore you until you ask them a question (unless your a 15-year-old hot babe) and then act in a rush when you do ask them a question. It's really annoying.
If you have a Teavana near you, they have a 'buy one, get one cup of tea free for mom' printable coupon. Good until Mother's Day. (There are restrictions, of course.)
I'll take the ignoring teens over the pushy folks I always seem to run into! Can we trade? :)
Or most importantly right now the whole credit card fraud thing, if they didn't already suck I would avoid them like mad just to not have to deal with getting a new card and all that jazz.
I just placed an order with them online and got a call from the fraud prevention department of my credit card company. If I hadn't known about the breach I would have been concerned about the call.
Della Terra's awfully generous with their coupon codes; I feel like they've shared them every week the past 3 or something, ha. Dunno how long it'll last but they've got one for 20% off right now, SUNNY20. I'd use it myself except I uh, already did two times in a row the last few times they had discounts.
Thanks, just came to see if there was a della terra code before placing an order to send to a friend. WE were working together last week and she saw all my fun teas and was jealous.
Odds are high there will be 30% off all weekend. I think it's been that way every weekend for months. Tempting!
They DO a sale every weekend, they're pretty awesome!
Haha yes. I'm beginning to realize with them it's sort of, you can automatically reprice their teas with 20-30% off and think of that as their "actual" prices, because the sales/coupons are ever present.
(So not complaining…love them and love how affordable they are.)
Well I did place the order for my friend, but if there's another sale, I may have to restock some of my Lemon Chiffon. :)
A&D is having free shipping to canada and the US for mothers day again, until friday. this is a FANTASTIC deal for canadians at least.
Just got a Teavana coupon in the mail for $10 off a purchase of $50+. I won't be using it, but anyone else is welcome to. The code to use it online is "SUMMER10". I'm happy to drop the actual coupon in the mail for anyone in the US who wants to use it in-store. Expires 6/30/13.
Chapters has the harney and sons- amnesia teas in their clearance section now. 30% off choco nut, lingonberry and the safari breakfast.
Some republic of teas are 75% off too
That sounds pretty tempting but I wouldn't know which ones to get. Better to just back off instead of researching it. :P
For anyone in NYC, the Tea and Honey Store on Lexington between 63rd and 64th is having a 50% off sale on everything including Les Palais Des Thes. I just picked up a sampler! :-)
Free Stainless Steel Mesh Ball Infuser on orders over $50 coupon code: I-WANT-ONE-M35H
We are only a couple of year sold. If you sign up to our newsletter you get a 15% coupon code. Plus, we just added 9 new blends to our inventory.


text:
Without a Roadmap: What Direction will ACOs Travel?
By Sandra E. Quilty, Baudino Law Group, PLC, Des Moines, IA
Federal officials representing the Federal Trade Commission (FTC), the Centers for Medicare and Medicaid Services (CMS), and the Health and Human Services (HHS) Office of the Inspector General convened a public workshop 1 to discuss how to form an accountable care organization (ACO) without violating fraud and abuse laws. 2 The workshop, attended by more than 300 health industry representatives, was kicked off by opening remarks from the CMS Administrator, the FTC Chairman, and the HHS Inspector General. The federal officials discussed creation and development of ACOs section 3022 waiver authority 3 , and the creation of new safe harbors and exceptions with regard to fraud and abuse laws.
The overall tenor of the agencies' opening remarks was a genuine desire for integrated care to thrive and not to stifle or hinder competition given the unique opportunity presented in healthcare reform legislation. "We want to make sure ACOs are not unduly inhibited by existing fraud and abuse laws . . . [they] should not stand in the way of improving quality and reducing costs through ACOs" stated HHS Inspector General Daniel Levinson. The federal officials discussed cooperation among the government agencies and assured a continued collaboration with hospitals and physicians for ACOs.
CMS Administrator Dr. Donald Berwick started the day long workshop by discussing the need for "consistent commitment to cooperation." He opined that to achieve this level of collaboration, an ACO must be centered on the patient and the patient's preferences. ACOs have to be comprised of providers who share in the decision making process and are transparent, thereby being willing to be accountable for true patient-centered care. In addition, for optimum operation, the ACO must be "data rich." This data will enable use of real-time information needed for a clinical decision as a patient moves through the so-called "cooperative."
Both FTC Chairman Jon Leibowitz and HHS Inspector General Levinson focused their opening remarks on safe harbor creation and advocating for an "expedited review process" for those ACOs that fall outside of the safe-harbor. The federal officials echoed the need for innovation, creativity, and "fresh thinking." "We need your real-world experience to help us understand what kind of ACOs you're considering, and how you see them operating in the healthcare marketplace," Leibowitz said.
The panelists consisted of an esteemed group of industry professionals representing the full gamut of healthcare. In fact, several in attendance were named "Top 10 People to Know in the World of ACOs" as noted in a recent publication. 4 Some of the panelists included: Gloria Austin, CEO of Brown & Toland, a San Francisco-based Independent Practice Association; Dr. Lawrence Casalino M.D., Chief of the Division of Outcomes and Effectiveness at Weill-Cornell Medical College; Dr. Lee Sacks, M.D., President of Advocate Physician Partners in Illinois; and Dr. Ceci l Wilson, President of the American Medical Association. Panelists considered levels of integration and circumstances under which permissible price negotiations may occur among the ACO providers without risking potential antitrust implications; and, how best to encourage formation so that competition would improve quality and affordability in any geographic market. Also addressed throughout the day were issues of whether Section 3022 waiver authority should be exercised by CMS and what safeguards, if any, should be implemented. Allowing an ACO a waiver with regard to the fraud and abuse laws may lend itself to an environment where ACOs could experiment more with structure and formation without risking running afoul of well established fraud and abuse laws. Generally, the waiver authority was discussed quite favorably by the panelists and there seemed to be consensus that if waiver authority was utilized then the protection should be applied uniformly. 5
Many seemed to agree that a "level playing field" was ideal if the industry were to move quickly. Some discussed the need for experimentation and flexibility to allow ACOs to flourish. Others raised the need for flexibility with regard to reorganizing and restructuring if the ACO model chosen did not initially prove to be successful. Many panelists raised the notion of accreditation to provide the type of guidance needed in creating an ACO. Regardless of any type of accreditation, an ACO can be certain that it will need to be able to produce data-rich evidence of its quality measures to prove and establish that it is functioning and operating appropriately and as intended by the federal healthcare reform legislation. Should the FTC create any safe harbor or exceptions, it will undoubtedly be a balancing act with regard to the scale needed to meet objectives and any potential market power type issues.
Panelists stressed electronic health records (EHRs), enabling technologies and interoperability as critical pieces in achieving integration required for the ACO. 6 A few of the panelists shared their first- hand experiences on how accessibility to real-time clinical data translates into real clinical value. An administrative system encompassing such enabling technology would foster an infrastructure that allowed for ongoing monitoring. The ability to share, capture and analyze data is an essential element to a healthy and robust ACO. This type of data would allow an ACO to continuously evaluate and monitor its overall culture and internal behaviors. Capturing and analyzing the data internally would also allow the ACO to ensure there is no fragmentation but rather cost and care integration. An ACO would be better positioned to see how best to achieve accountability and true clinical efficiencies by employing this type of data analysis.
To compete in the new marketplace developing as a result of the Patient Protection and Affordable Care Act, an ACO's infrastructure – governance, leadership and management -- will need to have clarity and transparency to achieve the integration needed for true care coordination. After all, with this new type of model, the physician – especially the primary care physician – is the true steward of accountability. Moreover, an ACO needs to consider performance measures in areas such as clinical processes, cost and quality, and overall patient experience with or without any potential regulatory guidance. As panelist Elizabeth Gilbertson 7 stated, a solid formula for performance metric setting is the total sum of aggregating total cost, total quality and total patient experience. This should ultimately result in a system of care that is genuinely patient centered.
Several panelists discussed the need for regulatory guidance for both providers and payors in establishing performance metrics. Internal operational metrics are necessary, but should be set at an achievable bar and constantly raised to encourage improvement – improvement with regard to performance measures as well as total cost management. From an operational standpoint, performance measures could be continuously refined based on the clinical data to ensure quality metrics are being met and improved on an ongoing basis. Physicians should be engaged in the development process, as they are the true heart and soul of an ACO. As stated by Panelist Dr. Cecil Wilson, President of the American Medical Association, quality performance committees made up of physicians enables buy-in and allows measures to be followed better because they are established by the clinical experts.
In addition to integration, the panelists discussed market power and issues of exclusivity given current consolidation trends the market is experiencing as a result of the ACO. While some discussed the fact that consolidation need not be automatically equated to gaining market power, payors discussed their experiences with market power issues such as price increases in markets where certain providers are dominant. There was a great deal of discussion regarding the need for scale in order to absorb the cost of infrastructure investment, delivering cost effective care efficiently, and spreading risk. While a number of advisory opinions issued by the FTC discuss non-exclusivity in great detail, a plethora of the panelists believed exclusivity to be critical for an ACO to best achieve high level integration and not fragment physician loyalties. One panelist went as far to suggest that without exclusivity it simplyundoes many of the core goals needed and desired for success.
The day-long workshop affirmed for many that while much more guidance is needed, it will take time, patience, collaboration, flexibility, experimentation, and transparency to get the ACO model right. As ACOs are formed, no single ACO model will look alike. "If you have seen one, you have seen one," stated one of the panelists. While many in the industry may differ on the extent and level of detail of potential regulatory guidance, any exception, safe harbor(s), or exercise of waiver authority will have to be a process closely monitored by all. Regulators are in the earliest stages of determining what factors to consider in employing, if at all, its Section 3022 waiver authority. The same can be said for development and creation of any safe-harbors or exceptions. Regulators will have to consider that current fraud and abuse laws may not be in line with the objectives of the new ACO model and what its success may engender despite the great deal of information and guidance already in existence for clinical integration. Through this workshop and comments already provided by representatives in the industry, regulators will have a better sense of how best to proceed and provide guidance for hospitals and physicians as the new marketplace takes shape.
The ABA Health eSource is distributed automatically to members of the ABA Health Law Section . Please feel free to forward it! Non-members may also sign up to receive the ABA Health eSource.


text:
October is Vegetarian Awareness Month, and while it is only Oct. 5, I'm happy to say I have not had any meat yet this month. It honestly hasn't been hard for me, but then again, I don't eat meat all that often to begin with. Anyway, the reason I mention veggie month is because my next review and contest is all about a vegetarian dragon! 🙂
I received a copy of the book "Herb, The Vegetarian Dragon" from Barefoot Books and was immediately impressed by the beautiful and vibrant illustrations. The book is about a dragon named Herb who tends to a nice vegetable garden in the forest and is a vegetarian, unlike the rest of the carnivorous dragons in the forest, lead by Meathook, who enjoy raiding the castle for amusement (and tasty eats). The knights of the castle set forth to rid the forest of the meat-eating dragons, but the meat-eaters hear of their plan and hide out in a cave. Meanwhile Herb is the only dragon left in the forest. The knights find him, assume he is a meat-eater as well and capture him. Herb is about to meet his fate at the mouths of hungry alligators when an unlikely hero emerges and saves the day.
Herb, The Vegetarian Dragon is a story about diversity, acceptance and living together in harmony. I do want to note that it's recommended for ages 4-10 and I have to agree that the content is a bit more mature than toddler-level. When the book first arrived in the mail, I read it to Ava (3) before I had read it for myself or realized it was meant for older children. (Oops.) While she expressed some concern about Herb's safety and other parts of the book, I explained it to her well and she did OK with it. We have since read the book a few times and she knows how it ends which I think reassures her that Herb will be OK even when the knights are tying him up.
Herb, The Vegetarian Dragon would make a great addition to any child's library. The illustrations are gorgeous and the book teaches some valuable lessons about life.
Barefoot Books – founded by two moms in 1993 – "offers products that are thoughtfully written, beautifully designed and illustrated, and have high educational value." If Herb isn't your cup of tea, check out the rest of the site. There's a huge variety of books available – many of which celebrate other cultures. I love that their books are free of media, toy or candy references, and are unique and meaningful children's literature. Most of the paperback books sell for $6.99 and the hardcover books sell for between $16.99-19.99. I suggest keeping Barefoot Books in mind for birthdays and holiday shopping for the kids this year.
Win it!
If you'd like to win your own copy of Herb, The Vegetarian Dragon, please leave me a COMMENT below STATING WHAT BOOK YOU LIKE BEST FROM BAREFOOT BOOKS. Make sure to include a valid email address so I can contact you if you win. The deadline to enter is Tuesday, Oct. 16. The winner will be chosen at random (using random.org) and announced on Oct. 17. It's not required, but please feel free to post about this contest on your blog if you'd like to share the love. 🙂


text:
American Physical Society Sites|APS|Journals|PhysicsCentral|Physics
- American Physical Society Sites
- Publications
- Meetings & Events
- Programs
- Membership
- Policy & Advocacy
- Careers In Physics
- Newsroom
- About APS
- Become a Member
I'm currently a sophomore majoring in Physics at Rutgers University. I have always been curious about naturals phenomenon around me, seeking explanations and questioning my teachers. Being introduced in High School to Physics, Biology and Chemistry, Physics appears to be the science that could answer most of my questions. Naturally, I knew that I had to major in Physics in college. My interest in the area grew with every Physics course that I took; the more I learn about the subject, the more I want to expand my study beyond bachelor degree and eventually study biophysics in graduate school.


text:
Skip to Main Navigation. Each navigation link will open a list of sub navigation links.
A loan is a form of aid that must be repaid with interest. Federal Perkins Loan repayments are made to the University of Wyoming. If you have any outstanding principal or interest owing on any student loan (Perkins/NDSL) borrowed prior to July 1, 1987, you begin paying six months after you graduate, leave school, or drop below half-time status. If you are a new borrower who has no previous outstanding principal or interest owing on any loan, you begin repayment nine months after you graduate, leave school or drop below half-time status. You may be allowed up to 10 years to repay your loan. The amount of your payment depends on the size of your debt, but you must pay at least $40 per month.
Note: When you leave UW, you must complete a loan exit interview.
Certain conditions may make you eligible to defer the repayment of your Federal Perkins Loan. These conditions include at least part-time enrollment at a post-secondary school; volunteer work in Peace Corps or ACTION programs or comparable full-time volunteer work for a tax-exempt organization; active duty in the U.S. armed forces, Commissioned Corps of U.S. Public Health Service or National Oceanic and Atmospheric Administration Corps; temporary total disability or inability to work because you are caring for a temporarily totally disabled spouse or other dependent; working at an eligible internship; unemployment; or (if the loan was borrowed before 7/1/1993) being the mother of preschool-age children and working at a salary of no more than $1 over the minimum wage.
If you default on a Federal Perkins Loan and the school is unable to collect, it will harm your credit rating and the federal government may take action to recover the loan. You cannot receive further financial aid if you are in default status.
Your loan will be canceled if you die (upon receipt of an appropriate death certificate) or become totally and permanently disabled. Part of your loan will be canceled for each complete academic year you are a full-time teacher of handicapped children or for each year you teach math, science, foreign languages, or bi-lingual education in a designated elementary or secondary school. In both of these cases, your entire loan will have been canceled after the fifth consecutive year of teaching. To find out whether the school you are teaching in is a designated school, contact the Office of Student Financial Operations. The same is also true if you are a nurse or medical technician providing health care services, or an employee of a private or public non-profit child or family service agency working with high-risk children from low-income communities. Part of your loan will be canceled for each year of full-time teaching in specified Head Start programs. After the seventh year, your entire loan will have been canceled. Up to 70% of your loan can be canceled for service as a Peace Corps or VISTA volunteer--30% during the first two years, 40% during the next two years. If you serve as an enlisted person in certain selected specialties of the U.S. Army, the Army Reserves, the Army National Guard, the Department of Defense will, as an enlistment incentive, repay a portion of your loan. If you think you may qualify, contact your recruiting officer. Up to 100% of your loan may be canceled when you make a timely request if you work full-time as a law enforcement or corrections officer after the loan period, if you borrowed after 11/29/90.
Experience has shown that the best and quickest way to resolve a problem is to contact the person or office involved. If you are having a problem related to your loan, and you are unable to resolve that problem to your satisfaction with UW, you may contact the Ombudsman's office in the U.S. Department of Education. A representative of that office will research your situation, help you to understand the loan process and requirements, and intervene on your behalf should that become necessary.
Customer Service Phone:877-557-2575
Website: Contact Ombudsman Group


text:
Sales Have Ended
Sales Have Ended
3on3 4on4 5on5 Development Basketball League
The best way to produce development time through play, is to offer 3on3 and 4on4 competitive platforms for younger ages that can act as a transition from recreational to competitive 5on5 basketball. For our earlier age groups, we are applying the philosophies of Mini-Basket in conjunction with Basketball Canada and the Ontario Basketball Association. Our Wednesday evenings offer something for all ages, genders and skill levels, with 3on3, 4on4 & 5on5 games. Players are divided into teams where they can compete against peers within their community. All skill levels from beginners on up.
Fall Season @ Knox CS 410 North Scugog Court
Wednesdays Boys & Girls League
Fall Dates: October 11 to November 29, 2017
Time/Ages: 6:00 pm - 7:00 pm (Ages 6-9)
Time/Ages: 7:00 pm - 8:00 pm (Ages 10-16)
Cost: $125/Season (8 weeks) or $250/Year (4 seasons)
How to Register?
- Select the most recent date
- Select your Ticket (i.e. Summer Season)
- Check Out and process your registration form
Registrations can be completed onsite, please contact for locations options and hours - 289-314-1152


text:
Most anyone who's read Max Brooks' 2006 apocalyptic novel "World War Z" has been excited since hearing that it's being adapted for the big screen. Despite a complete oversaturation of Zombies in the last few years (I'll still take that over sparkling Vampires), fans seemed relieved to hear the movie would stay pure as possible to the book, was supported by an all-star cast and a hefty Hollywood budget and would be produced by none-other than the movie's leading character, Brad Pitt. However one thing we all know is that any movie can fail - and if the rumors are true, World War Z might be a full on, 50 foot free-fall belly flop onto wet pavement.
According to the project's screenwriter, Damon Lindelof in the June Issue of Vanity Fair, the project has hit disastrous proportions, including being over-budget by 50 million, troubles with inexperienced director Marc Forester, last minute re-writes, and incoherent storylines. We're not just talking minor script changes either - it was realized after filming was finished, that the entire 40 minute ending was "abrupt and incoherent".
Not going to lie, I had just assumed the movie "11-11-11" was about a bunch of people hanging out and looking at their office calendars on the date. Turns out it's about it's about a guy who travels to Barcelona and starts to realize the date might be a warning sign for the end of the world. Or something. To be honest I'm not sure what the movie is about from the new trailer and I kind of like my idea better.
Good news for fans of "The Walking Dead"! AMC has renewed the show for a 3rd season. No word as to when they'll start filming the next season or when it'll air. My guess would be sometime in late 2014 based on how quickly they put out the second season. Let the countdown begin.
And I can't imagine anyone is surprised by this one but it's my journalistic duty to tell you that Paramount Pictures is currently scratching their heads trying to figure out what the story will be for "Paranormal Activity 4". Here's a suggestion, how about the ladies from the last 3 films all meet up in a strip club and audition for jobs? Oh and a demon breaks some glasses or something too.
And here's your first look at Scarlett Johansson as a alien disguised as an attractive woman in stone washed jeans who abducts hitchhikers in the new film "Under the Skin". If she was wearing a Poison t-shirt she would have been my dream girl when I was 13.
The 3-D craze has donned its leather jacket and pointed its motorcycle towards the ramp as Paramount Pictures has announced plans to make a new "Ring" movie, this time in 3-D. David Loucka has already been tapped to pen the script in what Paramount hopes is a reinvention of the franchise.
"Scream" scribe Kevin Williamson put to rest the rumors that Jamie Kennedy will be returning for "Scream IV". "We're not that universe where you can bring people back from the dead. That would be just a cheat." Good to see that there's still some integrity in Hollywood!
While not really news in and of its self, be sure to keep your eyes peeled for this sweet "Nightmare on Elm Street" popcorn bucket when you head out to catch the remake this weekend!
In a recent press junket for his latest feature "Robin Hood", director Ridley Scott let slip that the both of the upcoming "Alien" prequels will be released in 3D. If you're paying attention, you'll notice that I used the word 'both'. Scott also let it be known that he is planning on two prequels for the franchise.
"Machete" is no longer the only feature to spawn from the fake trailers shown in 2007's "Grindhouse". The next trailer to be stretched to full length will be "Hobo With a Shotgun" starring none other that Rutger Hauer.


text:
Miya Stone was a super sexy Peruvian babe that Levi had met while getting his check up at the doctor's office. When he dropped his pants for her, and she got a look at that cock, she knew she had to have it. Before Levi left the office, he made sure to get Miya's number. A couple days later Tony and Levi were hanging out, and he called Miya to tell her to send him some naughty pics on her phone. That got her horny, so she asked Levi if she could come over. Of course, he was down. When Miya got there, she didn't expect to see Tony and the camera.
But she didn't even really care because she'd come for Levi's cock and nothing was going to stop her now. Levi rubbed that hot Latina pussy and got her wet. Miya then had fun sucking his big dick. But after a while she wanted it in her. Levi was more than happy to fulfill that request. He banged her tight Latina pussy deep and hard all over the apartment. Levi switched up to several different positions and made Miya cum hard. Then he had his turn and busted a huge load of cum all over her pretty face.


text:
New Lung Association in California Report Assigns Grades to All California Cities and Counties
On Key Tobacco Control Policies and Follows Money Trail to See How Tobacco Industry Addicts Children
While California again falls short in adequately funding tobacco prevention programs to protect children and curb tobacco-caused disease, Northern California cities and counties show mixed results according to American Lung Association State of Tobacco Control 2013reports released today. The national report tracks progress on key tobacco control policies at the federal and state levels, assigning grades based on whether laws are adequately protecting citizens from the enormous toll tobacco use takes on lives and the economy.
In conjunction with the national report, the American Lung Association in California released its State of Tobacco Control 2013 – California Local Grades, which issues grades for all cities and counties in California on local tobacco control policies including those for smokefree outdoor environments, smokefree housing, and reducing sales of tobacco products. To view the complete California report, including grades for cities in Northern California such as Sacramento and Chico, visit www.lung.org/california.
Davis received the highest grade in the region with an overall tobacco control grade of B, while Sacramento received an overall C grade. In addition, the municipalities of Anderson, Shasta Lake, Redding, Chico, and Orland made advances in protecting their residents from secondhand smoke by restricting smoking in multi-unit housing common areas and in outdoor areas. Grades were mixed throughout, highlighting the need for local officials to do more to protect Northern California residents from tobacco-caused death and disease.
"Cities and counties in California have always led the way with strong tobacco control policies, and that continues to this day," said Stephanie Yoder, Chair of the Leadership Board of the American Lung Association in California – Sacramento. "Safeguarding our communities from the negative consequences of tobacco is essential. These grades represent tangible lives and real health outcomes."
Once a national leader in tobacco control policies, California's efforts are now lagging. While the state earned an A grade for smokefree air policies, it received a D for its low cigarette tax, an F for failing to adequately fund tobacco prevention and control programs, and another F for poor coverage of smoking cessation and treatment services.
Exacerbating California's weakened position on tobacco prevention, the state has not increased its cigarette tax since 1999 and now ranks 33rd in the country at 87 cents per pack, compared to the US average of $1.48 per pack. States including Texas, Oklahoma and Montana now have higher tobacco taxes than California.
Although California receives $68 million in tobacco-related revenue annually, it spends a meager 15 percent of what the Centers for Disease Control and Prevention recommends to adequately fund tobacco prevention programs and services to help people quit smoking.
According to the U.S. Surgeon General, the failure of states to invest in policies and programs to reduce tobacco use has resulted in 3 million new youth smokers in the United States. Every year in California, 34,400 kids start smoking while tobacco use causes an estimated 37,000 deaths annually and costs the state's economy more than $18 billion in healthcare costs and lost productivity - a tremendous burden that the state cannot afford.
"The California Legislature must make it a priority to pass a tobacco tax and invest the money in California's pioneering tobacco prevention program," said Marsha Ramos, Chair, American Lung Association in California Governing Board. "Tobacco taxes for tobacco prevention will save the state billions of dollars in health care costs while preventing kids from ever beginning to smoke and helping current smokers quit."
Despite progress at the local level, California joins many other states in neglecting to implement proven strategies to save lives and reduce tobacco-caused disease. This year's State of Tobacco Control report shows that money is often at the root of the leading cause of preventable death, as state policymakers fail to fight tobacco interests.
A tobacco industry contributions report by the American Lung Association in California's Center for Tobacco Policy & Organizing shows that tobacco interests spent nearly $5 million during the first year of the 2011-2012 election cycle (January 2011-June 2012) to influence state legislators and policies in California. In addition, Big Tobacco spent more than $46 million to defeat Proposition 29 in June 2012, which would have increased California's cigarette tax by $1.00 per pack and directed over $600 million annually to cancer research, tobacco prevention and quit smoking programs.
"We need to do more to fight the influence of tobacco interests in California politics," said Ramos. "It's time for the California Legislature to remove Big Tobacco's welcome mat. Our state elected officials have an opportunity to change course in 2013 and make big strides in the fight to end tobacco-caused death and disease. It's going to take a great deal of political will, but we are confident our elected officials are up to the challenge. Our children's health is depending on them."
About the American Lung Association
Now in its second century, the American Lung Association is the leading organization working to save lives by improving lung health and preventing lung disease. With your generous support, the American Lung Association is "Fighting for Air" through research, education and advocacy. For more information about the American Lung Association or to support the work it does, call 1-800-LUNG-USA (1-800-586-4872) or visit www.lung.org/california.


text:
Table of contents:
Open MPI supports the verbs and (starting in version 1.2)
udapl APIs from the OpenFabrics Alliance (OFA) software
stack. The OpenFabrics software stack supports both InfiniBand and
iWARP networks. Note that the OpenFabrics Alliance used to be known
as the OpenIB project -- so if you're thinking "OpenIB", you're
thinking the right thing. Open MPI's support of the OpenFabrics stack
is provided through the multiple different components; see the table
Open MPI also supported the Mellanox VAPI (mVAPI) software stack
through the v1.2 series. See this FAQ
entry for more details on VAPI support.
iWARP is fully supported as of the Open MPI v1.3 release.
RoCE is fully supported as of the Open MPI v1.4.4 release.
For more details on how to run Open MPI over RoCE, see this FAQ entry.
OFED (OpenFabrics Enterprise Distribution) is basically the release
mechanism for the OpenFabrics software packages. So OFED releases are
officially tested and released versions of the OpenFabrics stacks.
The "Download" section of the OpenFabrics web site has
links for the various OFED releases.
Additionally, several InfiniBand vendors have their own releases of
the OFED stack, customized for their hardware and/or support
requirements. Consult with your IB vendor for more details.
Yes, Open MPI is included in the OFED software. And yes, you
can easily install a later version of Open MPI on OFED-based
clusters, even if you're also using the Open MPI that was included in
You can simply download the Open MPI version that you want and install
it to an alternate directory from where the OFED-based Open MPI was
installed. You therefore have multiple copies of Open MPI that do not
conflict with each other. Make sure you set the PATH and
LD_LIBRARY_PATH variables to point to exactly one of your Open MPI
installations at a time, and never try to run an MPI executable
compiled with one version of Open MPI with a different version of Open
Ensure to specify to build Open MPI with OpenFabrics support (e.g.,
use the "--with-openib=/usr/local/ofed" option to Open MPI's
configure command, potentially replacing /usr/local/ofed with the
path of your OFED installation).
The following versions of Open MPI shipped (or will soon ship)
in OFED (note that OFED stopped including MPI implementations as of
NOTE: A prior version of this
FAQ entry specified that "v1.2ofed" would be included in OFED v1.2,
representing a temporary branch from the v1.2 series that included
some OFED-specific functionality. All of this functionality was
included in the v1.2.1 release, so OFED v1.2 simply included than.
Some public betas of "v1.2ofed" releases were made available, but
this version was never officially released.
Before the iWARP vendors joined the OpenFabrics Alliance, the
project was known as OpenIB. Open MPI's support for this software
stack was originally written during this timeframe -- the name of the
group was "OpenIB", so we named the BTL openib.
Since then, iWARP vendors joined the project and it changed names to
"OpenFabrics". Open MPI did not rename its BTL mainly for
historical reasons -- we didn't want to break compatibility for users
who were already using the openib BTL name in scripts, etc.
Yes, but only through the Open MPI v1.2 series; mVAPI support
was removed starting with v1.3.
The mVAPI support is an InfiniBand-specific BTL (i.e., it will not
work in iWARP networks), and reflects the prior generation of
InfiniBand software stacks.
However, the Open MPI team is doing no new work with mVAPI-based
networks; all effort is being put into the openib BTL. We'll
provide critical bug fixes for the mvapi BTL, but we consider the
openib BTL to be the future of InfiniBand and iWARP support in Open
Generally, much of the information contained in this FAQ category
applies to both the OpenFabrics openib BTL and the mVAPI mvapi BTL
-- simply replace openib with mvapi to get similar results.
However, new features and options are continually being added to the
openib BTL (and are being listed in this FAQ) that will not be
back-ported to the mvapi BTL. So not all items in this FAQ will
apply to the mvapi BTL.
We encourage users to switch to the OFED-based stack if possible.
In general, you specify that the openib BTL
components should be used. However, note that you should also
specify that the self BTL component should be used. self is for
loopback communication (i.e., when an MPI process sends to itself),
and is technically a different communication channel than the
OpenFabrics networks. For example:
shell$ mpirun --mca btl openib,self ...
Failure to specify the self BTL may result in Open MPI being unable
to complete send-to-self scenarios (meaning that your program will run
fine until a process tries to send to itself).
Note that openib,self is the minimum list of BTLs that you might
want to use. It is highly likely that you also want to include the
vader or sm (shared memory) BTLs in the list as well, like this:
shell$ mpirun --mca btl openib,self,vader ...
# Or
shell$ mpirun --mca btl openib,self,sm ...
See this FAQ
entry for more details on selecting which MCA plugins are used at
Finally, note that if the openib component is available at run time,
Open MPI should automatically use it by default (ditto for self).
Hence, it's usually unnecessary to specify these options on the
mpirun command line. They are typically only used when you want to
be absolutely positively definitely sure to use the specific BTL.
No. See this FAQ entry for more details.
The ompi_info command can display all the parameters
available for the openib BTL component:
# Show the openib BTL parameters
# Note that Open MPI v1.8 and later will only show an abbreviated list
# of parameters by default. Use "--level 9" to show all available
# parameters.
shell$ ompi_info --param btl openib --level 9
In order for us to help you, it is most helpful if you can
run a few steps before sending an e-mail to both perform some basic
troubleshooting and provide us with enough information about your
environment to help you. Please include answers to the following
questions in your e-mail:
Gather up this information and see
this page about how to submit a help request to the user's mailing
"Registered" memory means two things:
These two factors allow network adapters to move data between the
network fabric and physical RAM without involvement of the main CPU or
operating system.
Note that many people say "pinned" memory when they actually mean
"registered" memory.
However, a host can only support so much registered memory, so it is
treated as a precious resource. Additionally, the cost of registering
(and unregistering) memory is fairly high. Open MPI takes aggressive
steps to use as little registered memory as possible (balanced against
performance implications, of course) and mitigate the cost of
registering and unregistering memory.
With OpenFabrics (and therefore the openib BTL component),
you need to set the available locked memory to a large number (or
better yet, unlimited) -- the defaults with most Linux installations
are usually too low for most HPC applications that utilize
OpenFabrics. Failure to do so will result in a error message similar
to one of the following (the messages have changed throughout the
release versions of Open MPI):
WARNING: It appears that your OpenFabrics subsystem is configured to only
allow registering part of your physical memory. This can cause MPI jobs to
run with erratic performance, hang, and/or crash.
This may be caused by your OpenFabrics vendor limiting the amount of
physical memory that can be registered. You should investigate the
relevant Linux kernel module parameters that control how much physical
memory can be registered, and increase them to allow registering all
physical memory on your machine.
See this Open MPI FAQ item for more information on these Linux kernel
module parameters:
Local host: node02
Registerable memory: 32768 MiB
Total memory: 65476 MiB
Your MPI job will continue, but may be behave poorly and/or hang.
[node06:xxxx] mca_mpool_openib_register: \
error registering openib memory of size yyyy errno says Cannot allocate memory
[x,y,z][btl_openib.c:812:mca_btl_openib_create_cq_srq] \
error creating low priority cq for mthca0 errno says Cannot allocate memory
libibverbs: Warning: RLIMIT_MEMLOCK is 32768 bytes.
This will severely limit memory registrations.
The OpenIB BTL failed to initialize while trying to create an internal
queue. This typically indicates a failed OpenFabrics installation or
faulty hardware. The failure occurred here:
Host: compute_node.example.com
OMPI source: btl_openib.c:828
Function: ibv_create_cq()
Error: Invalid argument (errno=22)
Device: mthca0
You may need to consult with your system administrator to get this
problem fixed.
The OpenIB BTL failed to initialize while trying to allocate some
locked memory. This typically can indicate that the memlock limits
are set too low. For most HPC installations, the memlock limits
should be set to "unlimited". The failure occurred here:
Host: compute_node.example.com
OMPI source: btl_opebib.c:114
Function: ibv_create_cq()
Device: Out of memory
Memlock limit: 32767
You may need to consult with your system administrator to get this
problem fixed. This FAQ entry on the Open MPI web site may also be
There are two typical causes for Open MPI being unable to register
memory, or warn that it might not be able to register enough memory:
There are two ways to control the amount of memory that a user
process can lock:
* soft memlock <number>
where "<number>" is the number of bytes that you want user
processes to be allowed to lock by default (presumably rounded down to
an integral number of pages). "<number>" can also be
* hard memlock <number>
where "<number>" is the maximum number of bytes that you want
user processes to be allowed to lock (presumably rounded down to an
integral number of pages). "<number>" can also be
Users can increase the default limit by adding the following to their
shell startup files for Bourne style shells (sh, bash):
shell$ ulimit -l unlimited
Or for C style shells (csh, tcsh):
shell% limit memorylocked unlimited
This effectively sets their limit to the hard limit in
/etc/security/limits.d (or limits.conf). Alternatively, users can
set a specific number instead of "unlimited," but this has limited
usefulness unless a user is aware of exactly how much locked memory
will require (which is difficult to know since Open MPI manages locked
memory behind the scenes).
It is important to realize that this must be set in all shells where
Open MPI processes using OpenFabrics will be run. For example, if you are
using rsh or ssh to start parallel jobs, it will be necessary to
set the ulimit in your shell startup files so that it is effective
on the processes that are started on each node.
More specifically -- it may not be sufficient to simply execute the
following, because the ulimit may not be in effect on all nodes
where Open MPI processes will be run:
shell$ ulimit -l unlimited
shell$ mpirun -np 2 my_mpi_application
Ensure that the limits you've set (see this FAQ entry) are actually being
used. There are two general cases where this can happen:
That is, in some cases, it is possible to login to a node and
not have the "limits" set properly. For example, consider the
following post on the Open MPI User's list:
In this case, the user noted that the default configuration on his
Linux system did not automatically load the pam_limits.so
upon rsh-based logins, meaning that the hard and soft
limits were not set.
There are also some default configurations where, even though the
maximum limits are begin set system-wide in limits.d (or
limits.conf on older systems), something
during the boot procedure sets the default limit back down to a low
number (e.g., 32k). In this case, you may need to override this limit
on a per-user basis (described this FAQ
entry), or effectively system-wide by putting "ulimit -l unlimited"
(for Bourne-like shells) in a strategic location, such as:
Also, note that resource managers such as SLURM, Torque/PBS, LSF,
etc. may affect OpenFabrics jobs in two ways:
The files in limits.d (or the limits.conf file) does not usually
apply to resource daemons! The limits.s files usually only applies
to rsh or ssh-based logins. Hence, daemons usually inherit the
system default of maximum 32k of locked memory (which then gets passed
down to the MPI processes that they start). To increase this limit,
you typically need to modify daemons' startup scripts to increase the
limit before they drop root privliedges.
Finally, note that some versions of SSH have problems with getting
correct values from /etc/security/limits.d/ (or limits.conf) when
using privilege separation. You may notice this by ssh'ing into a
node and seeing that your memlock limits are far lower than what you
have listed in /etc/security/limits.d/ (or limits.conf) (e.g., 32k
instead of unlimited). Several web sites suggest disabling privilege
separation in ssh to make PAM limits work properly, but others imply
that this may be fixed in recent versions of Open SSH.
If you do disable privilege separation in ssh, be sure to check with
your local system administrator and/or security officers to understand
the full implications of this change. See this Google search link for more information.
OpenFabrics network vendors provide Linux kernel module
parameters controlling the size of the size of the memory translation
table (MTT) used to map virtual addresses to physical address. The
size of this table controls the amount of physical memory that can be
registered for use with OpenFabrics devices.
With Mellanox hardware, two parameters are provided to control the
size of this table:
The amount of memory that can be registered is calculated using this
In newer hardware:
max_reg_mem = (2^log_num_mtt) * (2^log_mtts_per_seg) * PAGE_SIZE
In older hardware:
max_reg_mem = num_mtt * (2^log_mtts_per_seg) * PAGE_SIZE
At least some versions of OFED (community OFED,
Mellanox OFED, and upstream OFED in Linux distributions) set the
default values of these variables FAR too low! For example, in
some cases, the default values may only allow registering 2 GB -- even
if the node has much more than 2GB of physical memory.
Mellanox has advised the Open MPI community to increase the
log_num_mtt value (or num_mtt value), not the log_mtts_per_seg
value (even though an
IBM article suggests increasing the log_mtts_per_seg value).
It is recommended that you adjust log_num_mtt (or num_mtt) such
that your max_reg_mem value is at least twice the amount of physical
memory on your machine (setting it to a value higher than the amount
of physical memory present allows the internal Mellanox driver tables
to handle fragmentation and other overhead). For example, if a node
has 64 GB of memory and a 4 KB page size, log_num_mtt should be set
to 24 and (assuming log_mtts_per_seg is set to 1). This will allow
processes on the node to register:
max_reg_mem = (2^24) * (2^1) * (4 kB) = 128 GB
NOTE: Starting with OFED 2.0, OFED's default kernel parameter values
should allow registering twice the physical memory size.
Open MPI prior to v1.2.4 did not include specific
configuration information to enable RDMA for short messages on
ConnectX hardware. As such, Open MPI will default to the safe setting
of using send/receive semantics for short messages, which is slower
than RDMA.
To enable RDMA for short messages, you can add this snippet to the
bottom of the $prefix/share/openmpi/mca-btl-openib-hca-params.ini
[Mellanox Hermon]
vendor_id = 0x2c9,0x5ad,0x66a,0x8f1,0x1708
vendor_part_id = 25408,25418,25428
use_eager_rdma = 1
mtu = 2048
Enabling short message RDMA will significantly reduce short message
latency, especially on ConnectX (and newer) Mellanox hardware.
Open MPI uses registered memory in several places, and
therefore the total amount used is calculated by a somewhat-complex
formula that is directly influenced by MCA parameter values.
It can be desirable to enforce a hard limit on how much registered
memory is consumed by MPI applications. For example, some platforms
have limited amounts of registered memory available; setting limits on
a per-process level can ensure fairness between MPI processes on the
same host. Another reason is that registered memory is not swappable;
as more memory is registered, less memory is available for
(non-registered) process code and data. When little unregistered
memory is available, swap thrashing of unregistered memory can occur.
Each instance of the openib BTL module in an MPI process (i.e.,
one per HCA port and LID) will use up to a maximum of the sum of the
following quantities:
Note that this MCA parameter was introduced in v1.2.1.
By default, btl_openib_free_list_max is -1, and the list size is
unbounded, meaning that Open MPI will try to allocate as many
registered buffers as it needs. If btl_openib_free_list_max is
greater than 0, the list will be limited to this size. Each entry
in the list is approximately btl_openib_eager_limit bytes --
some additional overhead space is required for alignment and
internal accounting. btl_openib_eager_limit is the
maximum size of an eager fragment.
By default, btl_openib_free_list_max is -1, and the list size is
unbounded, meaning that Open MPI will allocate as many registered
buffers as it needs. If btl_openib_free_list_max is greater
than 0, the list will be limited to this size. Each entry in the
list is approximately btl_openib_max_send_size bytes -- some
additional overhead space is required for alignment and internal
accounting. btl_openib_max_send_size is the maximum
size of a send/receive fragment.
Each MPI process will use RDMA buffers for eager fragments up to
btl_openib_eager_rdma_num MPI peers. Upon receiving the
btl_openib_eager_rdma_threshhold'th message from an MPI peer
process, if both sides have not yet setup
btl_openib_eager_rdma_num sets of eager RDMA buffers, a new set
will be created. The set will contain btl_openib_max_eager_rdma
buffers; each buffer will be btl_openib_eager_limit bytes (i.e.,
the maximum size of an eager fragment).
In general, when any of the individual limits are reached, Open MPI
will try to free up registered memory (in the case of registered user
memory) and/or wait until message passing progresses and more
registered memory becomes available.
Use the ompi_info command to view the values of the MCA parameters
described above in your Open MPI installation:
# Note that Open MPI v1.8 and later require the "--level 9"
# CLIP option to display all available MCA parameters.
shell$ ompi_info --param btl openib --level 9
See this FAQ entry
for information on how to set MCA parameters at run-time.
iWARP support is only available in version Open MPI v1.3 or
For the Chelsio T3 adapter, you must have at least OFED v1.3.1 and
Chelsio firmware v6.0. Download the firmware from service.chelsio.com and put the uncompressed t3fw-6.0.0.bin
file in /lib/firmware. Then reload the iw_cxgb3 module and bring
up the ethernet interface to flash this new firmware. For example:
# Note that the URL for the firmware may change over time
shell# cd /lib/firmware
shell# wget http://service.chelsio.com/drivers/firmware/t3/t3fw-6.0.0.bin.gz
[...wget output...]
shell# gunzip t3fw-6.0.0.bin.gz
shell# rmmod iw_cxgb3 cxgb3
shell# modprobe iw_cxgb3
# This last step may happen automatically, depending on your
# Linux distro (assuming that the ethernet interface has previously
# been properly configured and is ready to bring up). Substitute the
# proper ethernet interface name for your T3 (vs. ethX).
shell# ifup ethX
If all goes well, you should see a message similar to the following in
your syslog 15-30 seconds later:
kernel: cxgb3 0000:0c:00.0: successful upgrade to firmware 6.0.0
Open MPI will work without any specific configuration to the openib
BTL. Users wishing to performance tune the configurable options may
wish to inspect the receive queue values. Those can be found in the
"Chelsio T3" section of mca-btl-openib-hca-params.ini.
Prior to Open MPI v1.0.2, the OpenFabrics (then known as
"OpenIB") verbs BTL component did not check for where the OpenIB API
could return an erroneous value (0) and it would hang during startup.
Starting with v1.0.2, error messages of the following form are
ibv_create_qp: returned 0 byte(s) for max inline data
This is caused by an error in older versions of the OpenIB user
library. Upgrading your OpenIB stack to recent versions of the
OpenFabrics software should resolve the problem. See this post on the
Open MPI user's list for more details:
Open MPI, by default, uses a pipelined RDMA protocol.
Additionally, in the v1.0 series of Open MPI, small messages use
send/receive semantics (instead of RDMA -- small message RDMA was added in the v1.1 series).
For some applications, this may result in lower-than-expected
bandwidth. However, Open MPI also supports caching of registrations
in a most recently used (MRU) list -- this bypasses the pipelined RDMA
and allows messages to be sent faster (in some cases).
For version the v1.1 series, see this FAQ entry for more
information about small message RDMA, its effect on latency, and how
to tune it.
To enable the "leave pinned" behavior, set the MCA parameter
mpi_leave_pinned to 1. For example:
shell$ mpirun --mca mpi_leave_pinned 1 ...
NOTE: The mpi_leave_pinned parameter was
broken in Open MPI v1.3 and v1.3.1 (see
this announcement). mpi_leave_pinned functionality was fixed in v1.3.2.
This will enable the MRU cache and will typically increase bandwidth
performance for applications which reuse the same send/receive
NOTE: The v1.3 series enabled "leave
pinned" behavior by default when applicable; it is usually
unnecessary to specify this flag anymore.
Starting with Open MPI version 1.1, "short" MPI messages are
sent, by default, via RDMA to a limited set of peers (for versions
prior to v1.2, only when the shared receive queue is not used). This
provides the lowest possible latency between MPI processes.
However, this behavior is not enabled between all process peer pairs
because it can quickly consume large amounts of resources on nodes
(specifically: memory must be individually pre-allocated for each
process peer to perform small message RDMA; for large MPI jobs, this
can quickly cause individual nodes to run out of memory). Outside the
limited set of peers, send/receive semantics are used (meaning that
they will generally incur a greater latency, but not consume as many
system resources).
This behavior is tunable via several MCA parameters:
Note that long messages use a different protocol than short messages;
messages over a certain size always use RDMA. Long messages are not
affected by the btl_openib_use_eager_rdma MCA parameter.
Also note that, as stated above, prior to v1.2, small message RDMA is
not used when the shared receive queue is used.
Note that this answer generally pertains to the Open MPI v1.2
series. Later versions slightly changed how large messages are
Open MPI uses a few different protocols for large messages. Much
detail is provided in this
The btl_openib_flags MCA parameter is a set of bit flags that
influences which protocol is used; they generally indicate what kind
of transfers are allowed to send the bulk of long messages.
Specifically, these flags do not regulate the behavior of "match"
headers or other intermediate fragments.
The following flags are available:
Open MPI defaults to setting both the PUT and GET flags (value 6).
Open MPI uses the following long message protocols:
NOTE: Per above, if striping across multiple
network interfaces is available, only RDMA writes are used. The
reason that RDMA reads are not used is solely because of an
implementation artifact in Open MPI; we didn't implement it because
using RDMA reads only saves the cost of a short message round trip,
the extra code complexity didn't seem worth it for long messages
(i.e., the performance difference will be negligible).
Note that the user buffer is not unregistered when the RDMA
transfer(s) is(are) completed.
Note that phases 2 and 3 occur in parallel. Each phase 3 fragment is
unregistered when its transfer completes (see the
paper for more details).
Also note that one of the benefits of the pipelined protocol is that
large messages will naturally be striped across all available network
The sizes of the fragments in each of the three phases are tunable by
the MCA parameters shown in the figure below (all sizes are in units
of bytes):
This protocol behaves the same as the RDMA Pipeline protocol when
the btl_openib_min_rdma_size value is infinite.
The Open MPI v1.3 (and later) series generally use the same
protocols for sending long messages as described for the v1.2
series, but the MCA parameters for the RDMA Pipeline protocol
were both moved and renamed (all sizes are in units of bytes):
The change to move the "intermediate" fragments to the end of the
message was made to better support applications that call fork().
Specifically, there is a problem in Linux when a process with
registered memory calls fork(): the registered memory will
physically not be available to the child process (touching memory in
the child that is registered in the parent will cause a segfault or
other error). Because memory is registered in units of pages, the end
of a long message is likely to share the same page as other heap
memory in use by the application. If this last page of the large
message is registered, then all the memory in that page -- to include
other buffers that are not part of the long message -- will not be
available to the child. By moving the "intermediate" fragments to
the end of the message, the end of the message will be sent with copy
in/copy out semantics and, more importantly, will not have its page
registered. This increases the chance that child processes will be
able to access other memory in the same page as the end of the large
message without problems.
Some notes about these parameters:
Note that messages must be larger than
btl_openib_min_rdma_pipeline_size (a new MCA parameter to the v1.3
series) to use the RDMA Direct or RDMA Pipeline protocols.
Messages shorter than this length will use the Send/Receive protocol
(even if the SEND flag is not set on btl_openib_flags).
When mpi_leave_pinned is set to 1, Open MPI aggressively
tries to pre-register user message buffers so that the RDMA Direct
protocol can be used. Additionally, user buffers are left
registered so that the de-registration and re-registration costs are
not incurred if the same buffer is used in a future message passing
NOTE: Starting with Open MPI v1.3,
mpi_leave_pinned is automatically set to 1 by default when
applicable. It is therefore usually unnecessary to set this value
NOTE: The mpi_leave_pinned MCA parameter
has some restrictions on how it can be set starting with Open MPI
v1.3.2. See this FAQ
entry for details.
Leaving user memory registered when sends complete can be extremely
beneficial for applications that repeatedly re-use the same send
buffers (such as ping-pong benchmarks). Additionally, the fact that a
single RDMA transfer is used and the entire process runs in hardware
with very little software intervention results in utilizing the
maximum possible bandwidth.
Leaving user memory registered has disadvantages, however. Bad Things
happen if registered memory is free()ed, for example --
it can silently invalidate Open MPI's cache of knowing which memory is
registered and which is not. The MPI layer usually has no visibility
on when the MPI application calls free() (or otherwise frees memory,
such as through munmap() or sbrk()). Open MPI has implemented
complicated schemes that intercept calls to return memory to the OS.
Upon intercept, Open MPI examines whether the memory is registered,
and if so, unregisters it before returning the memory to the OS.
These schemes are best described as "icky" and can actually cause
real problems in applications that provide their own internal memory
allocators. Additionally, only some applications (most notably,
ping-pong benchmark applications) benefit from "leave pinned"
behavior -- those who consistently re-use the same buffers for sending
and receiving long messages.
It is for these reasons that "leave pinned" behavior is not enabled
by default. Note that other MPI implementations enable "leave
pinned" behavior by default.
Also note that another pipeline-related MCA parameter also exists:
mpi_leave_pinned_pipeline. Setting this parameter to 1 enables the
use of the RDMA Pipeline protocol, but simply leaves the user's
memory registered when RDMA transfers complete (eliminating the cost
of registering / unregistering memory during the pipelined sends /
receives). This can be beneficial to a small class of user MPI
When mpi_leave_pinned is set to 1, Open MPI aggressively
leaves user memory registered with the OpenFabrics network stack after
the first time it is used with a send or receive MPI function. This
allows Open MPI to avoid expensive registration / deregistration
function invocations for each send or receive MPI function.
However, registered memory has two drawbacks:
The second problem can lead to silent data corruption or process
failure. As such, this behavior must be disallowed. Note that the
real issue is not simply freeing memory, but rather returning
registered memory to the OS (where it can potentially be used by a
different process). Open MPI has two methods of solving the issue:
How these options are used differs between Open MPI v1.2 (and
earlier) and Open
MPI v1.3 (and later).
Be sure to read this FAQ entry first.
Open MPI 1.2 and earlier on Linux used the ptmalloc2 memory allocator
linked into the Open MPI libraries to handle memory deregistration.
On Mac OS X, it uses an interface provided by Apple for hooking into
the virtual memory system, and on other platforms no safe memory
registration was available. The ptmalloc2 code could be disabled at
Open MPI configure time with the option --without-memory-manager,
however it could not be avoided once Open MPI was built.
ptmalloc2 can cause large memory utilization numbers for a small
number of applications and has a variety of link-time issues.
Therefore, by default Open MPI did not use the registration cache,
resulting in lower peak bandwidth. The inability to disable ptmalloc2
after Open MPI was built also resulted in headaches for users.
Open MPI v1.3 handles
leave pinned memory management differently.
With Open MPI 1.3, Mac OS X uses the same hooks as the 1.2 series,
and most operating systems do not provide pinning support. However,
the pinning support on Linux has changed. ptmalloc2 is now by default
built as a standalone library (with dependencies on the internal Open
MPI libopen-pal library), so that users by default do not have the
problematic code linked in with their application. Further, if
OpenFabrics networks are being used, Open MPI will use the mallopt()
system call to disable returning memory to the OS if no other hooks
are provided, resulting in higher peak bandwidth by default.
To utilize the independent ptmalloc2 library, users need to add
-lopenmpi-malloc to the link command for their application:
shell$ mpicc foo.o -o foo -lopenmpi-malloc
Linking in libopenmpi-malloc will result in the OpenFabrics BTL not
enabling mallopt() but using the hooks provided with the ptmalloc2
library instead.
To revert to the v1.2 (and prior) behavior, with ptmalloc2 folded into
libopen-pal, Open MPI can be built with the
--enable-ptmalloc2-internal configure flag.
When not using ptmalloc2, mallopt() behavior can be disabled by
disabling mpi_leave_pined:
shell$ mpirun --mca mpi_leave_pinned 0 ...
Because mpi_leave_pinned behavior is usually only useful for
synthetic MPI benchmarks, the never-return-behavior-to-the-OS behavior
was resisted by the Open MPI developers for a long time. Ultimately,
it was adopted because a) it is less harmful than imposing the
ptmalloc2 memory manager on all applications, and b) it was deemed
important to enable mpi_leave_pinned behavior by default since Open
MPI performance kept getting negatively compared to other MPI
implementations that enable similar behavior by default.
As with all MCA parameters, the mpi_leave_pinned parameter (and
mpi_leave_pinned_pipeline parameter) can be set from the mpirun
command line:
Prior to the v1.3 series, all the usual methods
to set MCA parameters could be used to set mpi_leave_pinned.
However, starting with v1.3.2, not all of the usual methods to set
MCA parameters apply to mpi_leave_pinned. Due to various
operating system memory subsystem constriaints, Open MPI must react to
the setting of the mpi_leave_pinned parameter in each MPI process
before MPI_INIT is invoked. Specifically, some of Open MPI's MCA
parameter propagation mechanisms are not activated until during
MPI_INIT -- which is too late for mpi_leave_pinned.
As such, only the following MCA parameter-setting mechanisms can be
used for mpi_leave_pinned and mpi_leave_pinned_pipeline:
To be clear: you cannot set the mpi_leave_pinned MCA parameter via
Aggregate MCA parameter files or normal MCA parameter files. This is
expected to be an acceptable restriction, however, since the default
value of the mpi_leave_pinned parameter is "-1", meaning
"determine at run-time if it is worthwhile to use leave-pinned
behavior." Specifically, if mpi_leave_pinned is set to -1, if any
of the following are true when each MPI processes starts, then Open
MPI will use leave-pinned bheavior:
mpirun --mca mpi_leave_pinned 1 ..."
Note that if either the environment variable
OMPI_MCA_mpi_leave_pinned or OMPI_MCA_mpi_leave_pinned_pipeline is
set to to "-1", then the above indicators are ignored and Open MPI
will not use leave-pinned behavior.
Users may see the following error message from Open MPI v1.2:
WARNING: There are more than one active ports on host '%s', but the
default subnet GID prefix was detected on more than one of these
ports. If these ports are connected to different physical OFA
networks, this configuration will fail in Open MPI. This version of
Open MPI requires that every physically separate OFA subnet that is
used between connected MPI processes must have different subnet ID
This is a complicated issue.
What it usually means is that you have a host connected to multiple,
physically separate OFA-based networks, at least 2 of which are using
the factory-default subnet ID value (FE:80:00:00:00:00:00:00). Open
MPI can therefore not tell these networks apart during its
reachability computations, and therefore will likely fail. You need
to reconfigure your OFA networks to have different subnet ID values,
and then Open MPI will function properly.
Please note that the same issue can occur when any two physically
separate subnets share the same subnet ID value -- not just the
factory-default subnet ID value. However, Open MPI only warns about
the factory default subnet ID value because most users do not bother
to change it unless they know that they have to.
All this being said, note that there are valid network configurations
where multiple ports on the same host can share the same subnet ID
value. For example, two ports from a single host can be connected to
the same network as a bandwidth multiplier or a high-availability
configuration. For this reason, Open MPI only warns about finding
duplicate subnet ID values, and that warning can be disabled. Setting
the btl_openib_warn_default_gid_prefix MCA parameter to 0 will
disable this warning.
See this FAQ entry for instructions
on how to set the subnet ID.
Here's a more detailed explanation:
Since Open MPI can utilize multiple network links to send MPI traffic,
it needs to be able to compute the "reachability" of all network
endpoints that it can use. Specifically, for each network endpoint,
Open MPI calculates which other network endpoints are reachable.
In OpenFabrics networks, Open MPI uses the subnet ID to differentiate
between subnets -- assuming that if two ports share the same subnet
ID, they are reachable from each other. If multiple, physically
separate OFA networks use the same subnet ID (such as the default
subnet ID), it is not possible for Open MPI to tell them apart and
therefore reachability cannot be computed properly.
You can use any subnet ID / prefix value that you want.
However, Open MPI v1.1 and v1.2 both require that every physically
separate OFA subnet that is used between connected MPI processes must
have different subnet ID values.
For example, if you have two hosts (A and B) and each of these
hosts has two ports (A1, A2, B1, and B2). If A1 and B1 are connected
to Switch1, and A2 and B2 are connected to Switch2, and Switch1 and
Switch2 are not reachable from each other, then these two switches
must be on subnets with different ID values.
See this FAQ entry for instructions
on how to set the subnet ID.
It depends on what Subnet Manager (SM) you are using. Note that changing the subnet ID will likely kill
any jobs currently running on the fabric!
shell# /etc/init.d/opensm stop
shell# opensm -c -o
shell# /etc/init.d/opensm start
When multiple active ports exist on the same physical fabric
between multiple hosts in an MPI job, Open MPI will attempt to use
them all by default. Open MPI makes several assumptions regarding
active ports when establishing connections between two hosts. Active
ports that have the same subnet ID are assumed to be connected to the
same physical fabric -- that is to say that communication is possible
between these ports. Active ports with different subnet IDs
are assumed to be connected to different physical fabric -- no
communication is possible between them. It is therefore very important
that if active ports on the same host are on physically separate
fabrics, they must have different subnet IDs. Otherwise Open MPI may
attempt to establish communication between active ports on different
physical fabrics. The subnet manager allows subnet prefixes to be
assigned by the administrator, which should be done when multiple
fabrics are in use.
The following is a brief description of how connections are
established between multiple ports. During initialization, each
process discovers all active ports (and their corresponding subnet ID)
on the local host and shares this information with every other process
in the job. Each process then examines all active ports (and the
corresponding subnet ID) of every other process in the job and makes a
one-to-one assignment of active ports within the same subnet. If the
number of active ports within a subnet differ on the local process and
the remote process, then the smaller number of active ports are
assigned, leaving the rest of the active ports out of the assignment
between these two processes. Connections are not established during
MPI_INIT, but the active port assignment is cached and upon the first
attempted use of an active port to send data to the remote process
(e.g., via MPI_SEND), a queue pair (i.e., a connection) is established
between these ports. Active ports are used for communication in a
round robin fashion so that connections are established and used in a
fair manner.
NOTE: This FAQ entry generally applies to v1.2 and beyond. Prior to
v1.2, Open MPI would follow the same scheme outlined above, but would
not correctly handle the case where processes within the same MPI job
had differing numbers of active ports on the same physical fabric.
Measuring performance accurately is an extremely difficult
task, especially with fast machines and networks. Be sure to read this FAQ entry for
many suggestions on benchmarking performance.
Pay particular attention to the discussion of processor affinity and
NUMA systems -- running benchmarks without processor affinity and/or
on CPU sockets that are not directly connected to the bus where the
HCA is located can lead to confusing or misleading performance
Fully static linking is not for the weak, and is not
recommended. But it is possible.
Read both this
FAQ entry and this FAQ entry
in their entirety.
The answer is, unfortunately, complicated. Be sure to also
see this FAQ entry as
Hence, you can reliably query OMPI to see if it has support for
fork() and force OMPI to abort if you request fork support and it
doesn't have it.
This feature is helpful to users who switch around between multiple
clusters and/or versions of Open MPI; they can script to know whether
the OMPI that they're using (and therefore the underlying IB stack)
has fork support. For example:
have_fork_support=`ompi_info --param btl openib --level 9 --parsable | grep have_fork_support:value | cut -d: -f7`
if test "$have_fork_support" = "1"; then
# Happiness / world peace / birds are singing
# Despair / time for Haagen Daas
Alternatively, you can skip querying and simply try to run your job:
shell$ mpirun --mca btl_openib_want_fork_support 1 --mca btl openib,self ...
Which will abort if OMPI's openib BTL does not have fork support.
All this being said, even if Open MPI is able to enable the
OpenFabrics fork() support, it does not mean
that your fork()-calling application is safe.
It is important to note that memory is registered on a per-page basis;
it is therefore possible that your application may have memory
co-located on the same page as a buffer that was passed to an MPI
communications routine (e.g., MPI_Send() or MPI_Recv()) or some
other internally-registered memory inside Open MPI. You may therefore
accidentally "touch" a page that is registered without even
realizing it, thereby crashing your application.
This is unfortunately no way around this issue; it was intentionally
designed into the OpenFabrics software stack. Please complain to the
OpenFabrics Alliance that they should really fix this problem!
Starting with v1.2.6, the MCA pml_ob1_use_early_completion
parameter allows the user (or administrator) to turn off the "early
completion" optimization. Early completion may cause "hang"
problems with some MPI applications running on OpenFabrics networks,
particularly loosely-synchronized applications that do not call MPI
functions often. The default is 1, meaning that early completion
optimization semantics are enabled (because it can reduce
point-to-point latency).
See Open MPI ticket #1224 for further
NOTE: This FAQ entry only applies to the v1.2 series. This
functionality is not required for v1.3 and beyond because of changes
in how message passing progress occurs. Specifically, this MCA
parameter will only exist in the v1.2 series.
InfiniBand QoS functionality is configured and enforced by the Subnet
Manager/Administrator (e.g., OpenSM).
Open MPI (or any other ULP/application) sends traffic on a specific IB
Service Level (SL). This SL is mapped to an IB Virtual Lane, and all
the traffic arbitration and prioritization is done by the InfiniBand
HCAs and switches in accordance with the priority of each Virtual
For details on how to tell Open MPI which IB Service Level to use,
please see this FAQ entry.
InfiniBand 2D/3D Torus/Mesh topologies are different from the more
common fat-tree topologies in the way that routing works: different IB
Service Levels are used for different routing paths to prevent the
so-called "credit loops" (cyclic dependencies among routing path
input buffers) that can lead to deadlock in the network.
Open MPI complies with these routing rules by querying the OpenSM
for the Service Level that should be used when sending traffic to
each endpoint.
Note that this Service Level will vary for different endpoint pairs.
For details on how to tell Open MPI to dynamically query OpenSM for
IB Service Level, please refer to this FAQ entry.
NOTE: 3D-Torus and other torus/mesh IB
topologies are supported as of version 1.5.4.
There are two ways to tell Open MPI which SL to use:
1. Providing the SL value as a command line parameter for the openib BTL
Use the btl_openib_ib_service_level MCA parameter to tell
openib BTL which IB SL to use:
shell$ mpirun --mca btl openib,self,vader --mca btl_openib_ib_service_level N ...
The value of IB SL N should be between 0 and 15, where 0 is the
default value.
NOTE: Open MPI will use the same SL value
for all the endpoints, which means that this option is not valid for
3D torus and other torus/mesh IB topologies.
2. Querying OpenSM for SL that should be used for each endpoint
Use the btl_openib_ib_path_record_service_level MCA
parameter to tell the openib BTL to query OpenSM for the IB SL
that should be used for each endpoint. Open MPI will send a
PathRecord query to OpenSM in the process of establishing connection
between two endpoints, and will use the IB Service Level from the
PathRecord response:
shell$ mpirun --mca btl openib,self,sm --mca btl_openib_ib_path_record_service_level 1 ...
NOTE: The
btl_openib_ib_path_record_service_level MCA parameter is supported
as of version 1.5.4.
First of all, what exactly is RoCE?
RoCE (which stands for RDMA over Converged Ethernet) provides
InfiniBand native RDMA transport (OFA Verbs) on top of lossless
Ethernet data link.
Since we're talking about Ethernet, there's no Subnet Manager, no
Subnet Administrator, no InfiniBand SL, nor any other InfiniBand Subnet
Administration parameters.
Connection management in RoCE is based on the OFED RDMACM (RDMA
Connection Manager) service:
So how does Open MPI run on top of RoCE?
OMPI uses the OFED Verbs-based openib BTL for traffic and its
internal rdmacm CPC (Connection Pseudo-Component) for establishing
connections for MPI traffic.
So if you just want the data to run over RoCE and you're
not interested in VLANs, PCP, or other VLAN tagging parameters, you
can just run OMPI with the openib BTL and rdmacm CPC:
shell$ mpirun --mca btl openib,self,sm --mca btl_openib_cpc_include rdmacm ...
(or set these MCA parameters in other ways)
How does Open MPI run with Routable RoCE (RoCEv2)?
Routable RoCE is supported in Open MPI starting v1.8.8.
In order to use it, RRoCE needs to be enabled from the command line.
shell$ mpirun --mca pml ob1 --mca btl openib,self,sm --mca btl_openib_cpc_include rdmacm --mca btl_openib_rroce_enable 1 ...
How does MXM run on top of RoCE?
In order to use RoCE with MXM, the Ethernet port
must be specified using the MXM_RDMA_PORTS environment parameter.
For example:
shell$ mpirun --mca pml cm -mca mtl mxm -x MXM_RDMA_PORTS=mlx4_0:2 ...
MXM can also be run with the Yalla PML. Using RoCE requires the same
environment parameter:
shell$ mpirun --mca pml yalla -x MXM_RDMA_PORTS=mlx4_0:2 ...
Older versions of MXM may also require an additional environment
shell$ mpirun --mca pml cm -mca mtl mxm -x MXM_RDMA_PORTS=mlx4_0:2 -x MXM_TLS=self,shm,ud ...
How does MXM run with Routable RoCE (RoCEv2)?
In order to use RRoCE when running with MXM, the following environment parameters need to be set:
shell$ mpirun --mca pml yalla -x MXM_RDMA_PORTS=mlx4_0:2 -x MXM_IB_GID_INDEX=1 ...
How do I tell Open MPI to use a specific RoCE VLAN?
When a system administrator configures VLAN in RoCE, every VLAN is
assigned with its own GID. The QP that is created by the
rdmacm CPC uses this GID as a Source GID. When OMPI (or
any other application for that matter) posts a send to this QP, the
driver checks the source GID to determine which VLAN the traffic is
supposed to use, and marks the packet accordingly.
Note that InfiniBand SL (Service Level) is not involved in this
process - marking is done in accordance with local kernel policy.
To control which VLAN will be selected, use the
btl_openib_ipaddr_include/exclude MCA parameters and
provide it with the required IP/netmask values. For
example, if you want to use a VLAN with IP 13.x.x.x:
shell$ mpirun --mca btl openib,self,sm --mca btl_openib_cpc_include rdmacm \
--mca btl_openib_ipaddr_include "184.108.40.206/8" ...
NOTE: VLAN selection in the OMPI v1.4 series works only with version
v1.4.4 or later.
XRC (eXtended Reliable Connection) decreases the memory consumption
of Open MPI and improves its scalability by significantly decreasing
number of QPs per machine.
XRC is available on Mellanox ConnectX family HCAs with OFED 1.4 and
See this FAQ entry for instructions
how to tell Open MPI to use XRC receive queues.
You can use the btl_openib_receive_queues MCA parameter to
specify the exact type of the receive queues for the OMPI to use.
This can be advantageous, for example, when you know the exact sizes
of messages that your MPI application will use -- Open MPI can
internally pre-post receive buffers of exactly the right size. See this paper for more
The btl_openib_receive_queues parameter
takes a colon-delimited string listing one or more receive queues of
specific sizes and characteristics. For now, all processes in the job
must use the same string. You can specify three kinds of receive
The default value of the btl_openib_receive_queues MCA parameter
is sometimes equivalent to the following command line:
shell$ mpirun ... --mca btl_openib_receive_queues P,128,256,192,128:S,2048,256,128,32:S,12288,256,128,32:S,65536,256,128,32 ...
In particular, note that XRC is (currently) not used by default.
NOTE: Open MPI chooses a default value of btl_openib_receive_queues
based on the type of OpenFabrics network device that is found. The
text file $openmpi_packagedata_dir/mca-btl-openib-device-params.ini
(which is typically
contains a list of default values for different OpenFabrics devices.
See that file for further explanation of how default values are
Per-Peer Receive Queues
Per-peer receive queues require between 1 and 5 parameters:
Example: P,128,256,128,16
Shared Receive Queues
Shared Receive Queues can take between 1 and 4 parameters:
Example: S,1024,256,128,32
XRC Queues
XRC queues take the same parameters as SRQs. Note that if you use
any XRC queues, then all of your queues must be XRC. Therefore,
to use XRC, specify the following:
shell$ mpirun --mca btl openib... --mca btl_openib_receive_queues X,128,256,192,128:X,2048,256,128,32:X,12288,256,128,32:X,65536,256,128,32 ...
NOTE: the rdmacm CPC is not supported with
XRC. Also, XRC cannot be used when btls_per_lid > 1.
NOTE: the rdmacm CPC cannot be used unless the first QP is per-peer.
FCA (which stands for Fabric Collective
Accelerator) is a Mellanox MPI-integrated software package
that utilizes CORE-Direct
technology for implementing the MPI collectives communications.
You can find more information about FCA on the product web page.
FCA is available for download here: http://www.mellanox.com/products/fca
Building Open MPI 1.5.x or later with FCA support
By default, FCA is installed in /opt/mellanox/fca. Use the following
configure option to enable FCA integration in OMPI:
shell$ ./configure --with-fca=/opt/mellanox/fca ...
Verifying the FCA Installation
To verify that Open MPI is built with FCA support, use the following command:
shell$ ompi_info --param coll fca --level 9 | grep fca_enable
A list of FCA parameters will be displayed if Open MPI has FCA support.
How do I tell Open MPI to use FCA?
shell$ mpirun --mca coll_fca_enable 1 ...
By default, FCA will be enabled only with 64 or more MPI processes.
To turn on FCA for an arbitrary number of ranks ( N ), please use
the following MCA parameters:
shell$ mpirun --mca coll_fca_enable 1 --mca coll_fca_np <N> ...
MXM is a MellanoX Messaging
library which provides enhancements to parallel communication libraries.
It is integrated into the Mellanox OFED stack, and also provided in
a stand-alone package.
The MXM stand-alone package is available for download here: http://www.mellanox.com/products/mxm
MX>M requirements
Building Open MPI with MXM support
By default, MXM is installed in /opt/mellanox/mxm. Use the following
configure option to enable MXM support in OMPI:
shell$ ./configure --with-mxm=/opt/mellanox/mxm ...
Enabling MXM in Open MPI 1.6.1 or later
In Open MPI 1.6 and above, MXM will be enabled by default, regardless of the number of ranks.
Running an Open MPI job with MXM:
MXM can be run in one of two ways:
shell$ mpirun --mca pml cm --mca mtl mxm ...
shell$ mpirun --mca pml yalla ...
The Yalla PML calls MXM directly and therefore improves the
performance significantly. Note that the Yalla PML and CM PML are
mutually exclusive.
NOTE: Please note that the Yalla PML is available only from Open MPI
v1.10 and above.
In older versions of Open MPI (before v1.6), MXM is automatically
selected when the job has at least 128 MPI processes. To activate MXM
for any number of MPI processes, run the following command:
shell$ mpirun --mca mtl_mxm_np 0 ...
The --cpu-set parameter allows you to specify the logical CPUs to use in an MPI job.
From 'mpirun --help':
Comma-separated list of ranges specifying logical cpus allocated to this job.
shell$ mpirun -cpu-set 0,1,2,3 ...
The hwloc package can be used to get information about the topology on your host.
More information about hwloc is available here.
Here is a usage example with 'hwloc-ls'.
Consider the following command line:
shell$ mpirun --report-bindings -n 4 --oversubscribe --cpu-set 14,15,0,1 --bind-to hwthread --map-by hwthread hostname
[clx-orion-001:25028] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B./../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-001:25028] MCW rank 1 bound to socket 0[core 0[hwt 1]]: [.B/../../../../../../../../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-001:25028] MCW rank 2 bound to socket 0[core 7[hwt 0]]: [../../../../../../../B./../../../../../..][../../../../../../../../../../../../../..]
[clx-orion-001:25028] MCW rank 3 bound to socket 0[core 7[hwt 1]]: [../../../../../../../.B/../../../../../..][../../../../../../../../../../../../../..]
The explanation is as follows.
When 'hwloc-ls' is run, the output will show the mappings of physical cores to logical ones.
As per the example in the command line, the logical PUs 0,1,14,15 match the physical cores - 0 and 7. (as shown in the map above)
NUMANode L#0 (P#0 128GB)
Socket L#0 + L3 L#0 (35MB)
L2 L#0 (256KB) + L1d L#0 (32KB) + Core L#0
L2 L#7 (256KB) + L1d L#7 (32KB) + Core L#7
It is also possible to use 'hwloc-calc'. The following command line will show all the available logical CPUs on the host:
hwloc-calc all -I pu
The following will show two specific hwthreads specified by phys ids 0 and 1:
hwloc-calc -I pu --physical-input pu:0 pu:1
When using InfiniBand, Open MPI supports host communication between
separate subnets using the Mellanox IB-Router.
The support for IB-Router is available starting with Open MPI v1.10.3.
To enable routing over IB, follow these steps:
For example, to run the IMB benchmark on host1 and host2 which are on
separate subents (i.e., they have have different subnet_prefix
values), use the following command line:
shell$ mpirun -np 2 --display-map --map-by node -H host1,host2 \
--mca pml ob1 \
--mca btl self,sm,openib \
--mca btl_openib_cpc_include rdmacm \
--mca btl_openib_if_include mlx5_0:1 \
--mca btl_openib_gid_index 1 \
--mca btl_openib_allow_different_subnets 1 \
./IMB/src/IMB-MPI1 pingpong
NOTE: The rdmacm CPC cannot be used unless the first QP is per-peer.
If the default value of btl_openib_receive_queues is to use only SRQ
QPs, please set the first QP in the list to a per-peer QP.
Please see this FAQ entry for more
information on this mca parameter.


text:
Bears original frame
Signature: signed B. West and dated 1766, center left
About Benjamin West
Extremely popular among the 18th-century British aristocracy and royalty, Benjamin West's work is primarily composed of commissioned portraits and history paintings. West is best known for his 1770 painting The Death of General Wolfe, which caused a stir when it was displayed at the Royal Academy because the figures were shown wearing contemporary clothing rather than classical garb. Later in his career, West began to focus on religious subject matter in his work and began to devote a great deal of effort to teaching American-born artists in England, such as Charles Willson Peale and Matthew Pratt.
American, 1738-1820, Swarthmore, PA, United States, based in London, United Kingdom