text:
Someone contacted the list with the following claims:
Assertion 1: An agnostic is someone who is neither a theist (someone who believes a god exists) nor an atheist (someone who does not believe a god exists OR someone who denies a god exists).
While I agree with this, I soon found out I have different reasons for doing so. I go by the theologically classical definition of agnostic as someone who addresses knowledge regarding god, and finds it lacking, versus the Gnostic, who believes that knowledge about god is accessible and perhaps even that he has such knowledge. The person making claim 1 above, however, asserts that an agnostic is someone who neither believes nor disbelieves that a god exists. They wished to use that definition alone because it is what their friends agreed was right, and it was listed as a "colloquially" acceptable definition in his dictionary, some version of Merriam-Webster.
I already knew the dilemma this person was creating for himself. To define the agnostic off the bat as not being a person who believes a god exists leaves little room for then trying to defend that the agnostic is someone who does not believe a god exists. I think most people would see the problem with the position even before it unfolds:
If a "senior" is defined as one who is at or over 65 years old, and you assert that I am "not a senior," there is no escaping that you have just indicated I am not at or over 65. And the writer does agree that a theist is a person who does believe a god exists. Thankfully he understands at least one significant definition in theological terminology.
I knew he was going to encounter difficulty, then, in defending his claim that this agnostic is no atheist. Literally, anyone who is not a theist (someone who does believe a god exists) is an atheist (someone who does not believe a god exists). And this person who contacted us, let's call him "J," for brevity, agreed that the atheist is correctly defined as "someone who does not believe a god exists." In fact, he agreed to this during a call to the show–on the air–recorded for posterity, before he wrote to us.
So then, his argument begins with this:
The agnostic does not believe a god exists (or else he'd be a theist), but he also does not, not believe a god exists (or else he'd be an atheist). So that this magical being, the agnostic, both does "not believe" and also does not "not believe"–which is a logical impossibility. I can no more accomplish this as a human being than I can both be a senior and not be a senior.
I accepted the writer as merely a person who is ignorant regarding where the term "agnostic" originated and how it is used in actual theological discussion of these issues. The Gnostic movement is one that concerned itself with knowledge about god. Certainly knowledge is a subset of belief, but one that was the focus of the Gnostics that separated their ideology from broader definitions of belief. And I do not claim this distinction is without problems.
The term that defines the response to Gnosticism, "agnosticism," was coined by Thomas Huxley, in the mid-1800s, to describe his rejection of Gnosticism, and subsequently all claims to knowledge regarding gods. He was not claiming that no one has belief in gods and that belief in gods is unavailable to people. He was making a statement about a subset of belief, knowledge–which is usually far more narrowly (and problematically) defined.
The idea that an agnostic is simply someone who is wishy-washy about their belief in god is a misconception that has grown as discussion of "atheist issues" has become more common. And the reason it grows is that people love to talk about religion, but they don't seem to like to actually research it or inform their opinions before they open their mouths.
In short, it's like the word "founder," which means to have trouble staying afloat. The word fell out of common usage and began to be "replaced" with "flounder"–a type of fish. One suggestion is that people equated it to "flopping around" and being unable to move well, and that's why they began to describe something that doesn't progress well as "floundering," more and more often. Today, when you look up the word "flounder," it actually does generally have a secondary definition that makes it synonymous with "founder."
Dictionaries are wonderful tools. They inform us both of classically correct usage, but also must reflect common usage–which can become correct usage over time. With "agnostic," we have a common misconception that may one day find its way into a secondary form of correct usage. But it carries with it the problem of making all such defined agnostics atheists. And there was a moment when J began to realize this as a necessary conclusion, when he accused me of trying to say that all agnostics were, in fact, atheists. I honestly replied that I do not use his definition of "agnostic," and that I know many agnostic theists; but that if I am compelled to use the term "agnostic" by defining it off the top as a person who is not a theist, and, by necessity, then, a person who does not believe a god exists, then I cannot agree to the second half of the definition–which is logically impossible–that he also is not an atheist and not someone who does not believe a god exists. A human cannot be both someone who does "not believe" a claim and someone who does not "not believe" a claim.
So, this, I chalked up to ignorance and misconception on J's part. I did so, at least, until it went on for more than a few exchanges and I began to suspect there was more here than just simple correction of a misconception required. J was defending, not honestly communicating. A person honestly communicating would have, at or before this point, pretty well have said, "Maybe I hadn't thought this through, and it appears I might be working under some misconceptions." But not J. J has something to prove, which leads to his next assertion:
Assertion 2. The atheist is making a positive claim and we can extrapolate other atheist beliefs from the position "I do not believe a god exists."
Specifically, when J first called us, J wanted to be able to say that we can know what atheists do believe, by knowing what they do not believe. And that is simply not the case. And I should add that it's no more the case than me claiming I can know what someone believes when they claim they do believe a god exists. I have no idea what they mean by "god" nor what impact their god has on anything, including on themselves. But, more specifically, he thinks he knows what an atheist thinks of universal origins, according to his call. His argument is along the lines of this: Either a god created the universe or a god did not. If you don't believe in a god, then you must believe in naturalistic origins (and I assume that would lead to the common misconception that all atheists believe big bang, which I know to be false).
The idea here goes something like this: "I assert that fairies created the universe. If you do not believe in fairies, then I know how you think the universe was created."
Obviously that would be ludicrous. But as is so often the case, the theist can't see how absurd it is when you use "god" instead of "fairies." But using fairies, anyone should be able to see how ludicrous this claim becomes. Of course you could assert you know that however I believe the universe came to be, it is a non-fairy model. That much is fair–but as far as asserting tha
t you have some insight into what I do think pumped out the universe (if I'm not obstinately holding to steady-state theory, and asserting that the idea it was "produced" at all is nonsensical to me)–what I do believe about it–is unjustified.
Here I should note that J does not dispute the broad definition of atheist on the surface. If you show him a dictionary that indicates that the atheist either does not believe in the existence of gods OR believes no gods exist, J will say "OK." However, I don't think J really comprehends what he's agreeing to here–or at least he didn't at first.
The idea that I say you can be "one or the other" means that the "one" is not the "other." And while I think anyone could understand that, J is, apparently, not just anyone. I happened to pull a definition that read "disbelieve" rather than "does not believe," and J decided to fly with this. In fact, he tried to fly this to the moon. "Disbelieving," he asserted, is not at all the same as "not believing" something. I kid you not. This was his response.
Bear in mind that if I knew "disbelieve" would trip him up so badly, I'd have pulled a valid authoritative dictionary from the start that said "does not believe"–because they are out there. But since J had agreed during our call that an atheist is one who does "not believe" a god exists, it did not occur to me he'd now try to claim "does not believe" isn't valid since this one dictionary I pulled had "disbelieve." So, back-peddle number one is that he tried to duck out of his initial agreement that it's fair to label someone who does "not believe" a god exists is an "atheist."
And here we have a lesson in definitions. And by that I don't mean that there are not myriad dictionaries that will support than an atheist "does not believe" (if it's "disbelieve" that is all that is freaking you out) or that there are not myriad dictionaries that assert that "disbelieve" does include "not believe," but we need to see something here about broad and narrow definitions, in general, and how they must be understood by any fair and honest person:
If I assert that Word-X means "A" and you assert you are using it as "B," and I say you are wrong to claim it means "B," and we look it up in 6 dictionaries, and some say "A" and some say "B" and some say "A or B" or "A, B and sometimes C," then I am wrong even though "A" is not incorrect. I did not assert I use it as "A" and you use it as "B." I asserted it is incorrect to use it as "B." And I am wrong. And in our discussion about agnosticism, despite my knowledge that he was abusing the term by using a definition that represents a common misconception, I still agreed to accept it and roll with it. That's what people do when they are trying to have a fair and honest dialogue to understand what you think and why.
It is possible to find dictionaries to support that "disbelieve" means to reject belief in a way that condemns the claim (in this case, "a god exists") as false. But to claim that "disbelieve" does not mean "not believe" is to ignore all of the other dictionaries that assert that "not believe" is an acceptable usage of the word disbelieve. It is to tell me I am wrong to use "B", while "B" is supported by myriad authoritative sources. In order to stop me from rightly using a valid definition, the burden would be on you to demonstrate why those definitions are incorrect and the sources are faulty, or to demonstrate why those definitions might not apply in the context of our particular discussion. In this particular case, however, I even used J's own dictionary–Merriam Webster–to demonstrate "disbelieving" as "not believing." And he was still unwilling to to admit the words can validly be said to carry the same meaning.
At this point I could not give the benefit of the doubt–that this was ignorance rather than pride– any longer, so I asserted rather that dishonesty might be involved in some way as a motive. But it would be more true to say the motive for his unwillingness to accept what was in front of him was defensiveness. This, in my book, includes being willing to make ridiculous assertions in the face of rock-solid, contrary evidence, by way of lying to oneself and/or others. I think J was insulted by the "dishonest" comment–but it was that or "stupid," and of the two, I would think "dishonest" would be the more complimentary. However, admittedly, I might have gone for duplicitous, hypocritical or disingenuous.
Hypocrisy 1:
Eventually J stated that the agnostic does not believe god exists "on the face of it"–and I have no idea what difference it makes. If he does "not believe" a god exists, he is an atheist. If he does "not believe" because he's uncertain what to believe, because he's investigated and found it to be unjustified to believe, because he's drunk and it's Wednesday–it really doesn't matter. As long as we can honestly say this person does "not believe" a god exists (and if we agree, as J and I did, he's no theist, then we can), and as long as we can say that an atheist does "not believe" a god exists (and J agreed to this initially, and a dictionary survey and history would support this), then we cannot deny that this person is an atheist, while he is not a theist.
It no more matters why I don't believe in god than it matters why I do believe in god. And here is where we get into the sort of hypocrisy that could stir me to righteous indignation if I were to allow it.
Can you imagine how ridiculous and presumptuous it would be, if I went to a theist e-list and began asserting that only theists who believe a god exists because god has personally spoken to them are theists–and that anyone else doesn't really "believe" and is an "agnostic"? What if I asserted that those at the e-list who believe only because of what they've read in their Bibles can't be labeled "theists"?
Where do I sign up to cherry pick for theists which reasons for "belief" are valid reasons under the theist definition, "someone who believes a god exists"? Would J think that was rational of me, to go and tell theists that if their belief is based on "A," then it counts, but if they believe for reason "B," then their belief isn't really "belief" under the theist definition? The reason they believe is not relevant. All that matters is that they believe. The definition of "theist" doesn't have an asterisk leading to a note indicating that "if you believe for the following reasons, then 'theist' is not what you are." You can believe for any reason. And a you can not believe for any reason. You still believe or you still disbelieve. And whether you believe or disbelieve is all that matters to these definitions–not "why."
Hypocrisy 2:
In a context of a particular field, it is possible for definitions to have agreed upon meanings. For example, the term "stripper" in publishing used to mean a person who worked in preparing materials for pre-press. This is very different than what the general population thinks of when they think of "strippers." And I think we understand this pretty well. In theology, where theists equate "belief" with things like "faith," we are often confronted with models of the martyrs–those who exhibited such deep conviction to their views that they would suffer and die for
them. Theists make quite a verbal dog and pony show, often becoming offended at any slight to their "deeply held beliefs" or their "god," in which they believe and whom they "revere." This "belief" they speak of is important, sometimes life-altering, something they teach their children, something to spread to the far corners of the world, something that brings them great "joy" and "peace" and "happiness." Theists make it known that "belief" is no small thing. In fact, in the Bible it says that if a person "believes," they can be saved–receiving eternal bliss with god.
This is how the theist frames theistic "belief."
Further, during our call with J, Matt said belief was "acceptance of a claim as true," and J agreed. Later, the e-list exchange, we used a definition of "conviction of the truth of a claim," and J did not take issue–at first. The definition went along, accepted by both parties for a few exchanges. Then, suddenly, out of the blue, J decided that the sort of "belief" we were discussing, the belief in god that distinguishes a theist, was this sort of "belief":
Me: "Where is Tammy?"
You: "I'm not sure. I believe she said she was going to the store–but I might have not heard her correctly."
His point was that you can have varying degrees of belief in god–that not having conviction doesn't mean not "believing." So, the agnostic is like this–he is unsure and sort of "believes."
Beyond back-peddling–literally going back and saying that a definition he accepted twice isn't working out as well as he'd hoped, so he's now going to just reject it and claim I'm the one being unfair–there are two huge problems with J's assertion:
First of all, if this truly is "belief," then we have a problem with J's Assertion 1: If I believe a god exists, I'm a theist by definition–using any source you want to pick. And that means that, according to J, this person is no longer an agnostic, because he has belief in god, and, therefore, must be a theist (which J asserted his agnostic is absolutely not). If there can be "little" belief–then his agnostic would be correctly labeled a "theist"–and J must say there is no such thing as an agnostic. He would actually be saying that anyone who has doubts about the existence of gods has some small "belief"–and those who have no doubts and think a god does not exist, are atheists. So, rather than defend his agnostic, he has successfully defined his agnostic right out of existence.
But secondly, and even more pathetically and dishonestly, J is now taking his belief in god (he is a theist) and throwing it under the bus, in order to salvage his sorry position. No longer is "belief" a conviction or accepting a claim as true. No longer does god require deep faith and the courage to live and die by that conviction that He exists and came to save the world from sin. No longer does god demand worship and reverence and commitment. It seems that when the Bible talks about believing and being saved–it only means not being sure god doesn't exist. If that's the case, many atheists will be thrilled to learn they're saved according to the Bible–for they believe. And that's apparently what theists mean by "belief"–according to J.
Why is this surprising though? Why should it be a shock that when it supports a theist's argument for how wonderful it is to be a theist, belief is a conviction that can fulfill your life, but the moment you want to say that without that conviction, a person can't be said to "believe," then belief becomes nothing more than the thinnest shred of a doubt about the false nature of the any claim? In other words, I should say I "believe" fairies exist, according to J, because I have to admit that, logically speaking, I cannot "know" they do not exist, even though I really, really, really, really doubt they exist, and feel fair saying "I do not believe in fairies." I still "believe fairies exist" according to J, because I have to acknowledge that it would be logically unsupportable for me to assert that I know they absolutely do not.
And J keeps asking me why this is so hard?
From where J stands, it isn't possible that he's twisting in the wind. I sometimes have pity on him because it's hard to see a person humiliate himself repeatedly on this level; but then he acts like this is a debate or honest disagreement we're having, rather than me trying to educate an ignorant, defensive individual, and I get my perspective back.
This brings us back to "one OR the other." If these doubts constitute "belief," then we have a problem with the definition of atheist, with which J took no issue. J agrees that the atheist is either someone who disbelieves/does not believe a god exists OR someone who believes no god exists (denies a god exists). And here's the rub: If "disbelieve" means to accept a claim is false, and if "believe" means to be anything but 100 percent sure the claim is false–then what is the difference between "disbelieve/not believe" and "believing no god exists/denying god exists"?
J, who does not claim to take issue with the definition of "atheist" that reads, someone who disbleieves a god exists OR someone who believes no god exists (denies a god exists), is now trying to say that "disbelieving" and "believing the opposite" represent the same condition, thus rendering the "OR," and the definition of "atheist," nonsensical. In other words, while he claims to take no issue with the definition, J is actually trying to assert that an atheist is only someone who denies a god exists, and that a person who disbelieves a god exists is actually no different. In my last communication, I asked him what he imagines the dictionary (his Merriam Webster) is trying to demonstrate as the difference between "disbelieve" and "believe the opposite"? I just sent it this morning, so I can't report on the answer to that head-scratcher.
But I must say that I reject any claim that it is honest to assert that I "believe" fairies exist. I am unable to logically defend that it is impossible for a small race of magical woodland winged creatures to exist. Do I believe fairies exist? If you think it's reasonable to say I do, I know this guy, J, you really need to meet, because I suspect the two of you will really get along. But I do not expect anyone will ever find a fairy. I do not accept that all the writings about fairies are a compelling reason to think they exists. And if someone presented me with one, I would have to admit I have been "wrong" regarding the existence of fairies. But why? If J is correct, I always believed in them, since I always was willing to admit that I could not be 100 percent certain they do not exist.
Also, J rejected that agnosticism had anything to do with knowledge. And I submitted that theism and atheism addressed belief, not knowledge. And he never took issue with this. Meanwhile, he seems to be saying the atheist has to assert knowledge (certainty), and I don't see that in the definition. Surely an atheist could feel certain there is no god. But I simply note it is not necessary to be an atheist.
I sent J, several e-mails back, a link to a wonderful series of articles written by Austin Cline, who has been for many years the host of about.com's Agnosticism/Atheism section. I don't think J read the articles. That's too bad. If this is an issue that interests any of you, I encourage you to do some further reading at Cline's site. Here are a few links you might enjoy:
At his main page is a link to "Atheism 101," a series that talks about these same misconceptions.
h1 = document.getElementById("title").getElementsByTagName("h1")[0];h1.innerHTML = widont(h1.innerHTML); Here are a few quotes I thought were very appropriate to this discussion:
"Atheists are simply those who do not accept the truth of this claim — they may deny it out right, they may find it too vague or incomprehensible to evaluate properly, they may be waiting to hear support for the claim, or they may simply not have heard about it yet. This is a broad and diverse category and there is no particular counter-claim made by all atheists."
"Many have trouble comprehending that "not believing X" (not believe gods exist) doesn't mean the same as "believing not X" (believe gods do not exist). The placement of the negative is key: the first means not having the mental attitude that proposition X (gods exist) is true, the second means having the mental attitude that proposition X (gods exist) is false. The difference here is between disbelief and denial: the first is disbelief in the broad or narrow sense whereas the second is denial."
"A belief is the mental attitude that some proposition is true. For every given proposition, every person either has or lacks the mental attitude that it is true — there is no middle ground between the presence of absence of a belief. In the case of gods, everyone either has a belief that at least one god of some sort exists or they lack any such belief."
"A person who is an agnostic, who does not claim to know for sure if any gods exist, still either has some sort of belief in the existence of some sort of god (believing without knowing for sure is common in many subjects) or lacks a belief in the existence of any gods (not believing without knowing for sure may be more common). Confusing the definitions of atheism and agnosticism is a popular tactic with some religious theists because it allows them to essentially define the territory of debate in their favor. They should not, however, be permitted to misdefine and misrepresent basic categories in this manner."
I still am stunned at the presumptuousness of a theist calling an atheist public outreach program to argue with the hosts about what an atheist is, writing to an atheist educational foundation to assert they don't understand the definition of agnostic or atheist, and potentially going to Austin Cline's section to say that a person who has dealt in atheist issues for longer than I've been an atheist (and who has extensively handled this question particularly) doesn't have a clue about atheism. I would never dream of contacting the Baptist Convention to say they don't know the first thing about what a Baptist is. In fact, if I did get into a discussion with the president of a Baptist educational foundation (as J disputed with Matt D as well), and he told me that I misunderstood some aspect of what it means to be "Baptist," and dictionaries and reference sources largely supported his assertions–why wouldn't I back down and own up to my misconception? Austin Cline has a thought on that in one of his Atheism 101 articles:
"Another reason for insisting that only the narrow sense of atheism is relevant is that it allows the theist to avoid shouldering the principle burden of proof. You see, if atheism is simply the absence of a belief in any gods, then the principle burden of proof lies solely with the theist. If the theist cannot demonstrate that their belief is reasonable and justified, then atheism is automatically credible and rational. When a person is unable to do this, it can be easier to claim that others are in the same boat than to admit one's own failure."
I think this pretty well nails it.


text:
Jan. 15, 2010
KINGSTON, RI-The University of Massachusetts Dartmouth's indoor track and field team began their second semester action on a solid note at the Sorlien Memorial Invitational at the Henry H. Mackal Fieldhouse Friday.
The women's part of the meet was held Friday, and no team scores were recorded.
Top performers for the Corsairs included sophomore Patience Noah (Framingham, MA/Keefe Tech), who finished third in the triple jump at 34 feet, 10 ¼ inches; junior Linee Mello-Frost (New Bedford, MA/New Bedford HS), who was sixth in the 55 meter hurdles in 8.78 seconds; and sophomore Caitlin McGarigal (Shutesbury, MA/Amherst-Pelham Regional HS), who finished seventh at 3000 meters in 10:55.23. Mello-Frost's performance qualified her for the Open New England Championships, while McGarigal qualified for the New England Division III Championships.
"It was good competition for us after the holiday break," said UMass Dartmouth Head Coach Steve Gardiner. "Linee is ahead of where she was when ran 8.49 two years ago, and Caitlin improved her performance over what she ran in December. It was good to see Patience, Linee and Caitlin compete like that after a layoff. After being off for 25 days and training for a week, this meet was a good way for us to get the rust off, and a good warm-up for our meet at Springfield next weekend."
The Corsairs will return to action when the two-day Sorlien Memorial Invitational concludes with men's action Saturday.

subset: commoncrawl


text:
For no discernable reason, at any given time I could be presented with any of:
a review link, with badges indicating the size of the flag queue and the size of the review queue:
the same review link, but without a badge for the flag queue (even though it's certainly non-empty):
a tools link in place of the review one:
other variations on this theme that I can't recall/reproduce right now.
For the avoidance of doubt, all three of the above were taken within moments of eachother all whilst visiting the homepage.
What's going on? Bug? Surely this erratic behaviour is not by-design?


text:
ChowChow.org - Privacy policy
This policy explains in detail how "ChowChow.org" along with its affiliated companies (hereinafter "we", "us", "our", "ChowChow.org", "http://forum.chowchow.org") and phpBB (hereinafter "they", "them", "their", "phpBB software", "www.phpbb.com", "phpBB Limited", "phpBB Teams") use any information collected during any session of usage by you (hereinafter "your information").
Your information is collected via two ways. Firstly, by browsing "ChowChow.org" will cause the phpBB software to create a number of cookies, which are small text files that are downloaded on to your computer's web browser temporary files. The first two cookies just contain a user identifier (hereinafter "user-id") and an anonymous session identifier (hereinafter "session-id"), automatically assigned to you by the phpBB software. A third cookie will be created once you have browsed topics within "ChowChow.org" and is used to store which topics have been read, thereby improving your user experience.
We may also create cookies external to the phpBB software whilst browsing "ChowChow.org", though these are outside the scope of this document which is intended to only cover the pages created by the phpBB software. The second way in which we collect your information is by what you submit to us. This can be, and is not limited to: posting as an anonymous user (hereinafter "anonymous posts"), registering on "ChowChow.org" (hereinafter "your account") and posts submitted by you after registration and whilst logged in (hereinafter "your posts").
Your account will at a bare minimum contain a uniquely identifiable name (hereinafter "your user name"), a personal password used for logging into your account (hereinafter "your password") and a personal, valid email address (hereinafter "your email"). Your information for your account at "ChowChow.org" is protected by data-protection laws applicable in the country that hosts us. Any information beyond your user name, your password, and your email address required by "ChowChow.org" during the registration process is either mandatory or optional, at the discretion of "ChowChow.org". In all cases, you have the option of what information in your account is publicly displayed. Furthermore, within your account, you have the option to opt-in or opt-out of automatically generated emails from the phpBB software.
Your password is ciphered (a one-way hash) so that it is secure. However, it is recommended that you do not reuse the same password across a number of different websites. Your password is the means of accessing your account at "ChowChow.org", so please guard it carefully and under no circumstance will anyone affiliated with "ChowChow.org", phpBB or another 3rd party, legitimately ask you for your password. Should you forget your password for your account, you can use the "I forgot my password" feature provided by the phpBB software. This process will ask you to submit your user name and your email, then the phpBB software will generate a new password to reclaim your account.
There are hundreds of posts each day on the "ChowChow.org" forums, so we can't always promise you'll receive a warning every time action has to be taken. By registering at the "ChowChow.org" forums, you agree that "ChowChow.org" and its moderators have the right to remove, edit, move or close any topic and/or post at any time should we see fit. "ChowChow.org" also has the right, at its sole discretion, to revoke board privileges at any time without notice or reason. As a user you agree to any information you have entered to being stored in a database. While this information will not be disclosed to any third party without your consent, neither "ChowChow.org" nor phpBB shall be held responsible for any hacking attempt that may lead to the data being compromised. Think of these rules as a forewarning and, if you break them, you're taking your account status into your own hands.
"ChowChow.org" has the right but not the obligation to monitor and/or moderate the "ChowChow.org" forums. "ChowChow.org" reserves the right to change any or all other parts of this terms of use agreement at its sole discretion and without notice, including but not limited to implementing terms of use and/or other measures.
Also note that the Terms of Use extend to all areas of the "ChowChow.org" forums system, including but not limited to Private Messages, Signatures, and Unban Requests. This also includes asking for or offering anything against the Terms of Use via Private Messages.
"ChowChow.org" is not responsible for messages posted on the "ChowChow.org" forums or the content therein. Unless expressly stated otherwise, this includes messages posted by "ChowChow.org" personnel, agents, representatives et al. "ChowChow.org" reserves the right to reveal the identity of or whatever information we know about any user in the event of a complaint or legal action arising from any message posted by said user.


text:
In Arts & Social Science forum, you can discuss about career options, courses, colleges, institutes, entrance exams and results.
Find out various types of courses in the field of arts, available in India. You may share your thoughts about the eligibility, admission procedure and the launch of new arts courses by different universities too, in this sub forum.
In this sub-forum, you may post threads and reply to discussions on the career options and prospects after completing courses in the field of arts and social sciences.
Here, you may discuss about entrance exams conducted for arts courses. You may post threads and reply to the same, regarding the eligibility, exam pattern, syllabus, admission procedure etc. of various entrance tests conducted for humanities courses.
This forum on law is aimed at facilitating discussions on career options, higher studies, entrance exams, syllabus, course structure, exam schedules, results and other such topics related to law.
Discuss on various career options and prospects after completing law and legal studies.
Find about various types of law courses available in India and abroad. Also share the job prospects and career opportunities after finishing these courses.
Explore various entrance exams for Law/LLB/LLM. Also share your experiences on preparation for law entrance tests.
Discuss about colleges offering LLB/LLM and other courses in law in India. This may also cover facilities and infrastructure available at these schools/colleges.


text:
San Jose Blouse
- Wool blend
- Dry clean only
- Full front button placket
- Scalloped lace sleeves
- Revolve Style No. SCOL-WS41
- Manufacturer Style No. SCFBAN04
- Modeled in size 1.
- Model Measurements: Height 5"11", Waist 24", Bust 34", Hips 35"
- Size Guide
Stone Cold Fox designers Cydney Morris and Dallas Wand reinterpret vintage inspiration, transforming soft and delicate silhouettes into something casual and chic for the coastal girl. By creating handmade details, pieces in the collection are unique yet timeless and fabrics take on an original feel.
If an item you would like to purchase is out of stock, you can place a Special Order Inquiry. REVOLVE will then check the availability and obtain an approximate arrival date for the item, if available. Typically, it takes up to 3-4 weeks to verify this information.
Placing a Special Order Inquiry is absolutely free and there is no obligation to follow through with a purchase.
- You will be notified by email once we receive the item.
- You may cancel at any time without penalty
- Your card will ONLY be charged when your order is shipped.
- Availability is not guaranteed.


text:
Hope College is one of only 11 colleges and universities nationwide, and the only institution from Michigan, to receive a 2012 Beckman Scholars Program award in support of student research.
The Beckman Scholars Program is an invited program for accredited universities and four-year colleges in the United States. It provides scholarship support to select students at the recipient institutions in chemistry, biochemistry, and the biological and medical sciences with an emphasis on sustained, in-depth laboratory research experiences with faculty mentors.
The $77,200 award to Hope will support a total of four students across the next three years as they conduct research in biology, biochemistry/molecular biology, chemistry or bio-engineering. Selected as sophomores or juniors, the students will each receive a total of $19,300 in support to conduct research with faculty members full-time during two summers and part-time during the intervening school year. They will also be expected to major in one of the supported disciplines and to stay involved in research at Hope as seniors.
In addition to providing stipends for the students, the award also includes funding for related supplies and travel. The foundation hosts an annual research symposium for the program's student scholars and their faculty mentors each summer.
The Beckman Scholars are among scores of students who conduct research campus-wide at Hope during both the summer and the school year. Each summer, more than 160 students engage in research full-time with Hope faculty mentors.
The Arnold and Mabel Beckman Foundation was established in 1977. The foundation makes grants to program-related, non-profit research institutions to promote research in chemistry and the life sciences, broadly interpreted, and particularly to foster the invention of methods, instruments, and materials that will open up new avenues of research in science.
Hope also received "Beckman Scholar Awards" from the foundation in 1998, which was the year that the program began, 2000, 2002, 2005 and 2007. This year's proposal from Hope was written and submitted by Dr. Maria Burnatowska-Hledin, who is the Frederich Garrett and Helen Floor Dekker Professor of Biology and Chemistry.
In addition to Hope, the colleges and universities to receive Beckman Scholars Program awards for 2012 are: Allegheny College; Bucknell University; Furman University; San Francisco State University; the University of Alabama, Birmingham; the University of California, Davis; the University of California, Los Angeles; the University of Chicago; the University of Oregon; and Vanderbilt University.


text:
INDIANAPOLIS --- The Lees-McRae College women's soccer team is ranked among the nation's top 10 teams in a pair of categories, and is listed in a total of 11 individual and team categories, announced NCAA officials Monday with the release of the NCAA Division II women's soccer statistics.
Lees-McRae (10-2-1, 9-0 Conference Carolinas) has earned a result from each of its last 11 matches, posting a 10-0-1 record in that span to open conference play with an unblemished 9-0 record. The Bobcats enter the week ranked sixth in the nation in scoring offense (3.0) and eighth in shutout percentage (0.692), while also being listed 24th in winning percentage (0.808) and 30th in goals against average (0.681).
Sophomore MaryBeth Sullivan (Greensboro, N.C.) enters the week ranked 16th in the nation in match-winning goals (four) after delivering a pair of decisive strikes against Belmont Abbey College and Mount Olive College to give the Bobcats a two-game lead in the conference standings with three contests left on the schedule. The 5-7 forward is also ranked 23rd in total goals (10), 27th in goals per match (0.769), and 44th in points per contest (1.69).
Freshman goalkeeper Kaitlyn Briggs (Oviedo, Fla.) has been tremendous between the pipes for the Bobcats this season, entering the week ranked 31st in goals against average (0.655) and 79th in save percentage (0.800) after recording her seventh and eighth clean sheets of the campaign last week. Senior Vickie Rich (Arlington Heights, Ill.) is also ranked 41st in the nation in assists per match (0.46), while freshman Meagan Thornton (Millville, N.J.) is listed 50th in goals per match (0.692) and 68th in points per contest (1.54).
Lees-McRae returns to the pitch on Wednesday when the Bobcats host Erskine College in a conference matchup at Tate Field. The match is slated to kick off at 5 p.m. following the Bobcats' senior night festivities.


text:
Rather suddenly, graphics in my email messages I receive do not display.
I had been running Barca 2.1, so I downloaded and installed 2.5, which solved a few other vexing problems, but not this one.
Can anyone offer suggestions, please?
And yes, I do click the little "download images for this message only" button.


text:
This is an off-the-wall question, but does Gracenote have guide data available for various IPTV sources that can be made available to Schedules Direct subscribers? I believe that DirecTV includes a NasaTV channel for example (or at least did), which means that the data for that channel may be available. And it would be desirable to be able to get that guide data "somehow" into a lineup that could be downloaded and associated with a IPTV channel in your application of choice. There are numerous international freely available IPTV channels "out there". I would desire that there be guide data available for them as part of my subscription. Perhaps a new region (IPTV?) for lineups and having a m3u8 channel definitions would be an additional plus.
I do understand that this is a likely a very very long term question/issue, and nothing is likely to happen quickly. I just figured I would throw it out there. Thanks.


text:
Karina Brite
Karina Brite's Details
Karina Brite's Likes
I like being myself c2c, virgin guys , like make the Teamweiver games, slaves, bigays , phone in private, role play , panthose play, feet fetish, )))Ultimate Money Mistress always looking for new money slaves /// /T& / sissyfication / Punishment, Spanking, joi / cei, sph, t
My Fantasies
You are toy in my hands, that I will use to my own enjoyment. you are a pathetic little worm to Me, you are on your knees in front of me. you don't deserve my focus, or to lay my eyes on you! and i will humiliate you like you've never had been before/
My Obsessions & Desires
worship, panthose (Wolford ) fetish,feetjob,Cuckolding & sissy , legs fetish, teamviewer, cbt, cei, joi, sph, punishm , strapon, role play,Trampling, Balloons,Chastity Training,Cross Dressing,Tickling
Additional Info
Im Intelligent woman with devilish eyes and wonferful imagination. I feel your desires. I read your mind. I make you feel weak at my beauty. Lets come our fantasies true. I like to be treated like queen. I love to feel you and make us enjoy each other. Im opened mind and would like to try new things with you. Join me in private room now!
Special Info
I have some toys to play with. Stockings, heels, dildos, skirts, oil ,vaginal balls, strapon
My Fetishes
CBT, Cuckolding, Domination, Feet, Findom, Humiliation, Jerk off Instruction, Legs, Sissy Training, Stockings/Panty Hose
My Sex Toys
Anal Beads, Giant Dildo, Strap-on, Vibrator, Whip
Role-Playing Scenarios
Boss / Secretary, Cowboy/Cowgirl, Dirty Talk, Doctor / Nurse, Master / Slave, Service Worker/Housewife, Student/Teacher
Top Gifters
Karina Brite's Awards
My Recent ReviewsMore Reviews Post Review
Recent Blog EntriesMore
Mistress and submissive slave (part 2)His lips were so deftly fingering pussy lips, as in the first time, his language is as feverishly fingered clit girl, but it was with a new... Read More »
Mistress and submissive slave (part 2)It was painful! His cock and his mind so wanted to stay a hot slits near the Lady, just a little, just a little, but it was impossible. He had to... Read More »
Mistress and submissive slave (part 1)He longed to get a tongue in this small hole in a magnificent ass, ass small but firm, that is so sweet girl stuck out slightly back. Rising to his... Read More »
Lick my feet slave !!!!!! (part 3)I continue to redouble their efforts and finally jumps off stocking heels. It only remains, gently hugging sock feet, pulled a stocking full. But... Read More »
Lick my feet slave !!!!!! (part 2)I feel at home in the language of soft nylon stockings and start to gently suck and tongue stroking fingers Lady. I do it with great love and care,... Read More »
My Schedule
Please change the timezone to your locale by choosing your location from the drop-down menu.
Largest Tip
Virtual Gifts Received More
|Gift Info
|April 20, 2019
10 Credit Tip
Purchased for 10 credits.
|View Details
|April 18, 2019
Custom Tip (Rain)
Purchased for 250 credits.
|View Details
|April 18, 2019
Custom Tip (Rain)
Purchased for 250 credits.
|View Details
|April 18, 2019
10 Credit Tip
Purchased for 10 credits.
|View Details
|April 18, 2019
10 Credit Tip
Purchased for 10 credits.
|View Details


text:
This star rating is the opinion of the property owner only. A Self-Rated hotel has not been assessed by the AA or Visit Britain, whose official star ratings we use across LateRooms.com.
Seraphine Kensington Olympia4 Self rated
Price FromView Rooms
- Very Good
- 61 guest reviews
"We enjoyed our stay at this hotel. Room was small but clean and comfortable. Breakfast was basic but edible. A staff member was being traine..."
"Hotel was ideal when visiting London Olympia. Quiet, clean no problems at all whilst staying...."
"5 star hotel, 5 star service. Thank you, definitely I will be back...."
"The Staff were very helpful > myself and my girl friend enjoyed our stay...."
"Stayed here for 2 nights from 25/03/15, arrived late on, very good reception & check in service with friendly & helpful staff. The hotel is..."
About the hotel
Ideally located on High Street Kensington, just a few minutes' walk from the High Street Kensington Tube Station. This modern, intimate chic hotel overlooks Holland Park and is in the vicinity of Kensington Palace and Gardens and is the perfect choice for either a pleasurable few days holiday or short business trip. In the surrounding area of the hotel one can find stylish shops, bars and restaurants.
Art galleries, the Natural History and Science Museums, Hyde Park, Buckingham Palace and The Royal Albert Hall are just a few of the many attractions which are located within walking distance of the hotel. The hotel is very close to Olympia and Earl's Court Exhibition Centres. All our rooms, which are all individually designed, offer the ultimate facilities that one would expect from a hotel of this quality. All rooms have complimentary High Speed Internet access and WIFI, laptop size safe deposit box, full ensuite facilities with bath or shower, IPOD docking system, TV Entertainment system, Tea & Coffee Making facilities, air conditioning, telephone, triple glazed windows, desk, electronic door locking, opening windows and hair dryer.
Read MoreLess
Facilities Show/hide
- Main
- Hotel Non-Smoking Throughout, Air conditioning in rooms, 24 Hour Reception, Safety Deposit Box, Tea/Coffee making facilities in each room, Hairdryer in each room, Iron And Board In Each Room, Telephone In Each Room, Laundry/Valet Service, Interconnecting rooms available, Doctor on call, Lift to all floors, Public Air Conditioning
- Business
- Wi-Fi Available, In-room Broadband, In-room Modem Hook up, Wi-Fi Available In Public Areas, Wi-Fi Available In Rooms, Business Centre
- In Room Entertainment
- TV, Satellite TV, Radio
- Parking
- Car Parking Available Offsite
- Dining
- Bar
- Leisure
- [Channels] News
Useful Info Show/hide
- Between 14:00 and 23:00
- By 12:00 hours
- Up to 12 years old
- The price shown is based on the occupancy and dates you have searched for. Prices and availability are subject to change based on the arrival date, number of nights and occupancy searched for. All prices are inclusive of all local taxes.
- Pets are not allowed
Customer Reviews
Our reviews are from genuine LateRooms.com customers. We verify that the customer has stayed before leaving a review.
Very Good
80% would recommend it to a friend
Customer service
Food quality
Room quality


text:
In fact, despite the apparent bad press in some quarters and abject dismissal of this technique by some, the breath can be used as the primary object instead of the abdomen, albeit something not actively taught in this day and age.
If something is a concept and not reality - as Yuttadhammo asserts that the breath is -, the thinking that goes with that is that one cannot see the 3 characteristics of anicca, dukkha and anatta. And that means that anyone practising mindfulness on the breath cannot see the 3 characteristics and therefore experience insight knowledge. It therefore follows that one cannot achieve any of supramundane states (magga and phala) by practising mindfulness on the breath.
From my personal experience, I can tell you that the breath is rooted in reality and is not a concept; one can see the 3 characteristics when the suitable conditions arise.
I have enjoyed and rejoiced in the ongoing work by Bhikkhu Yuttadhammo in disseminating the Dhamma in the new media formats (youtube, facebook).
But now, seeing such a statement from him for those intent on meditation is not good at all.
reference: http://yuttadhammo.sirimangalo.org/arti ... questions/ (emphasis by me as seen below)
Why do we focus on the abdomen?
The abdomen is an obvious example of a physical phenomenon; it is the element of motion (vayo dhatu) as described in Buddhist texts. Whereas "mindfulness of breathing" (ānāpānasati) is technically considered tranquillity (samatha) meditation, "analysis of the elements" (catudhātuvavatthāna) is considered to be the basis of insight meditation. This is because, while the elements are experienced directly, the breath itself is a concept. For this reason, though watching the abdomen can be thought of as mindfulness of breathing in a way, it differs from watching in and out breathing in an important regard: the latter often leads to tranquillity rather than directly to insight into the nature of reality, while the former is set firmly in ultimate reality and thus conducive to insight into the nature of reality.


text:
For every student who walks in my office, I'm thinking about helping them in some way or another.
Students stopping by the dean's office are often only looking for a signature to change majors or withdraw from a course. Little do they know, Associate Dean of Arts and Sciences Lynn Breyfogle has other ideas about those quick visits.
"For every student who walks in my office, I'm thinking about helping them in some way or another," Breyfogle says. Whether she provides academic advice or a connection to new resources, Breyfogle wants to make sure everybody at Bucknell has every opportunity to succeed.
"I'm hoping to reach out especially to those students who are in typically underrepresented groups," she says. "I hope to be a support and an advocate for students of color as well as students from often marginalized groups, low socioeconomic status students and students who are first generation, who often do not have the knowledge of how a university works."
For instance, selecting classes can be overwhelming for new college students. In 2013, Breyfogle and her colleagues introduced the Summer Academic Counselor (SAC) program to connect with new students before they arrive on campus. Faculty members contacted every new student in the College of Arts and Sciences to talk about majors, course scheduling and other questions. When possible, professors conducted the calls by video chat to make the connection that much more personal. Breyfogle hopes the calls will help students not only settle registration questions early, but also assuage any uneasiness they have about going to college.
Once students arrive on campus, social connections with peers are at least as critical as connections with the faculty. To help students meet others who share their academic interests, Breyfogle is working to bolster the prominence of two existing programs.
Each first year class at Bucknell has about 100 Dean's Scholars, selected based on their academic records in high school. Breyfogle plans to host events, such as a reception for the scholars during orientation, to help them meet upper classmen and faculty as soon as they get to campus.
Another distinguished group on campus is Alpha Lambda Delta, the national first-year honor society. The organization already recognizes the academic accomplishments of its members; Breyfogle is working with the chapter to develop its potential as a social outlet for members and a resource for other students.
As Breyfogle has directed her career from high school mathematics teacher to the head of Bucknell's math teacher education program to the dean's office, she has maintained one focus. "I'm still helping students," she says. "That hasn't changed."
Posted September 26, 2013


text:
Actress Laura Dern has finalized her divorce from her husband, singer Ben Harper.
The "Jurassic Park" star split from Harper, her husband of five years and father of her two children, in 2010 and they attempted a reconciliation last year before separating again.
Dern's marriage is now officially over after a judge signed off on their divorce, according to TMZ.com.
Terms of the settlement were not made public, but the former couple will reportedly share custody of Ellery, 12, and Jaya, 8.


text:
Happy New Year first of all!
I mostly lurk on here without posting, but suddenly this seemed the natural place to turn, as there's always some pretty deep musical knowledge on display on these boards. Couldn't see another thread dedicated to the topic, so here we go.
Having listened to roots and dub reggae for years I've decided it's time to start checking the music that's come out of Jamaica since 1980, which mostly means dancehall. I'm way out of my depth though, so I'm seeking any suggestions as to where to start.
I was recommended (and am enjoying) this Terror Danjah mix which has a lot of early and mid-90s stuff on it. Also I picked up the Sept/Oct issue of Wax Poetics which has interesting interviews with Beres Hammond, Sugar Minott, and Gregory Isaacs in it. I'm familiar with the latter's earlier (non-dancehall) work, but the rest is a black hole for me.
Those interviews have thrown up a few other artists too, but I think they're pretty limited to the period when dancehall was evolving out of earlier stuff. I'm interested in that for sure (I've been thinking about checking out one of Trojan's box sets covering that era), but that still leaves the whole of the last 20-odd years untouched. A mate has passed a copy of Sizzla's Black Woman And Child on to me, which is a starting point, I guess. Only picked it up today so I haven't had a chance to listen to it yet.
Also, I have to admit that danchall's reputation precedes it somewhat. I'm happy to give anything a listen from an academic perspective, but any records that are over-the-top with misogyny and homophobia aren't likely to get much rotation for pleasure in my house. So more righteous tips would be appreciated Although if anyone has anything insightful to add about slackness in Jamaican music, that's definitely a conversation I'm very interested in having, as it's not something I've really had opportunity to discuss before.


text:
Warning: the following post is full of broad, sweeping generalizations that are only true of me, and maybe not even then.
Imagine this as one of those "tell-all" articles featured prominently on the cover of Cosmopolitan, "Things Your Healer Won't Tell You", or "The Things You Think You Know (but don't)". Actually, don't imagine it that way. I've never actually read one of those articles and they might be really lame.
I've been reading some blogs about what healing is going to be like in Cataclysm, posts about what Blizzard wants it to be like, and that has me wondering; are we healers all just plain nuts? There are things I do, things I think, and I'm not sure how they'll match with our new healing methods.
- When you die, I feel guilty – no ifs, ands, or buts.
There's a reason why we joke about "Oh, no heals for you" – because we can't do it. Once you're a healer, in that mode, it's almost impossible to let someone's health bar drop low without trying to heal him. Often, we don't really recognize any health bar as a particular person. We know which ones represent the tanks, we have things organized so we sort of know who is where – but when I'm in the thick of things, I don't ever think "Oh yeah I was going to let xxArthaaasxxx die, he's a jerk".
In Cataclysm, we're going to have to let people sit at less-than-max health and that is going to be really really hard. All my instincts say "Heal that bar!"
- Healing is my thing
The same pleasure a rogue in super-epic-ilvl277 gear gets in topping a dps meter, I get in ending a fight with everyone alive. I like healing, I find it fun and challenging and the only job in the game that reaches out and grabs me by the throat. I've dpsed and tanked 11/12 fights in ICC – meh. I just don't care to do that regularly. But I'll heal ICC over and over and over again, because when I'm healing, every fight is different every time. You are fighting a boss and his minions: I am fighting death itself. Your scorecard is the amount of damage you put out; mine is the number of people I saved. If we wipe, you get upset with other people for screwing up; I get mad at myself for not saving them anyway.
- I find healing fun even as it is
But I do see their points – because I find healing fun in 10 mans and not so much in 25s. Spamming Rejuv on groups 3-5 is not really super fun. Spamming rejuv on the whole raid, looking where to place Efflorescence, rolling Lifebloom on a tank, checking to see if I need to spend a whole two seconds casting a single Nourish – that is fun, and I get to do that in ten mans. I'm worried that they will be crippling me by making me worry about mana. I'm going to give it a try, and learn how to roll with the punches. I mean, I was concerned that I wouldn't like druid healing after the changes and now, other than missing my tree, I'm ok with it.
- I don't really want to dps
I'll qualify this. When I outgear the content, sure, dropping Hurricane is fun. But I want encounters to be hard and expect me to focus on healing, not hard and expect me to focus on both healing and dps. Maybe some people like this, but please, Blizzard, give us a way to avoid it. I have an off spec, I use my off spec, let me dps in that.
- My toolset is awesome! All except for the wonky egg seperator…
Blizzard has given all the healing classes a range of tools to use. Some classes have a billion, like Holy Priests. Druids have about 9 and that's more manageable. Still, I eye them a little suspiciously. It's like my utensil drawer. There's my trusty spatula and a good knife, and then a whisk, and I guess I use that peeler sometimes if I'm using my knife for something else, but what is that weird plastic doodad anyway? And why does anyone need an asparagus trimmer?
My heals are like that. I love how Mastery works with druid hots. I love, love the way Efflorescence triggers off Swiftmend – it just feels right, like you're blooming the heal and causing life to grow. But we have these nice fast or instant cast hots – and then we have long, comparatively slow casts of Nourish and Healing Touch and they just don't feel right. I feel like one of them should be shorter, and one longer. Then we could really pick and choose what to use. As it is, I still am not using HT; if I have to cast anything, it's Nourish because I almost always have a hot on the target already (and thus, get my mastery bonus plus the backed in bonus to Nourish). Healing Touch is that egg separator – it's just taking up room in my drawer, I feel a bit embarrassed about it, and maybe one day I'll use it but I won't be bragging about it to my friends.
- We're gossipy little backbiters
If your raid has more than two healers, I guarantee you at least two of them whisper each other multiple times during the raid. It's like a rule. "That tank had four stacks of X"". "Healer Y is using THAT spell, can you believe it?" "Why won't the mage decurse? Ok we'll work around that". I'm getting worse, too. I used to just gripe at my husband; now it's whispers. Why don't we say these things out loud? Well, sometimes it's a healer-related FYI kinda thing. Sometimes it's because we don't want to sound accusatory. Also I think healers have a bit of an inferiority complex and we're afraid that if we say "X did Y" that what the raid hears is "I'm a lousy healer and I can't deal with this".
(Someone please tell me this is really universal. It has to be, right? I'm not always the first one to start whispering…)
- I've played games that are like spreadsheets. Healing isn't like playing a spreadsheet
Yes, there are bars on your screen and you spend a lot of time looking at them. No, this does not make WoW into "Excel with sound effects". If you want that, go play EVE Online. If you think we have that, go play EVE Online. Now there is a game that has spreadsheets. Yeah, so sometimes I miss cool effects or visuals. Tough. I like my little squares of healing. They are the source of my power.
- I don't want to worry about mana
But I will. I get that. I'll learn. I just don't want to – I'm terrified that every wipe in Cataclysm will be twice my fault – once because I let people die, once because I ran out of mana. That if I were a better healer my mana would stay full even when the dps is not doing their job. I want more enrage timers on bosses. Make it obvious why we failed. Don't blame it on me! There is nothing worse than staring at people dying, and your empty mana bar.
- I'm still going to heal
Even if I do have to use my egg separator, worry about mana, deal with guilt complexes, whatever. Because that's what I love in this game.
Bring it on.


text:
Former Bison Joseph named Eastern League All-Star
PORTLAND, Maine – Former Lipscomb standout catcher Caleb Joseph has been named to the 2013 Eastern League All-Star Game roster for the Western Division team.
The regular backstop for the Bowie Baysox, the AA affiliate of the Baltimore Orioles, Joseph has garnered all-star honors for the second time in his career after earning the laurel in 2011 for Bowie. The Nashville native is hitting .293 with 55 RBIs, 14 home runs and 20 doubles in 69 games for the Baysox.
Joseph was a two-time, first-team All-Atlantic Sun Conference selection for the Bisons in 2007 and 2008 and the only ABCA/Rawlings All-South Region team selection in school history.
The Eastern League All-Star Game is slated to take place on July 10 in New Britain, Conn.


text:
Wednesday, June 2, 2010
How I sped up my server by a factor of 6
Subtitle: IO-bound and CPU-bound applications are common - here's maybe a "memory contention bound" app
I've written a lot of servers. If you've read the architecture of mailinator or benchmarking talkinator or "blocking faster than non-blocking" you probably already have that idea.
I was working on a new server infrastructure recently I needed for a new product. The server has a novel internal architecture to me that seeks to never voluntarily block threads or cause context switches - but at the same time is highly multi-threaded and creates new ones whenever it needs to.
The server's nature isn't totally important but you can think of it along the lines of a twitter server. It gets messages from one place, collates them, and sends them to (potentially many) other places. I suppose it could be used directly as a "twitter server" but of course, they already have those and my start up focus is rather orthogonal to that.
I usually write servers in Java on linux so keep that in mind for the ideas raised here. At Google, I worked on servers in C++ and Java. And especially after that experience, I'll stay with Java. To hopefully avoid language wars, the #1 reason I write in Java is that I'm way better at Java than I am at C++ or Ruby or Python or whatever. In other words, it was a practical (and definitely reasonable) choice in this case.
Also, the benchmarking here isn't contrived - I'll save the details, but I truly am asking the server to do precisely what it would do in the wild except at a higher rate.
Now, given the nature of the server, I needed 2 things to help benchmark. A "producer" (a system that produces messages) and a "consumer" (a system that consumes them). The producer produces a sequenced message set, and the consumer verifies it receives every message intact *and* in-order.
I chose message sizes at somewhat less than twitter size (msg size = 100 bytes) to induce a reasonable amount of activity in the server (server purpose is not necessarily large messages and large messages tend to just start saturating bandwidth without causing contention or server cpu activity). My custom protocols add a few bytes and TCP has a 40byte header - so overall I'm guessing that I'm using an average 200 byte messages counting everything.
Producer -> Server -> Consumer
The server system is pretty flexible and in effect, location transient. That is, a server process might live on one machine today and find itself on another machine tomorrow. Or a few new server processes may join the mesh. Needless to say, the idea is to create a scalable, flexible system.
One ramification of that is that it's possible and even very likely that some producers, servers, and consumers could be on the exact same machine. Socket communication over localhost changes a lot of assumptions. Firstly, TCP fusion can occur to reduce overhead significantly. Secondly, there is not an effective bandwidth limitation - there isn't a real network involved, it's a virtual (and fast) one.
Running all on the same server, I would expect this benchmark to be CPU-bound given that I/O is now virtual and effectively a CPU operation. Like I said, loopback is a real scenario I need but I initially think I benchmarked on a single machine mostly as a matter of convenience.
So, with all 3 processes running on the same machine, I ran the test. The CPU was a Intel Core I7 920. It has 4 hyperthreaded cores that act like 8. (note: I've tested all of this on a Core2Quad (non-I7) cpu and CoreDuos and results are effectively the same)
Here's the image of htop during the test with all 3 processes running. (if you run linux and use top, upgrading to htop is highly recommended).
Don't get hung up on reading the numbers or text. The graph in the upper left gives you a sense of how busy the CPUs are. In this picture, they're all at least a little busy. That's no surprise given we're running many tens of threads.
Notice that none of the cores are "maxed". This benchmarked showed the server to receive and re-send about 120,000 messages per second (that's 120k from producer to server and the same 120k from server to consumer - so 240k "messages transmissions" but only 120k messages - this would be analogous to queries-per-second for a webserver).
Why aren't the active cores maxed?
It occurred to me that I was running 3 CPU-bound processes on the same machine and that the processes might be stepping on each other's toes. It's possible that if the server is running on core 4 one second and the producer is running there the next, the level 1 cache of that core could be ruined for the server the next time it ambles over there.
The simple solution was to assign cores to the processes. In linux, you do this with command: taskset
Taskset is followed by a bitmask value to assign CPUs - so I ran (in separate xterms):
taskset 0x3F java manyfeedServer
taskset 0x40 java theProducer
taskset 0x80 java theConsumer
and off they went. The server is highly multithreaded so I gave it 6 cores. The producer and consumer each got one.
The result? 270,000 messages/second! Wow. If my cache assumption was right (and I'm not claiming to know if it was) - it REALLY worked. One way or another though - something worked - we got better than a 2x speedup.
So you might be thinking the moral of this story is:
1) If you're in linux, install HTOP
2) If you're sharing a computer amongst CPU-bound processes, isolating the processes might (and very well "might not") be beneficial.
And ok, those are fine ideas. But what bugged me was that htop showed me that no CPUs were maxed yet. Again, what was slowing my application down ahead of CPU power?
I then tried limiting the server to 2 (hyperthreaded) cores. (I also tried keeping the producer and consumer on the same hyperthreaded "core" and given that I had cores to spare, also tried separating them, but the result was the same).
taskset 0x03 java manyfeedServer
Now, we get 530,000 messages/second. Nice. Reducing the cores from 6 to 2 nearly doubles our msgs/sec again. Here's the htop now:
You can see that cores 1 & 2 are plenty busy. Cores 3,4,5,6 are idle as expected. Core 7 (the producer) is pretty busy and so is Core 8 (the consumer).
530k msgs/second is nothing to sneeze at but.. um.. again, no core is maxed. Why - not? What's the bottleneck?
Obviously.. the last test is to throw the whole server on ONE cpu. Apart from the fact that I very purposely and meticulously coded this server be highly multi-threaded, fewer CPUs seem to make it happier. I am a conservative thread-safety-inducer. That is, I'm only really dangerous with firearms and synchronized blocks. But I'm by no means afraid to use the latter when needed.
taskset 0x01 java manyfeedServer
And finally, we hit 100% utilization on CPU 1 at 790,000 messages per second.
Here's the TL;DR of this blog post:
Some multi-threaded java applications apparently run faster on 1 core than on multiple cores.
Note the very non-committal phrasing in an attempt to make this a rather defensible statement. A graph showing the ManyFeed server's performance per number of cores:
So.. if you are a CPU or JVM guru and want to tell me your thoughts on what's going on, I'd love to hear it.
A long time ago, I benchmarked a bunch of different CPU types on doing contended memory stores. To my surprise, a little single-core Dothan processor did amazing in a few tests and I had no idea why. The discussion above is really the same idea. (I've retested that benchmark by the way on the Core I7 - and it has very different characteristics - probably a testament to the I7's new memory architecture).
Note that if you have a pre-I7 Intel multi-core cpu, you can reproduce my "1 core runs faster" results here using the code in that blog post (note: this code *is* a micro-benchmark, built to create a situation that causes extreme and unrealistic threading competition - for more details read that blog post). This gives a clearer picture of what's going on. My theory is that with 2 cores and 2 threads - each gets a core and every store operation competes for the memory bus in order to do its operation. Threads spend lots of time "competing" and less time doing real work. On one core (and 2 threads) - only one thread runs at a time - so every time it tries to store in memory, it can.
The Core-I7 is different (and maybe AMD cpus too) - it doesn't suffer from this memory contention problem at all. (Although - given that my server still runs (way way) better on 1 cpu even on I7, then maybe the contention is elsewhere in my server).
Here's that benchmark running for Static Volatile Store on a Core Duo CPU with 2 cores and again with 1 core. No performance loss on just one core, otherwise same exact run.
(numbers along the bottom are the number of threads in that run - it's not until we have 2 threads that contention creeps in and hurts us).
Oh.. and how does my little nifty manyfeed server do on a real network? Testing on 1 CPU on my local 1Gbps LAN I get about 300,000 messages per second.
With some quick (and hopefully not silly) math, this looks about right.
1Gbps = 1,000,000,000 bits/sec = 125,000,000 bytes/sec
125,000,000 bytes/sec divided by 200 bytes/message = 625,000 messages.
half for producer send and half for the server send (to consumer) = 312,500 each
On a real LAN, the server saturates bandwidth (i.e. becomes IO-bound) and that's no surprise (i.e. until I have access to 10Gbps networks, I don't need to speed up my server any more).
Interestingly I tested this idea of "1-core'ing-it" using apache tomcat and apache bench and saw no improvement. I also saw far fewer qps (loading a single, tiny, web-page over and over) and 100% core utilization even on multi-core. It'd be my guess that tomcat isn't contention-bound but truly cpu-bound.
In other words, don't follow this path without testing this yourself. The good news is that its extremely easy to try and you don't even have to change a line of code. Just try "1-core'ing" it.
55 comments:
- sep332 said...
Arent' pairs of "logical" CPU's actually mapped to a single core? So running producer & consumer on 7 & 8 means they're running on the same physical core, obviously competing for resources and you'll never get 100% utilization on both at once. Same with running the multithreaded server on cores 1 & 2. Try using only odd-numbered or only even-numbered cores. E.g. server on 1 & 3, producer on 5, consumer on 7.
- June 2, 2010 at 9:48 AM
- Chas Emerick said...
There's not a lot of info there to really know what's going on. Lock contention is a likely suspect, perhaps driven by GC. If you're really CPU-bound, you can peg 8 cores in a blink if you're building on stuff that has sane concurrency/parallelism semantics (e.g. use Clojure agents, or pmap, or similar).
If you're doing manual locking, you're just a little crazy these days given the better alternatives, IMO. Beyond that, peeking into what the GC is doing is worth a shot.
- June 2, 2010 at 10:39 AM
- Tom said...
Interesting article! At the very least you got me using htop :)
- June 2, 2010 at 10:47 AM
Maybe you should check out Erlang for what you are doing?
The telcos handle *millions* of concurrent processes with this language/vm.
- June 2, 2010 at 10:56 AM
- matt said...
Multithreading has the overhead of switching between threads, which is quite expensive, so of course it is going to be slower. However, I bet if you leave it running on a bunch of cores it will scale much better.
- June 2, 2010 at 11:01 AM
- matt said...
I also agree with using Erlang, even though I know you used Java because that's what you know :)
- June 2, 2010 at 11:02 AM
- mikeh said...
Hi! I'd be really interested in seeing what happens if you eliminate hyperthreading from the mix by turning it off in the BIOS. It complicates things somewhat.
- June 2, 2010 at 11:14 AM
- sep332 said...
Or you could just save yourself the confusion and just disable hyperthreading in the BIOS.
- June 2, 2010 at 11:20 AM
- xsdg said...
Sort of echoing sep332's and mikeh's sentiments here. From my (incomplete) understanding, "hyperthreading" basically allows you to run overlapping bits of code on the same core. This can be a win if one bit of code is waiting on IO, or if you can use different units on the same core concurrently (arithmetic op + FLOP). If the pieces of code are doing similar or the same things, though, you're basically asking for contention.
Also, is it possible the latencies are asymmetric? Perhaps just run the simple benchmark pairwise on virtual cores 1, 2, and 3? (3! == 6 runs, total)
- June 2, 2010 at 11:39 AM
- cpetit said...
Did you try to analyse and play with differents gc strategy ?
- June 2, 2010 at 11:44 AM
Actually Erlang is not very widely used by telcos. Ericsson uses other languages and tools.
One thing you should know if you program in Java is that unless you use the Solaris OS, Java threads are NOT native, i.e. they run on the same processor, not on different processors. That's one of the many shortcomings of the JVM.
Oh, and by the way, J2EE sucks huge donkey balls.
- June 2, 2010 at 11:47 AM
- Jan Rychter said...
On the i7 once you limit yourself to one core you additionally gain from the core becoming faster — i7 will boost its speed if the other cores are idle.
It won't account for the entire difference you're seeing between 2 cores and 1 core, but it is significant.
- June 2, 2010 at 12:06 PM
Anonymous, was the last time you used Java in 1997? Most JVMs have native threads and have for over a decade.
- June 2, 2010 at 12:15 PM
Erlang was apparently dropped by Ericsson for a while (1998-2004), but then picked up again.
Java has been native threaded on Linux since 1.3 (May 2000).
- June 2, 2010 at 12:20 PM
- Kristian Rosenvold said...
I have seen behaviour like this if your code basically locks all threads in hotspots. If you're starting a lot of threads and they basically just end up blocked/waiting you'll end up impeding total performance pretty quickly. Increasing performance with multithreading can be real hard.
You really *must* use a profiler to determine where you're blocking your threads. It's almost never where you think it is ;)
- June 2, 2010 at 1:43 PM
- Eric said...
The problem is clearly **NOT** memory contention. The OS isn't aware of time a thread spends waiting on physical memory, and so that time would end up counted as active CPU time, not idle time. (*virtual* memory if a hard page fault occurs is another story)
Best guess is that you have a lock contention issue. Even if your code isn't using locks explicitly, it's likely that the APIs you are using have locks under the covers, including the socket APIs for your server.
- June 2, 2010 at 1:48 PM
- PeterH said...
I'm thinking you may be facing overhead of data being transferred between the L1 caches of cores used by different threads. If the server was written so that the thread that received a message handled sending it out you might avoid that overhead.
- June 2, 2010 at 1:52 PM
- Parallax said...
I'd be interested to learn how your messaging server compares with http://www.rabbitmq.com/
- June 2, 2010 at 2:08 PM
- Paul Tyma said...
@sep332 - I mentioned above that I tested this on a non-hyperthreaded CoreQuad and the results were the same. But you're right, HT just clouded the issue - when I finally wrote the post it just happened to be the machine I was working on.
@chas - I have no need to optimize this, but you're right, peeking into the GC could yield some interesting info.
@ERLANGERs - As pointed out, nothing against Erlang. I have a project to finish and didn't want to be impeded by learning a new language.
@JanRychter - interesting. Did not know that.
@Eric - this is very insightful. I will be correcting the post (after a day of comment accumulation) to reflect your comment. Thanks.
@PeterH - as I said, I avoid unnecessary context switches. In the general case, I already do exactly what you suggest. In cases where there are high numbers of consumers, the primary thread will ask for help (but that did not occur in my testing above)
@Parallax - maybe I'll test that someday. Of course I looked at rabbitmq but I felt my needs were somewhat outside what RMQ accomplishes (then again, I may find myself wrong and wish I had used that :)
- June 2, 2010 at 2:23 PM
- sep332 said...
@Paul Ah, thanks. I totally misread that graph for some reason.
- June 2, 2010 at 6:46 PM
Your issue is almost certainly not memory contention per se. A stalled load/store to memory surfaces itself as excessive CPU utilization. Conceptually, it is an assembly instruction that takes many hundreds of cycles to complete and individual instructions are not preempt-able. Thus, you only see <100% CPU usage if the process/thread is not actually runnable and instead the core is spinning in the OS's idle loop.
Thus, your threads must be blocking at a point where they explicitly yield to the OS either when doing I/O, or when waiting on a lock/synchronized block. Depending on the number of threads, it's possible the contention lies in the OS's I/O subsystem, but I this seems unlikely. More likely is that when your app is running on many threads, there is contention for locks and threads are being put to sleep waiting for those locks. Then, possibly by fault of the OS, there is a lag time between when a lock is released by a thread, and when those waiting for it are returned to the OS run queue.
In short, I think you need to dig into what locks you're taking and conditions you're signaling in the course of processing each message. That your CPUs are spending time idling indicates lock contention over memory bandwidth contention.
- June 2, 2010 at 7:31 PM
Wrt. the hyperthreading, it would be interesting how a bitmask of 01010101 performs, i.e. hiding hyperthreading for these apps and only allowing 4 real instead of 8 fake cores.
- June 2, 2010 at 7:48 PM
- eyrieowl said...
Not the first person to suggest it, but it seems highly suggestive of context switching to me. This is a cost I think most people discount to easily. And it's not surprising to me that the fewer the cores, the higher the transaction rate, particularly if you have tens of threads. The way I see it, the more threads you have wanting to do work, the greater the likelihood that the next time your thread gets scheduled, it will end up on a 'random' core. However, if the number of cores is low, it improves the odds that it ends up being the same core it was running on before. Unless the rest of the threads are just hanging out twiddling their thumbs, they'll want some CPU time, and Java is not (yet) very good about trying to keep threads and data colocated. This is the Big Challenge for highly parallel systems, and it's why you see groups such as the one at Rice University working on the Habanero Project where they are building into their language and VM core affinity. Particular with Java, I think it becomes a challenge because even if you have only as many application threads as cores, you still have these other VM threads beyond your control which can knock one or more of your threads from their perch, which can cause everything to get off so as to maintain some fairness. Java 7 is adding some NUMA logic to the GC to help optimize memory placement of objects, but I'm not sure what improvements there might be in thread placement/scheduling.
- June 2, 2010 at 8:57 PM
Ever think of Scala - Akka actors + JVM makes me think that it was the perfect match for your use-case.
- June 2, 2010 at 11:31 PM
- Lari Hotari said...
"Processor Adjacent Sector Prefetch" and "Processor Hardware Prefetcher" bios settings might also make a difference.
Quoting http://www.softpanorama.org/Internals/Performance_tuning/lecture02.shtml#Processor_performance_ :
"Both prefetch settings do decrease the miss rate for the L2/L3 cache when they are enabled but they consume bandwidth on the front-side bus which can reach capacity under heavy load. By disabling both prefetch settings, multi-core setups achieve generally higher performance and scalability."
- June 2, 2010 at 11:33 PM
- AndyGlew said...
(1) Disable hyperthreading. Most supercomputers do.
(2) The comment that memory contention should still cause CPU utilization *should* be true. I can imagine ways in which it might break, but I don't think those brokenesses are in current kernels.
(3) I'm a CPU guy, so I would suggest using EMON - on Linux, accessible through tools such as lkml.org/lkml/2009/6/6/149, oprofile, perfctr and perfmon2.
I would want to compare number of cache misses (from all important cache levels: L1, L2 (MLC), L3 (LLC)), bus utilizations, etc.
E.g. if the "bus" (QPI on Nehala) is above 80% utilized, then multiple processors probably won't help, and will probably hurt.
Paradoxically, if you have done a good job of parallelizing your code, making the threads NOT contend, then you may have shot yourself in the foot data cache wise - since t is possible that the footprint of 1 CPU's worth of threads fits in the cache, while the footprint of the larger number of concurrently running threads in a multiple processor system does not. You attempts to let each thread run as long as possible would exacerbate this.
- June 3, 2010 at 12:13 AM
- Nimrod said...
Thanks for the interesting post and the insightful discussion.
Regarding hyperthreading - HT was specifically designed to maximize processor utilization when memory/cache cannot keep up with computation.
Whenever a thread stalls due to reading from memory (writes may be easier - due to write back queues), the processor automatically switches to another thread running on the 2nd virtual processor. So overall utilization of the single physical processor will be higher.
- June 3, 2010 at 1:05 AM
- Frank said...
Why is the server multithreaded anyway? All it does is something like
If new message, get it and broadcast it
so if you spawn 20 of these threads they are all just trying to get the message, right?
- June 3, 2010 at 2:18 AM
- Noel Grandin said...
You're obviously hitting some shared resource, which is triggering cache-line bouncing, and you're doing so little work outside of moving packets around, that the inter-CPU traffic which cache-line-sharing triggers is dominating the runtime.
Are you sure there are no shared resources? Maybe you're creating threads, in which case you're hitting java's thread create lock, or maybe you're synchronising on something?
I've found the new classes in java.util.concurrrent to be extremely useful in writing good parallel code.
- June 3, 2010 at 2:21 AM
this is what happens when you post a highly subjective topic:
you get people who know more than you chiming in...
good one taskset though
- June 3, 2010 at 5:48 AM
- OraclePerformance said...
The neat thing about messaging programs is that they are horizontally scalable. You can scale producers and consumers blindly.
What the benchmark is showing seems to be protocol overhead. If you want a really fast 1 producer to 1 consumer - just hand off without messaging. Otherwise - you now have evidence you have a protocol wall. Your next step is likely either to change the protocol or go into protocol analysis (which is a path to java api and jvm and truss/strace/dtrace hell).
You know how to validate this point - simply check the bare messaging without a payload.
- June 3, 2010 at 6:33 AM
- Hiram Chirino said...
I ran into a similar scenario a few months also when working on a messaging server. Cross thread contention is amazingly difficult to reduce. I just posed Scaling Up With HawtDispatch which outlines some of the techniques that HawtDispatch uses to help avoid the contention.
Hiram Chirino
- June 3, 2010 at 9:10 AM
- Zetan Drableg said...
Run a 'kill -QUIT ' to get a thread dump during the test. Look for threads in a BLOCKED state.
- June 3, 2010 at 10:17 AM
- Paul Tyma said...
to @anonymous who said:
"this is what happens when you post a highly subjective topic:
you get people who know more than you chiming in..."
As I mentioned in the original post - that was sort of the entire point.
And these comments have been awesome. I plan on collecting them and trying a bunch of these ideas and report back.
- June 3, 2010 at 11:29 AM
- Mattias said...
Posted this on the other benchmark blog post by mistake, should've been here!
a)There is a lot that can be going on behind the scenes in the jvm here to start with. Such as similar to your other benchmark where the jvm will know that is running on a single-core cpu and can skip on emiting lock-prefixes on some instructions. It can even turn memory barriers into no-ops etc. See:
And according to Dave Dice (JVM guy at Sun):
"On MP IA32 and AMD64 systems we have to use locked atomic instructions (e.g., lock:cmpxchg) to acquire locks. On uniprocessors we can do without the lock: prefix. lock:ed instructions have considerable latency on some Intel processors (>500 cycles)."
So running on most x86 one-core machines (also when just allowing the process to use one core?) it will be a lot quicker to do synchronization.
b) Hyperthread cores are paired with a real core and usually provide faster context switching etc but that's all. No additional ALUs etc.
c) The cpu itself, such as:
c1) An i7 shares the rather large and quick L3 cache among cores. Running one core only you are effectively running with ONE very big L3 cache instead of it being split up between cores.
c2) Running on a few cores or just one core, the i7 cpu overclocks the used cores.
c3) I7 runs synchronisations primitives a lot faster, LOCK COMPXCHG instructions in 60% of the time compared to the previous generation for example.
- June 3, 2010 at 12:55 PM
- Vincent said...
I believe Mattias has the write answer.
The LOCK prefix will certainly be used when synchronizing on a global data structure (probably a queue) on a multiprocessor machine.
LOCK prefix ensures a CPU has ownership of the bus, which, can flush the cache of *other* cpus if I'm not mistaken.
Eliminating the LOCK prefix is *huge* for performance, especially something like this.
- June 3, 2010 at 4:55 PM
- Christian said...
What happens if you run multiple independent copies of the server as separate Java processes on different CPUs? If your architecture supports it, this would make it much easier to scale and avoid all locks (as these are then handled on application layer and not on hardware/CPU layer).
- June 4, 2010 at 12:58 AM
- raggi said...
Of the order of 100s of MB per second, CPU architecture is so far from the problem it's not even funny. I'm amazed at some of the comments being made here. Discuss, read, and profile the software stack before you start making random wild guesses about hardware.
We can push >100MB/s using reputably slow languages like ruby, sitting over 60% of their time in the GC scanning heap temporaries and the stack over and over and over, being so inefficient it's laughable. The CPU can do that. Java should be far far more efficient, and indeed in tests it's easily provable to be so.
The PEAK rate described here is in this range and is so much slower than memory bandwidths (at least one order of magnitude if I remember by DDR2 based FSB speeds correctly).
The cache is irrelevant here.
On my slow-ass-by-comparison macbook here, lets do a quick test running off a volatile volume:
raggi@mbk: ~/volatile % time dd if=/dev/zero of=bigfile bs=1048576 count=500
500+0 records in
500+0 records out
524288000 bytes transferred in 1.551369 secs (337951845 bytes/sec)
real 0m1.556s
user 0m0.003s
sys 0m0.793s
Now, /dev/zero probably isn't all that efficient, who knows, I don't really care, lets see if cp(1) can do any better:
raggi@mbk: ~/volatile % time cp bigfile bigfile2
real 0m2.351s
user 0m0.003s
sys 0m1.136s
Meh. that's a bit faster. Parallel read and write, reading 500mb, writing 500mb, total aggregate data transfer of 1GB in 2.4s, or ~400MB/s.
Also, note the split between real, user and sys.
Have fun with comment speculation round 20 or whatever it is, and good luck with the product!
- June 4, 2010 at 11:15 AM
- raggi said...
Oh, and further to my last, I should note that I'm now a couple of hundred megs into swap as a result of doing that. Go figure.
- June 4, 2010 at 11:18 AM
- raggi said...
It also just occurred to me that some "bright spark" will chime in that I'm benching a single IO locked process in volatile memory with the prior examples I gave in comments, so to be sure I'm invoking at least two processes and the kernel, here's another 'raw-ish' test:
raggi@mbk: ~/volatile % time sh -c "cat bigfile | cat - > bigfile2"
real 0m2.846s
user 0m0.117s
sys 0m2.185s
Still well over 300MB/s and most of the CPU time is being spent in `diskimages-helper` (OSX SL). ciao.
- June 4, 2010 at 11:42 AM
The way Java encourages you to do thread interaction is truly awful, and you're probably suffering from following what the language is suggesting you do. You'd probably be better off removing all uses of the word 'synchronized' from your code and having threads interact with each other strictly through queues, with as few objects which multiple threads are stepping on as possible.
- June 10, 2010 at 9:29 AM
- Matt Williamson said...
Actually, Bram is recommending a fantastic idea.
- June 10, 2010 at 6:25 PM
my guess is that the jvm knows that when it only has 1 core available, it doesn't need to skip the cache for volatile memory ops
as there isn't any other processor that needs an up to date value of it
(this is assuming the jvm knows how many cpu's it is scheduled for)
- June 27, 2010 at 5:38 PM
- Peter Lawrey said...
If you have program which should be single threaded, but you make it multiple threaded, it shouldn't be surprising that if you force it to use less CPUs the performance will improve.
On my system, the "synchronized Static store" test scaled particularly badly. However, even the single threaded case (53K, the best for this test) performed far worse than the single threaded Static store. (1721K)
i.e. it should never have been made multi-threaded, even if it scaled linearly, you would have needed 35 threads to match the performance of a single thread of the simpler non-synchronized test.
My approach is to minimise the amount of interlock between threads, this minimises the dependancy on how the hardware performs. IMHO This is not practical in all situations, but this approach can be used more often than most people think.
- July 14, 2010 at 2:22 AM
- Peter Lawrey said...
BTW: The code would be simplified if Goer timed how long the test runs. If the test is run long enough, the difference should be small, but would save three lines of boilerplate code from every test. You can also make the test run for a fixed time rather than having a hand chosen looping factor.
In each test you set junk1 to be the counter, and in some tests you set junk2 to be the last value, but in others you appear to have a copy-and-paste error using a variable from a different test.
- July 14, 2010 at 2:29 AM
- shenoyjoseph said...
nice trick
- November 16, 2010 at 2:27 PM
- Likai Liu said...
I'm going to second with Chas Emerick who mentioned lock contention. However, the state of art garbage collector is very likely more scalable than your server architecture.
Maybe your server implements some sort of work stealing scheduler that isn't aggressive enough? Spending too much time in the critical section? Ever heard of Amdahl's Law?
- January 8, 2011 at 7:16 AM
I've seen many a Java app where I constrain it to one or two CPUs and see improvement in actual work done. Takes a bit of tuning with jvm threads at times and sometimes tradeoffs with using more CPU and few jvms than a constraining to a single CPU for ever jvm just to reduce the care of feeding and memory overhead. It also depends on the exact version of Java you are using. This used to be FAR worse with Sun Java 1.3 but 1.4 and higher really it much less of a factor
My theory on why less is more is lock contention just like several comments have already mentioned. When waiting on resources having lots of threads that can at the same time compete for the same resources makes for fights. When forced to run on a single CPU only one JVM thread ever runs at any time since all threads have to run in the same core and hence there is never a fight for resources.
With that being said watch out for singleton areas and avoid mutexes as much as possible. Try turning off logging and see how much logging overhead is a factor. Try running with all the CPUs enabled but reduce the number of JVM threads to be equal to the number of cores plus some fudge factor to account for housekeeping (2 times cores usually is about max). Also there are different lock manager that can be used that may not be as fair as the standard but don't have the overhead.
- January 12, 2011 at 10:03 PM
I have another idea that might be fun to try and speed up Mailinator
Replace your "aging hash map" which you use to do your spam filter with a time-based, counting bloom filter.
- January 19, 2011 at 8:32 AM
- March 27, 2011 at 12:24 PM
- Emerson Almeida said...
I am Of Brasil, Muito Bom.
- April 2, 2011 at 9:30 AM
- Rodrigo L. said...
Very practical and simple.
- April 2, 2011 at 11:35 AM
- Jonadab said...
Do your threads have to talk to each other much? Fundamentally, any time two threads (or processes, or programs, or computers, or networks) have to communicate with one another, at least one of them can end up sitting around twiddling its thumbs waiting to hear from another one.
Even if the communication is taking place over the loop interface, it still has to take place, and therefore things can be I/O bound. Unless your threads can consistently keep doing stuff while they wait to hear from one another, you're going to have the potentially for I/O bottlenecks to prevent full CPU utilization.
This is interesting and possibly worth being aware of, but you really only need to do anything about it if you're going to be deploying the code in a way that introduces higher latency (e.g., if some of the threads could end up on a different computer from one another, especially if they could end up with slower-than-LAN links to one another).
- July 4, 2011 at 3:22 PM
- Matt Kemp said...
From my understanding (and what I've empirically observed) is that synchronized in Java causes an explicit cache flush at the end of the synchronized block. Depending on the processor this most likely flushes caches on multiple cores. Which would then cause them to re-fetch data from memory, slowing down their processing speed. Not 100% sure on why the single core is so much faster, but it probably has to do with either JVM optimizations when running on a single core or more efficient OS scheduling of fetching data for threads (since all threads are bound to a single core). As a related point, binding specific Java threads to cores can drastically improve those threads performance.
- February 22, 2012 at 10:51 AM
- Andy "Krazy" Glew said...
" synchronized in Java causes an explicit cache flush at the end of the synchronized block. "
Not true - at l;east not on Intel and AMD x86es, which I worked on.
I don't think it's true on Sun/Oracle SPARCs either. But I have worked on machine proposals, never built, where it might be true. And it might be true on a non-cache coherent MP system, which are uncommon.
What synchronized does is that the synchronized b lock is surrounded by a lock acquisition and release. Depending on the machine, this can be quite expensive. On modern Intel machines, however, it is almost free unless there is lock contention.
Oh, yes: finally, given hardware TM, which is just starting, the moral equivalent of the cache flush might happen. If they use the design I did 7 years ago for Intel, doing too many transactions back to back may incur a delay while the cache is scanned to flip bits. Not actually a flush, but still possibly a scan. However, this is a "too many back to back" thing. If fairly well spaced, or if Intel has provided enough versions, no problem. Or if Intel did a flash clear.
- February 23, 2012 at 9:04 AM
Mailinator.com : Anatomy of a Spammy Campaign
Mailinator is a popular disposable email service. It's also become a great tool for QA Teams to test email receipt, acknowledgment, au...
I recently gave a keynote at the ICIS 2009 conference in Shanghai. The topic was why multithreaded programming seemed so easy, yet turned ou...
Almost 3.5 years ago I started the Mailinator(tm) service. I got the bulk of the idea from my drunk roommate at the time and the first incar...
Mailinator is a popular disposable email service. It's also become a great tool for QA Teams to test email receipt, acknowledgment, au...