system
Given the following post, respond as if you're commenting on Reddit:
human
Refuting Eliezer Yudowsky

I’m sure many people here are familiar with Eliezer Yudowsky and his ilk over at LessWrong. One of the most defining aspects of their philosophy is of what I’m going to term, for want of a better name, ‘literal utilitarianism’. That is to say, you do exactly what the utilitarian maths suggests, with no regard to whether or not the results seem wildly counterintuitive. So for example, you torture 1 person for 50 years to stop Graham’s number of people from getting dust specks in their eyes. This would lead to all kinds of hideous issues like the utility monster and the repugnant conclusion, but Yudowsky just steamrolls over all that by arguing that utilitarianism is rational, your intuitions are wrong, and you’re an irrational monkey if you see utilitarianism as wrong because it gives horrifying results. 

Now, I’m personally of the opinion that Yudowsky only advocates this form of utilitarianism in order to convince people that they have a moral imperative to donate to the AI research ‘charity’ that he works for, but that’s besides the point. Why would a standard philosopher say he’s wrong? The vast majority of philosophers accept the counterarguments against utilitarianism that its conclusions can be apalling, and try to somehow explain them away rather than trying to take them at face value. So what would a philosopher say in defence of the common sense idea that say, you shouldn’t kill 1 person for a 1 in a billion chance of saving 1 billion people? 

The main counterargument I can think of, without completely discarding utilitarianism, is that utilitarianism is only ‘correct’ because our intuitions tell us it is, and there’s real way to argue any given ethical theory is ‘rational’ because ethics is too subjective for that. Therefore, when it conflicts with our intuitions, instead of ‘shutting up and multiplying’, we change it so it doesn’t give results that look like that - because if you dismiss the validity of ethical intuition, you’re effectively dismissing the entire basis of utilitarianism in the first place. 

Are there any other counterarguments? What would a philosopher from a non LessWrong school of thought say? Why is the standard response to the utility monster, or the repugnant conclusion ‘your feelings are wrong and the maths is right’?
gpt
It seems like you’re less concerned with Yudowsky as an individual and more concerned with “literal utilitarianism”, so I’ll talk more about that instead of why Yudowsky’s beliefs in general tend to be wrong.

The biggest “non intuition based” argument against the form of utilitarianism you outlined (which to be clear isn’t the version most actual utilitarians believe) is epistemic. 

If we subscribe to the maxim that we ought to generate the most utility possible through our actions, but also acknowledge that we can’t know how much utility an action generates until *after the actions’ consequences play out,* then some serious issues start to arise.

For instance, a utilitarian in the year 1900 might believe that it’s a moral imperative to center our world economy around oil, because it’s cheap, abundant, and very energy dense compared to alternatives. However, this utilitarian doesn’t know about climate change, and thus is unwittingly hurting total utility in the long run!

This objection, in short, hinges on the notion that we cannot know how much utility any given action will generate. Since we cannot know how much utility an action will generate, basing an ethical system purely on “maximizing utility” is borderline incoherent unless we modify our criteria or introduce some new restrictions beyond actual utility.

One way around this is “rule utilitarianism”, but I’m guessing Yudowsky wouldn’t like that too much!