hp.html
Hackers and Painters

May 2003(This essay is derived from a guest lecture at Harvard, which incorporated
an earlier talk at Northeastern.)When I finished grad school in computer science I went
to art school to study painting.  A lot of people seemed surprised
that someone interested in computers would also be interested in painting.
They seemed to think that
hacking and painting were very different kinds of work-- that
hacking was cold, precise, and methodical, and that
painting was the frenzied expression of some primal urge.Both of these images are wrong.  Hacking and painting have a
lot in common.  In fact, of all the different types of people I've
known, hackers and painters are among the most alike.What hackers and painters have in common is that they're
both makers.  Along with composers, architects, and writers,
what hackers and painters are trying to do is make good things.
They're not doing research per se, though if in the course of
trying to make good things they discover some new technique,
so much the better.I've never liked the term "computer science."  The main
reason I don't like it is that there's no such thing.
Computer science is a
grab bag of tenuously related areas thrown together
by an accident of history, like Yugoslavia.
At one end you have people who are really mathematicians,
but call what they're doing computer science so they can get DARPA grants.
In the middle you have people working on
something like the natural history of computers-- studying the
behavior of algorithms for routing data through
networks, for example.  And then at the other extreme you
have the hackers, who are trying to
write interesting software, and for whom computers are just a
medium of expression, as concrete is for architects or
paint for painters.  It's as if
mathematicians, physicists, and architects all had to be in
the same department.Sometimes what the hackers do is called "software engineering,"
but this term is just as misleading.
Good software designers are no more engineers than architects are.
The border between architecture and engineering is not sharply
defined, but it's there.
It falls between what and how: architects decide what to do,
and engineers figure out how to do it.What and how should not be kept too separate.  You're
asking for trouble if you try to decide what to do without
understanding how to do it.
But hacking can certainly be more than just deciding how to
implement some spec.  At its best, it's creating the spec-- though
it turns out the best way to do that is to implement it.Perhaps one day
"computer science" will, like Yugoslavia, get broken up into its
component parts.  That might be a good thing.  Especially if it
meant independence for my native land, hacking.Bundling all these different types of work together in one
department may be convenient administratively, but it's confusing
intellectually. That's the other reason I don't like the name
"computer science."  Arguably the people in the middle are doing
something like an experimental science.  But the people at either
end, the hackers and the mathematicians, are not actually doing science.The mathematicians don't seem bothered by this.  They happily
set to work proving theorems like the other mathematicians
over in the math department, and probably soon stop noticing
that the building they work in says ``computer science'' on the
outside.  But for the hackers this label is a problem.
If what they're doing is called science, it makes them feel they
ought to be acting scientific.
So instead of doing what they really want to do, which is   
to design beautiful software, hackers in universities and
research labs feel they ought to be writing research papers.In the best case, the papers are just a formality.  Hackers write
cool software, and then write a paper about it, and the paper
becomes a proxy for the achievement represented by the software.
But often this mismatch causes problems.  It's easy to
drift away from building beautiful things toward building ugly
things that make more suitable subjects for research papers.Unfortunately, beautiful things don't always make the
best subjects for papers.
Number one, research must be original-- and
as anyone who has written a PhD dissertation knows, the way to
be sure that you're exploring virgin territory is to to stake
out a piece of ground that no one wants.  Number two, research must be
substantial-- and awkward systems yield meatier papers,
because you can write about the obstacles you have to overcome
in order to get things done.  Nothing yields meaty problems like
starting with the wrong assumptions.  Most of AI is an example
of this rule; if you assume that knowledge can be represented
as a list of predicate logic expressions whose arguments represent
abstract concepts, you'll have a lot of
papers to write about how to make this work.  As <NAME>
used to say, "Lucy, you got a lot of explaining to do."The way to create something beautiful is often to make subtle
tweaks to something that already exists, or to combine existing
ideas in a slightly new way.  This kind of work is hard to
convey in a research paper.So why do universities and research labs continue to judge
hackers by publications?
For the same reason that "scholastic aptitude"
gets measured by simple-minded standardized tests, or
the productivity of programmers gets measured in lines of code.
These tests
are easy to apply, and there is nothing so tempting as an easy test
that kind of works.Measuring what hackers are actually trying to do, designing
beautiful software, would be much more difficult.  You need
a good sense of design to judge 
good design.  And
there is no correlation, except possibly
a negative 
one, between people's ability to recognize good
design and their confidence that they can.The only external test is time.  Over time, beautiful
things tend to thrive, and ugly
things tend to get discarded.  Unfortunately, the amounts of time
involved can be longer than human lifetimes.  <NAME>
said it took a hundred years for a writer's reputation to
converge.  You have to wait for the writer's
influential friends to die, and then for all their followers
to die.I think hackers just have to resign themselves to having a large random
component in their reputations.  In this they are no different
from other makers.  In fact, they're lucky by comparison.   
The influence of fashion is not nearly so great in hacking as it
is in painting.There are worse things than having people misunderstand your
work.  A worse danger is that you
will yourself misunderstand your work.  Related fields are
where you go looking for ideas.  If you find yourself in the computer science
department, there is a natural temptation to believe, for example,
that hacking is the applied version of what theoretical computer
science is the theory of.   All
the time I was in graduate school I had an uncomfortable feeling
in the back of my mind that I ought to know more theory,
and that it was very remiss of me to have forgotten all that
stuff within three weeks of the final exam.Now I realize I was
mistaken.  Hackers need to understand the theory of computation
about as much as painters need to understand paint chemistry.
You need to know how to calculate time and
space complexity and about
Turing completeness.  You might also want to remember at
least the concept of a state machine, in case you have to write
a parser or a regular expression library.  Painters in fact   
have to remember a good deal more about paint chemistry than 
that.I've found that the best sources of ideas
are not the other fields that have the word "computer" in
their names, but the other fields inhabited by makers.
Painting has been a much richer source of ideas than the
theory of computation.For example, I was taught in college
that one ought to figure out a program
completely on paper
before even going near a computer.  I found that I did not
program this way.  I found that I liked to program
sitting in front of a computer, not a piece of paper.  Worse
still, instead of patiently writing out a complete program
and assuring myself it was correct, I tended to just spew
out code that was hopelessly broken, and gradually beat it into
shape.  Debugging, I was taught, was a kind of final pass where
you caught typos and oversights.  The way I worked, it
seemed like programming consisted of debugging.For a long time I felt bad about this, just as I once
felt bad that I didn't hold my pencil the way they taught me
to in elementary school.
If I had only looked over at
the other makers, the painters or the architects, I would
have realized that there was a name for what I was doing:
sketching.  As far as I can tell, the
way they taught me to program in college was all wrong.
You should figure out programs as you're writing them,
just as writers and painters and architects do.Realizing this has real implications for software design.
It means that a programming language should, above all, be
malleable.  A programming language is for 
thinking of
programs, not for expressing programs you've already thought
of.  It should be a pencil, not a pen.  Static typing would
be a fine idea if people actually did write programs the way
they taught me to in college.  But that's not how any of the  
hackers I know write programs.  We need a language that lets us
scribble and smudge and smear, not a language where you have
to sit with a teacup of types balanced on your knee and make
polite conversation with a strict old aunt of a compiler.While we're on the subject of static typing, identifying with
the makers will save us from another problem that afflicts
the sciences: math envy.  Everyone in the sciences
secretly believes that mathematicians are smarter than they are.  
I think mathematicians also believe this.  At any rate,
the result is that scientists tend to make their
work look as mathematical as possible.  In a field like
physics this probably doesn't do much harm, but the further you
get from the natural sciences, the more of a problem it
becomes.A page of formulas just looks so impressive.
(Tip: for extra impressiveness, use Greek variables.)  And
so there is a great temptation to work on problems you
can treat formally, rather than problems that are, say,
important.If hackers identified with other makers, like writers and
painters, they wouldn't feel tempted to do      
this.  Writers and painters don't suffer from math envy.
They feel as if they're doing something completely unrelated.
So are hackers, I think.If universities and research labs keep hackers from doing
the kind of work they want to do,
perhaps the place for them is in companies.
Unfortunately, most companies won't let hackers do what they
want either.  Universities and research labs force hackers
to be scientists, and companies force them to be engineers.I only discovered this myself quite recently.  When Yahoo bought
Viaweb, they asked me what I wanted to do.  I had never
liked the business side very much, and said that I just wanted to
hack.  When I got to Yahoo, I found that what hacking meant
to them was implementing software, not designing it.  Programmers
were seen as technicians who translated the visions (if
that is the word) of product managers into code.This seems to be the
default plan in big companies.  They do it because
it decreases the standard deviation of the outcome.
Only a small percentage of hackers can actually design software,
and it's hard for the
people running a company to pick these out.  So instead of
entrusting the future of the software to
one brilliant hacker, most companies set things up so that it is
designed by committee, and the hackers merely
implement the design.If you want to make money at some point, remember this,
because this is one of the reasons startups win.  Big companies want
to decrease the standard deviation of design outcomes because they
want to avoid disasters.  But when you damp oscillations, you
lose the high points as well as the low.  This is not a problem for
big companies, because they don't win by making great
products.  Big companies win by sucking less than other big companies.So if you can figure out a way to get in a
design war with a company big enough that its software is   
designed by product managers, they'll never be able to keep up
with you.  These opportunities are not easy to find, though.
It's hard to engage a big company in a design war,
just as it's hard to engage an opponent inside a castle in hand
to hand combat.  It would be pretty easy to write a better
word processor than Microsoft Word, for example, but Microsoft,
within the castle of their operating system monopoly,
probably wouldn't even notice if you did.The place to fight design wars is in new markets, where no one
has yet managed to establish any fortifications.  That's where
you can win big by taking the bold approach to design, and
having the same people both design and implement the product.  
Microsoft themselves did this at the start.  So did Apple.
And Hewlett-Packard.  I suspect almost every successful startup
has.So one way to build great software is to start your own
startup.  There are two problems with this, though.  One is
that in a startup you have to do so much besides write software.
At Viaweb I considered myself lucky if I
got to hack a quarter of the time.  And the things I had to   
do the other three quarters of the time ranged from tedious
to terrifying.  I have a benchmark for this, because I
once had to leave a board meeting to have
some cavities filled.  I remember sitting back in the
dentist's chair, waiting for the drill, and feeling like
I was on vacation.The other problem with startups is that there is not much
overlap between the kind of software that makes money and the
kind that's interesting to write.  Programming languages
are interesting to write, and Microsoft's first product was
one, in fact, but no one will pay for programming languages
now.  If you want to make money, you tend to be forced to work
on problems that are too nasty for anyone to solve for free.All makers face this problem.  Prices are
determined by supply and demand, and there is just not as much
demand for things that are fun to work on as there is for
things that solve the mundane problems of individual customers.
Acting in off-Broadway plays just doesn't pay as well as
wearing a gorilla suit in someone's booth at a
trade show.  Writing novels doesn't pay as well as writing
ad copy for garbage disposals.
And hacking programming languages doesn't pay as well
as figuring out how to connect some company's
legacy database to their Web server.I think the answer to this problem, in the case of software,
is a concept known to nearly all makers: the day job.
This phrase began with musicians, who
perform at night.  More generally, it means that you have one
kind of work you do for money, and another for love.Nearly all makers have day jobs early in their careers.
Painters and writers notoriously do.  If you're lucky
you can get a day job that's closely
related to your real work.  Musicians often
seem to work in record stores.  A hacker working on some
programming language or operating system might likewise be able to
get a day job using it.  [1]When I say that the answer is for hackers to have day jobs, 
and work on beautiful software on the side, I'm not proposing
this as a new idea.  This is what open-source hacking is all   
about.  What I'm saying is that open-source is probably the right
model, because it has been independently confirmed by all the  
other makers.It seems surprising to me that any employer would be reluctant
to let hackers work on open-source projects.
At Viaweb, we would have been reluctant to hire anyone
who didn't.  When we interviewed
programmers, the main
thing we cared about was what kind of software they
wrote in their spare time.
You can't do anything really well unless
you love it, and if you love to hack you'll inevitably
be working on projects of your own. [2]Because hackers are makers rather than scientists,
the right place to look for metaphors is not in the
sciences, but among other kinds of makers.  What else can painting
teach us about hacking?One thing we can learn, or at least confirm, from the
example of painting is how to learn to hack.  You learn to
paint mostly by doing it.
Ditto for hacking.  Most hackers don't learn to hack by
taking college courses in programming.  They learn to hack
by writing programs of their own at age thirteen.  Even in   
college classes, you learn to hack mostly by hacking. [3]Because painters leave a trail of work behind them, you
can watch them learn by doing.  If you look at the work
of a painter in chronological order, you'll find that each  
painting builds on things that have been learned in previous
ones.  When there's something in
a painting that works very well, you can usually find version 
1 of it in a smaller form in some earlier painting.I think most makers work this way.  Writers and architects seem
to as well.  Maybe it would be good for hackers
to act more like painters, and regularly start over from scratch,
instead of continuing to work for years on one project, and
trying to incorporate all their later ideas as revisions.The fact that hackers learn to hack by doing it is another
sign of how different hacking is from the sciences.  Scientists
don't learn science by doing it, but by doing labs and problem sets.
Scientists start out doing work that's perfect, in the sense
that they're just trying to reproduce work someone else has 
already done for them.
Eventually, they get
to the point where they can do original work.
Whereas hackers, from the start, are doing original work; it's
just very bad.  So hackers start original, and get good, and
scientists start good, and get original.
The other way makers learn is from examples.
For a painter, a museum is a reference library of techniques.
For hundreds of years it has been part of the traditional
education of painters to copy the works of the great masters,
because copying forces you to look closely
at the way a painting is made.Writers do this too.
<NAME> learned to write by summarizing the points   
in the essays of Addison and Steele and then trying to
reproduce them.  <NAME> did the same thing
with detective stories.Hackers, likewise, can learn to program by looking at 
good programs-- not just at what they do, but the source
code too.  One of the less publicized benefits
of the open-source movement is that it has made it easier
to learn to program.  When I learned to program, we had to rely
mostly on examples in books.  The one big chunk of
code available then was Unix, but even this was not   
open source.  Most of the people who read the source
read it in illicit photocopies of <NAME>' book, which
though written in 1977 was not allowed to be published
until 1996.Another example we can take from painting is the way that
paintings are created by gradual refinement.  Paintings usually
begin with a sketch.
Gradually the details get filled in.
But it is not merely a process of filling in. Sometimes   
the original plans turn out to be mistaken.
Countless paintings,
when you look at them in xrays, turn out to have limbs that
have been moved or facial features that have been readjusted.Here's a case where we can learn from painting.  I think hacking
should work this way too.  It's unrealistic
to expect that the specifications for a program will be
perfect. You're
better off if you admit this up front, and write programs in
a way that allows specifications to change on the fly.(The structure of large companies makes this hard for them
to do, so here is another place where startups have an advantage.)Everyone by now presumably knows about the danger of premature
optimization.  I think we should be just as worried about
premature design-- deciding too early what
a program should do.The right tools can help us avoid
this danger.
A good programming language should, like oil paint, make it
easy to change your mind.  Dynamic typing is a win here because
you don't have to
commit to specific data representations up front.
But the key to flexibility, I think, is to make the language
very abstract.
The easiest program to change is one that's very short.
This sounds like a paradox, but a great painting
has to be better than it has to be.
For example, when Leonardo
painted the portrait of <NAME>
in the National Gallery, he put a juniper bush behind her head.
In it he carefully
painted each individual leaf.  Many painters might have thought,
this is just something to put in the background to frame
her head.  No one will look that closely at it.Not Leonardo.  How hard he worked on part of a painting didn't
depend at all on how closely he expected anyone to look at it.
He was like <NAME>.  Relentless.Relentlessness wins because, in the aggregate, unseen details
become visible.
When people walk by the portrait of <NAME>,
their attention is often immediately arrested by it,
even before they look at the label and notice that it says
<NAME>.  All those unseen details combine to produce
something that's just stunning, like a thousand barely audible
voices all singing in tune.Great software, likewise, requires a fanatical devotion to
beauty.  If you look inside good software, you find that
parts no one is ever supposed to see are beautiful too.
I'm not claiming I write great software, but I
know that when it comes to code I behave in a way that would
make me eligible for prescription drugs if I approached everyday
life the same way.
It drives me crazy to see code that's badly indented,
or that uses ugly variable names.If a hacker were a mere implementor, turning a spec into code, then
he could just work his way through it from one end to the other like
someone digging a ditch.  But if the hacker is a creator, we have
to take inspiration into account.In hacking, like painting,
work comes in cycles.  Sometimes you get excited about some
new project and you want to work sixteen hours a day on it. 
Other times nothing seems interesting.To do good work you have to take these cycles into
account, because they're affected by how you react to them.
When you're driving a
car with a manual transmission on a hill, you have to back off
the clutch sometimes to avoid stalling.  Backing
off can likewise prevent ambition from stalling.
In both painting and hacking there are some
tasks that are terrifyingly ambitious, and others that are
comfortingly routine.  It's a good idea to save some easy
tasks for moments when you would otherwise stall.In hacking, this can literally mean saving up bugs.
I like debugging: it's the
one time that hacking is as straightforward as   
people think it is.  You have a
totally constrained problem, and all you have to do is solve
it.  Your program is supposed to do x.  Instead it does y.
Where does it go wrong? You know you're going to win
in the end.  It's as relaxing as painting a wall.The example of painting can teach us not only how to manage our
own work, but how to work together.  A lot of the
great art of the past is the work of multiple hands, though
there may only be one name on the wall next to it in the
museum.  Leonardo was an apprentice in the workshop of
Verrocchio and painted one of the angels in his Baptism of
Christ.  This sort of thing was the rule, not the exception.
Michelangelo was considered especially dedicated for insisting
on painting all the figures on the ceiling of the Sistine
Chapel himself.As far as I know, when painters worked together on a painting,
they never worked on the same parts.  It was common
for the master to paint the principal figures and for assistants
to paint the others and the background.  But you never had
one guy painting over the work of another.I think this is the right model for collaboration in software
too.  Don't push it too far.  When a piece of code is
being hacked by three or four different people, no one of whom
really owns it, it will end up being like a common-room.  It will
tend to feel bleak and abandoned, and accumulate cruft.
The right
way to collaborate, I think, is to divide projects into sharply
defined modules, each with a definite owner, and with interfaces
between them that are as carefully designed and, if possible,
as articulated as programming languages.Like painting, most software is intended for
a human audience.  And so hackers, like painters, must have
empathy to do really great work.  You have to be able to see
things from the user's point of view.When I was a kid I was always being told to look at things from
someone else's point of view.  What this always meant in
practice was to do what someone else wanted, instead of what
I wanted.  This of course gave empathy a bad name, and I made a
point of not cultivating it.Boy, was I wrong.  It turns out that looking at things from 
other people's point of view is practically the secret of
success.  It doesn't necessarily mean being self-sacrificing.
Far from it.  Understanding how someone else sees things
doesn't imply that you'll act in his interest; in some
situations-- in war, for example-- you want to do exactly
the opposite. [4]Most makers make things for a human audience.
And to engage an audience you have to understand what they need.
Nearly all the greatest paintings are paintings of people,
for example, because people are what people are interested in.Empathy is probably the single most important difference
between a good hacker and a great one.  Some hackers
are quite smart, but when it comes to empathy are
practically solipsists.  It's hard for such      
people to design great software [5], because they can't
see things from the user's point of view.One way to tell how good people are at empathy is to watch
them explain a technical question to someone without a technical
background.  We probably all know people who, though otherwise smart,
are just comically bad at this.  If someone asks them at
a dinner party what a programming language is, they'll
say something like ``Oh, a high-level language is what
the compiler uses as input to generate object code.''
High-level language?  Compiler?  Object code?  Someone who 
doesn't know what a programming language is obviously doesn't
know what these things are, either.Part of what software has to do is explain itself.  So to   
write good software you have to understand how little users   
understand.
They're going to walk up to the software with no preparation, and
it had better do what they guess it will, because they're
not going to read the manual.  The best system I've ever seen 
in this respect was the original Macintosh, in 1985.
It did what software almost never does: it just worked. [6]Source code, too, should explain itself.  If I could get people to
remember just one quote about programming, it would be the
one at the beginning of Structure and Interpretation of Computer
Programs.  

Programs should be written for people to read, and
only incidentally for machines to execute.

You need to have 
empathy not just for your users, but for your readers.  It's in  
your interest, because you'll be one of them.
Many a hacker has written a program only to
find on returning to it six months later that he has no idea   
how it works.  I know several people who've sworn off Perl after
such experiences. [7]Lack of empathy is associated with intelligence, to the point
that there is even something of a fashion for it in some places.
But I don't think there's any correlation.
You can do well in math and
the natural sciences without having to learn empathy, and people in these
fields tend to be smart, so the two qualities have come to be
associated.  But there are plenty of dumb people who are bad at
empathy too.  Just listen to the people who call in with questions on
talk shows.  They ask whatever it is they're asking in
such a roundabout way
that the hosts often have to rephrase the question for them.So, if hacking works like painting and writing, is it as cool?
After all, you only get one life.
You might as well spend it working on something great.Unfortunately, the question is hard to answer.  There is always
a big time lag in prestige.  It's like light from a distant star.
Painting has prestige now because of great work people did five hundred
years ago.  At the time, no one thought
these paintings were as important as we do today.  It would have
seemed very odd to people at the time that <NAME>,
the Duke of Urbino, would one day be known mostly as the guy
with the strange nose in a painting 
by <NAME>.So while I admit that hacking doesn't seem as cool as painting now,
we should remember that painting itself didn't seem as cool in
its glory days as it does now.What we can say with some confidence is that these are the glory
days of hacking.  In most fields the great work is done early on.
The paintings made between 1430 and 1500 are still unsurpassed.
Shakespeare appeared just as professional theater was being born,

and pushed the medium
so far that every playwright since has had to live in his shadow.
<NAME> did the same thing with engraving, and <NAME>
with the novel.Over and over we see the same pattern.  A new medium appears, and
people are so excited about it that they explore most of its
possibilities in the first couple generations.   Hacking seems
to be in this phase now.Painting was not, in Leonardo's time, as cool as his work
helped make it.
How cool hacking turns out to be will depend on what we can
do with this new medium.  

Notes[1] The greatest damage that photography has done
to painting may be the fact that it killed the best day job.
Most of the great painters in history supported
themselves by painting portraits.  [2] I've been told that Microsoft discourages
employees from contributing to open-source projects, even in
their spare time.
But so many of the best hackers work on open-source
projects now that the main effect of this policy may be
to ensure that they won't be able to hire any first-rate
programmers.[3] What you learn about programming in college is much like
what you learn about books or clothes or dating: what bad taste you
had in high school.[4] Here's an example of applied empathy.
At Viaweb, if we couldn't decide between two alternatives, we'd
ask, what would our competitors hate most?  At one point a
competitor added a feature to their software that was basically
useless, but since it was one of few they had that we didn't, they
made much of it in the trade press.
We could have tried to explain that the feature was useless,
but we decided it would annoy our competitor more if we
just implemented it ourselves, so we hacked together our own
version that afternoon.[5] Except text editors and compilers.  Hackers don't need empathy to
design these, because they are themselves typical users.[6] Well, almost.  They overshot the available RAM somewhat,
causing much inconvenient disk swapping, but this could be fixed
within a few months by buying an additional disk drive.[7] The way to make programs easy to read is not to
stuff them with comments. I would take Abelson and Sussman's
quote a step further.  Programming languages should be designed
to express algorithms, and only incidentally to tell computers
how to execute them.  A good programming language
ought to be better for explaining software than English.
You should only
need comments when there is some kind of kludge you need to warn
readers about, just as on a road there are only
arrows on parts with unexpectedly sharp curves.
Thanks to <NAME>, <NAME>, <NAME>, and <NAME> for reading drafts of this, and to <NAME>
and <NAME> for inviting me to speak.Japanese TranslationSpanish TranslationGerman TranslationPortuguese TranslationCzech TranslationWhy Good Design Comes from Bad DesignKnuth: Computer Programming as an Art


You'll find this essay and 14 others in
Hackers & Painters.


