Optimization and the Singularity

Lest any­one get the wrong im­pres­sion, I’m jug­gling mul­ti­ple balls right now and can’t give the lat­est Sin­gu­lar­ity de­bate as much at­ten­tion as it de­serves. But lest I an­noy my es­teemed co-blog­ger, here is a down pay­ment on my views of the Sin­gu­lar­ity—need­less to say, all this is com­ing way out of or­der in the post­ing se­quence, but here goes...

Among the top­ics I haven’t dealt with yet, and will have to in­tro­duce here very quickly, is the no­tion of an op­ti­miza­tion pro­cess. Roughly, this is the idea that your power as a mind is your abil­ity to hit small tar­gets in a large search space—this can be ei­ther the space of pos­si­ble fu­tures (plan­ning) or the space of pos­si­ble de­signs (in­ven­tion). Sup­pose you have a car, and sup­pose we already know that your prefer­ences in­volve travel. Now sup­pose that you take all the parts in the car, or all the atoms, and jum­ble them up at ran­dom. It’s very un­likely that you’ll end up with a travel-ar­ti­fact at all, even so much as a wheeled cart; let alone a travel-ar­ti­fact that ranks as high in your prefer­ences as the origi­nal car. So, rel­a­tive to your prefer­ence or­der­ing, the car is an ex­tremely im­prob­a­ble ar­ti­fact; the power of an op­ti­miza­tion pro­cess is that it can pro­duce this kind of im­prob­a­bil­ity.

You can view both in­tel­li­gence and nat­u­ral se­lec­tion as spe­cial cases of op­ti­miza­tion: Pro­cesses that hit, in a large search space, very small tar­gets defined by im­plicit prefer­ences. Nat­u­ral se­lec­tion prefers more effi­cient repli­ca­tors. Hu­man in­tel­li­gences have more com­plex prefer­ences. Nei­ther evolu­tion nor hu­mans have con­sis­tent util­ity func­tions, so view­ing them as “op­ti­miza­tion pro­cesses” is un­der­stood to be an ap­prox­i­ma­tion. You’re try­ing to get at the sort of work be­ing done, not claim that hu­mans or evolu­tion do this work perfectly.

This is how I see the story of life and in­tel­li­gence—as a story of im­prob­a­bly good de­signs be­ing pro­duced by op­ti­miza­tion pro­cesses. The “im­prob­a­bil­ity” here is im­prob­a­bil­ity rel­a­tive to a ran­dom se­lec­tion from the de­sign space, not im­prob­a­bil­ity in an ab­solute sense—if you have an op­ti­miza­tion pro­cess around, then “im­prob­a­bly” good de­signs be­come prob­a­ble.

Ob­vi­ously I’m skip­ping over a lot of back­ground ma­te­rial here; but you can already see the gen­e­sis of a clash of in­tu­itions be­tween my­self and Robin. Robin’s look­ing at pop­u­la­tions and re­source uti­liza­tion. I’m look­ing at pro­duc­tion of im­prob­a­ble pat­terns.

Look­ing over the his­tory of op­ti­miza­tion on Earth up un­til now, the first step is to con­cep­tu­ally sep­a­rate the meta level from the ob­ject level—sep­a­rate the struc­ture of op­ti­miza­tion from that which is op­ti­mized.

If you con­sider biol­ogy in the ab­sence of ho­minids, then on the ob­ject level we have things like dinosaurs and but­terflies and cats. On the meta level we have things like nat­u­ral se­lec­tion of asex­ual pop­u­la­tions, and sex­ual re­com­bi­na­tion. The ob­ject level, you will ob­serve, is rather more com­pli­cated than the meta level. Nat­u­ral se­lec­tion is not an easy sub­ject and it in­volves math. But if you look at the anatomy of a whole cat, the cat has dy­nam­ics im­mensely more com­pli­cated than “mu­tate, re­com­bine, re­pro­duce”.

This is not sur­pris­ing. Nat­u­ral se­lec­tion is an ac­ci­den­tal op­ti­miza­tion pro­cess, that ba­si­cally just started hap­pen­ing one day in a tidal pool some­where. A cat is the sub­ject of mil­lions of years and billions of years of evolu­tion.

Cats have brains, of course, which op­er­ate to learn over a life­time; but at the end of the cat’s life­time, that in­for­ma­tion is thrown away, so it does not ac­cu­mu­late. The cu­mu­la­tive effects of cat-brains upon the world as op­ti­miz­ers, there­fore, are rel­a­tively small.

Or con­sider a bee brain, or a beaver brain. A bee builds hives, and a beaver builds dams; but they didn’t figure out how to build them from scratch. A beaver can’t figure out how to build a hive, a bee can’t figure out how to build a dam.

So an­i­mal brains—up un­til re­cently—were not ma­jor play­ers in the plane­tary game of op­ti­miza­tion; they were pieces but not play­ers. Com­pared to evolu­tion, brains lacked both gen­er­al­ity of op­ti­miza­tion power (they could not pro­duce the amaz­ing range of ar­ti­facts pro­duced by evolu­tion) and cu­mu­la­tive op­ti­miza­tion power (their prod­ucts did not ac­cu­mu­late com­plex­ity over time). For more on this theme see Protein Re­in­force­ment and DNA Con­se­quen­tial­ism.

Very re­cently, cer­tain an­i­mal brains have be­gun to ex­hibit both gen­er­al­ity of op­ti­miza­tion power (pro­duc­ing an amaz­ingly wide range of ar­ti­facts, in time scales too short for nat­u­ral se­lec­tion to play any sig­nifi­cant role) and cu­mu­la­tive op­ti­miza­tion power (ar­ti­facts of in­creas­ing com­plex­ity, as a re­sult of skills passed on through lan­guage and writ­ing).

Nat­u­ral se­lec­tion takes hun­dreds of gen­er­a­tions to do any­thing and mil­lions of years for de novo com­plex de­signs. Hu­man pro­gram­mers can de­sign a com­plex ma­chine with a hun­dred in­ter­de­pen­dent el­e­ments in a sin­gle af­ter­noon. This is not sur­pris­ing, since nat­u­ral se­lec­tion is an ac­ci­den­tal op­ti­miza­tion pro­cess that ba­si­cally just started hap­pen­ing one day, whereas hu­mans are op­ti­mized op­ti­miz­ers hand­crafted by nat­u­ral se­lec­tion over mil­lions of years.

The won­der of evolu­tion is not how well it works, but that it works at all with­out be­ing op­ti­mized. This is how op­ti­miza­tion boot­strapped it­self into the uni­verse—start­ing, as one would ex­pect, from an ex­tremely in­effi­cient ac­ci­den­tal op­ti­miza­tion pro­cess. Which is not the ac­ci­den­tal first repli­ca­tor, mind you, but the ac­ci­den­tal first pro­cess of nat­u­ral se­lec­tion. Dist­in­guish the ob­ject level and the meta level!

Since the dawn of op­ti­miza­tion in the uni­verse, a cer­tain struc­tural com­mon­al­ity has held across both nat­u­ral se­lec­tion and hu­man in­tel­li­gence...

Nat­u­ral se­lec­tion se­lects on genes, but gen­er­ally speak­ing, the genes do not turn around and op­ti­mize nat­u­ral se­lec­tion. The in­ven­tion of sex­ual re­com­bi­na­tion is an ex­cep­tion to this rule, and so is the in­ven­tion of cells and DNA. And you can see both the power and the rar­ity of such events, by the fact that evolu­tion­ary biol­o­gists struc­ture en­tire his­to­ries of life on Earth around them.

But if you step back and take a hu­man stand­point—if you think like a pro­gram­mer—then you can see that nat­u­ral se­lec­tion is still not all that com­pli­cated. We’ll try bundling differ­ent genes to­gether? We’ll try sep­a­rat­ing in­for­ma­tion stor­age from mov­ing ma­chin­ery? We’ll try ran­domly re­com­bin­ing groups of genes? On an ab­solute scale, these are the sort of bright ideas that any smart hacker comes up with dur­ing the first ten min­utes of think­ing about sys­tem ar­chi­tec­tures.

Be­cause nat­u­ral se­lec­tion started out so in­effi­cient (as a com­pletely ac­ci­den­tal pro­cess), this tiny hand­ful of meta-level im­prove­ments feed­ing back in from the repli­ca­tors—nowhere near as com­pli­cated as the struc­ture of a cat—struc­ture the evolu­tion­ary epochs of life on Earth.

And af­ter all that, nat­u­ral se­lec­tion is still a blind idiot of a god. Gene pools can evolve to ex­tinc­tion, de­spite all cells and sex.

Now nat­u­ral se­lec­tion does feed on it­self in the sense that each new adap­ta­tion opens up new av­enues of fur­ther adap­ta­tion; but that takes place on the ob­ject level. The gene pool feeds on its own com­plex­ity—but only thanks to the pro­tected in­ter­preter of nat­u­ral se­lec­tion that runs in the back­ground, and is not it­self rewrit­ten or al­tered by the evolu­tion of species.

Like­wise, hu­man be­ings in­vent sci­ences and tech­nolo­gies, but we have not yet be­gun to rewrite the pro­tected struc­ture of the hu­man brain it­self. We have a pre­frontal cor­tex and a tem­po­ral cor­tex and a cere­bel­lum, just like the first in­ven­tors of agri­cul­ture. We haven’t started to ge­net­i­cally en­g­ineer our­selves. On the ob­ject level, sci­ence feeds on sci­ence, and each new dis­cov­ery paves the way for new dis­cov­er­ies—but all that takes place with a pro­tected in­ter­preter, the hu­man brain, run­ning un­touched in the back­ground.

We have meta-level in­ven­tions like sci­ence, that try to in­struct hu­mans in how to think. But the first per­son to in­vent Bayes’s The­o­rem, did not be­come a Bayesian; they could not rewrite them­selves, lack­ing both that knowl­edge and that power. Our sig­nifi­cant in­no­va­tions in the art of think­ing, like writ­ing and sci­ence, are so pow­er­ful that they struc­ture the course of hu­man his­tory; but they do not ri­val the brain it­self in com­plex­ity, and their effect upon the brain is com­par­a­tively shal­low.

The pre­sent state of the art in ra­tio­nal­ity train­ing is not suffi­cient to turn an ar­bi­trar­ily se­lected mor­tal into Albert Ein­stein, which shows the power of a few minor ge­netic quirks of brain de­sign com­pared to all the self-help books ever writ­ten in the 20th cen­tury.

Be­cause the brain hums away in­visi­bly in the back­ground, peo­ple tend to over­look its con­tri­bu­tion and take it for granted; and talk as if the sim­ple in­struc­tion to “Test ideas by ex­per­i­ment” or the p<0.05 sig­nifi­cance rule, were the same or­der of con­tri­bu­tion as an en­tire hu­man brain. Try tel­ling chim­panzees to test their ideas by ex­per­i­ment and see how far you get.

Now… some of us want to in­tel­li­gently de­sign an in­tel­li­gence that would be ca­pa­ble of in­tel­li­gently re­design­ing it­self, right down to the level of ma­chine code.

The ma­chine code at first, and the laws of physics later, would be a pro­tected level of a sort. But that “pro­tected level” would not con­tain the dy­namic of op­ti­miza­tion; the pro­tected lev­els would not struc­ture the work. The hu­man brain does quite a bit of op­ti­miza­tion on its own, and screws up on its own, no mat­ter what you try to tell it in school. But this fully wraparound re­cur­sive op­ti­mizer would have no pro­tected level that was op­ti­miz­ing. All the struc­ture of op­ti­miza­tion would be sub­ject to op­ti­miza­tion it­self.

And that is a sea change which breaks with the en­tire past since the first repli­ca­tor, be­cause it breaks the idiom of a pro­tected meta-level.

The his­tory of Earth up un­til now has been a his­tory of op­ti­miz­ers spin­ning their wheels at a con­stant rate, gen­er­at­ing a con­stant op­ti­miza­tion pres­sure. And cre­at­ing op­ti­mized prod­ucts, not at a con­stant rate, but at an ac­cel­er­at­ing rate, be­cause of how ob­ject-level in­no­va­tions open up the path­way to other ob­ject-level in­no­va­tions. But that ac­cel­er­a­tion is tak­ing place with a pro­tected meta-level do­ing the ac­tual op­ti­miz­ing. Like a search that leaps from is­land to is­land in the search space, and good is­lands tend to be ad­ja­cent to even bet­ter is­lands, but the jumper doesn’t change its legs. Oc­ca­sion­ally, a few tiny lit­tle changes man­age to hit back to the meta level, like sex or sci­ence, and then the his­tory of op­ti­miza­tion en­ters a new epoch and ev­ery­thing pro­ceeds faster from there.

Imag­ine an econ­omy with­out in­vest­ment, or a uni­ver­sity with­out lan­guage, a tech­nol­ogy with­out tools to make tools. Once in a hun­dred mil­lion years, or once in a few cen­turies, some­one in­vents a ham­mer.

That is what op­ti­miza­tion has been like on Earth up un­til now.

When I look at the his­tory of Earth, I don’t see a his­tory of op­ti­miza­tion over time. I see a his­tory of op­ti­miza­tion power in, and op­ti­mized prod­ucts out. Up un­til now, thanks to the ex­is­tence of al­most en­tirely pro­tected meta-lev­els, it’s been pos­si­ble to split up the his­tory of op­ti­miza­tion into epochs, and, within each epoch, graph the cu­mu­la­tive ob­ject-level op­ti­miza­tion over time, be­cause the pro­tected level is run­ning in the back­ground and is not it­self chang­ing within an epoch.

What hap­pens when you build a fully wraparound, re­cur­sively self-im­prov­ing AI? Then you take the graph of “op­ti­miza­tion in, op­ti­mized out”, and fold the graph in on it­self. Me­taphor­i­cally speak­ing.

If the AI is weak, it does noth­ing, be­cause it is not pow­er­ful enough to sig­nifi­cantly im­prove it­self—like tel­ling a chim­panzee to rewrite its own brain.

If the AI is pow­er­ful enough to rewrite it­self in a way that in­creases its abil­ity to make fur­ther im­prove­ments, and this reaches all the way down to the AI’s full un­der­stand­ing of its own source code and its own de­sign as an op­ti­mizer… then even if the graph of “op­ti­miza­tion power in” and “op­ti­mized product out” looks es­sen­tially the same, the graph of op­ti­miza­tion over time is go­ing to look com­pletely differ­ent from Earth’s his­tory so far.

Peo­ple of­ten say some­thing like “But what if it re­quires ex­po­nen­tially greater amounts of self-rewrit­ing for only a lin­ear im­prove­ment?” To this the ob­vi­ous an­swer is, “Nat­u­ral se­lec­tion ex­erted roughly con­stant op­ti­miza­tion power on the ho­minid line in the course of cough­ing up hu­mans; and this doesn’t seem to have re­quired ex­po­nen­tially more time for each lin­ear in­cre­ment of im­prove­ment.”

All of this is still mere analogic rea­son­ing. A full AGI think­ing about the na­ture of op­ti­miza­tion and do­ing its own AI re­search and rewrit­ing its own source code, is not re­ally like a graph of Earth’s his­tory folded in on it­self. It is a differ­ent sort of beast. Th­ese analo­gies are at best good for qual­i­ta­tive pre­dic­tions, and even then, I have a large amount of other be­liefs not yet posted, which are tel­ling me which analo­gies to make, etcetera.

But if you want to know why I might be re­luc­tant to ex­tend the graph of biolog­i­cal and eco­nomic growth over time, into the fu­ture and over the hori­zon of an AI that thinks at tran­sis­tor speeds and in­vents self-repli­cat­ing molec­u­lar nanofac­to­ries and im­proves its own source code, then there is my rea­son: You are draw­ing the wrong graph, and it should be op­ti­miza­tion power in ver­sus op­ti­mized product out, not op­ti­mized product ver­sus time. Draw that graph, and the re­sults—in what I would call com­mon sense for the right val­ues of “com­mon sense”—are en­tirely com­pat­i­ble with the no­tion that a self-im­prov­ing AI think­ing mil­lions of times faster and armed with molec­u­lar nan­otech­nol­ogy, would not be bound to one-month eco­nomic dou­bling times. Nor bound to co­op­er­a­tion with large so­cieties of equal-level en­tities with differ­ent goal sys­tems, but that’s a sep­a­rate topic.

On the other hand, if the next Big In­ven­tion merely in­fringed slightly on the pro­tected level—if, say, a se­ries of in­tel­li­gence-en­hanc­ing drugs, each good for 5 IQ points, be­gan to be in­tro­duced into so­ciety—then I can well be­lieve that the eco­nomic dou­bling time would go to some­thing like 7 years; be­cause the ba­sic graphs are still in place, and the fun­da­men­tal struc­ture of op­ti­miza­tion has not re­ally changed all that much, and so you are not gen­er­al­iz­ing way out­side the rea­son­able do­main.

I re­ally have a prob­lem with say­ing, “Well, I don’t know if the next in­no­va­tion is go­ing to be a re­cur­sively self-im­prov­ing AI su­per­in­tel­li­gence or a se­ries of neu­rophar­ma­ceu­ti­cals, but whichever one is the ac­tual case, I pre­dict it will cor­re­spond to an eco­nomic dou­bling time of one month.” This seems like sheer Kurzweilian think­ing to me, as if graphs of Moore’s Law are the fun­da­men­tal re­al­ity and all else a mere shadow. One of these es­ti­mates is way too slow and one of them is way too fast—he said, eye­bal­ling his men­tal graph of “op­ti­miza­tion power in vs. op­ti­mized product out”. If we are go­ing to draw graphs at all, I see no rea­son to priv­ilege graphs against times.

I am jug­gling many balls right now, and am not able to pros­e­cute this dis­pute prop­erly. Not to men­tion that I would pre­fer to have this whole con­ver­sa­tion at a time when I had pre­vi­ously done more posts about, oh, say, the no­tion of an “op­ti­miza­tion pro­cess”… But let it at least not be said that I am dis­miss­ing ideas out of hand with­out jus­tifi­ca­tion, as though I thought them un­wor­thy of en­gage­ment; for this I do not think, and I have my own com­plex views stand­ing be­hind my Sin­gu­lar­ity be­liefs, as one might well ex­pect.

Off to pack, I’ve got a plane trip to­mor­row.