Distinctions in Types of Thought

by sarahconstantin12 min read10th Oct 201724 comments


General IntelligenceDistinctionsWorld Modeling

Epistemic status: speculative


Effortful vs. Effortless



Furthermore, James divides attention into two types:





















Explicit vs. Implicit








Subject-Object vs. Relational





Aware vs. Unaware










Relationships and Corollaries






The Weirdness of Thinking on Purpose





Two weirdly similar quotes:


–Frank Herbert, Dune, 1965



–Ayn Rand, For the New Intellectual, 1963

(bold emphasis added, ellipses original).




24 comments, sorted by Highlighting new comments since Today at 8:03 PM
New Comment

Evolving brains took a long time. Learning to think, once we had a monkey's brain, was comparatively fast. If we focus on "do this algorithm's conscious thoughts at all resembles human conscious thoughts?", I expect AI progress will look like many decades of "not at all" followed by a brief blur as machines overtake humans.

I agree with you that there is a real cluster of distinctions here. I also agree with the takeaway:


But I suspect there isn't any concise explanation of "how it works"; thinking is a big complicated machine that works for a long list of messy reasons, with the underlying meta-reason that evolution tried a bunch of stuff and kept what worked.

It also looks increasingly likely that we won't understand how thinking works until after we've built machines that think, just as we can now build machines-that-perceive much better than we understand how perception works.

It seems casually obvious that raccoons and crows engage in deliberate thought. Possibly hunting spiders and octopods do. Also obvious at this point that we have more processing power available than hunting spiders.
Everything biology does has an explanation as concise as the algorithmic complexity of the genome, which isn't obviously intractable for understanding at all of the relevant levels of description.

I could believe that. My guess is that the things brains do which are "interpretable" or "explicit" are a very small subset.

OTOH, I'm interested in the possibilities that might arise from building machines that have things similar to "attention" or "memory" or "self-other distinctions", things which mammalian brains generally have but most machine learners don't. I think there's qualitatively new stuff that we're only beginning to touch.

I still find myself confused that you don't express thinking that the more complicated architectures provide a plausible guide for what to expect these things to look like in the brain. I have come to the conclusion that all the parts of intelligence - thinking and blindsight among them - have been tried by ML people at this point, but none of them at large enough scale or integrated well enough to produce something like us. It continues to seem strange to me that your objection is "they haven't actually tried the right thing", and that you are also optimistic about attention/memory/etc as being the right sort of thing to produce it. Do you think that thinking doesn't have an obvious construction from available parts? What are the cruxes, the diff of our beliefs here?

I really wish you would give his argument for the claim that we (even plausibly) have all the pieces, Lahwran. I would also love to see an abridged transcript of a discourse wherein the two of you reached a double-crux. My best guess is that Lahwran is thinking of 'only integrating existing systems' as a triviality which can be automated by the market rather than what it actually is, a higher-level instance of the design problem.

That said, the idea that thinking has been tried seems so insane to me that I may be failing to steelman it accurately.

I was under the impression that things like "deliberative thinking" and "awareness" haven't been simulated by ML thus far, so I think that's the diff between us -- though it's not that strongly held, there are lots of ML advances I may just not have heard of.

An example of what I would mean by thinking: https://arxiv.org/pdf/1705.03633.pdf

Thanks for the paper!

At first I was very surprised that they got such good performance at answering questions about visual scenes (e.g. "what shape is the red thing?" "the red thing is a cube.")

Then I noticed that they gave ground-truth examples not just for the answers to the questions but to the programs used to compute those answers. This does not sound like the machine "learned to reason" so much as it "learned to do pattern-recognition on examples of reasoning." When humans learn, they are "trained" on examples of other people's behavior and words, but they don't get any access to the raw procedures being executed in other people's brains. This AI did get "raw downloads of thinking processes," which I'd consider "cheating" compared to what humans do. (It doesn't make it any less of an achievement by the paper authors, of course; you have to do easier things before you can do harder things.)

That seems like weaseling out of the evidence to me. This is just another instance of neural networks being able to learn to do geometric computation to produce hard-edged answers, like alphago is; that they're being used to generate programs seems not super relevant to that. I certainly agree that it's not obvious exactly how to get them to learn the space of programs efficiently, but it seems surprising to expect it to be different in kind vs previous neural network stuff. This doesn't seem that different to me vs attention models in terms of what kind of problem learning the internal behavior presents.

Promoted to Featured, for carefully communicating a number of novel and interesting models regarding the path to AGI (a topic that's both useful and important).

This immediately reminded me of this paper which I saw on Short Science last week.

Yep yep yep!!! I am a Tenenbaumist in my outlook, pretty much. Sometimes I forget how big an influence he's had on my thinking.

I'm not influenced by him, but his (and Gary Marcus') approach seems like common sense to me. It seems likely that Sarah's ability to track ideology to its philosophical roots can contribute to Tenenbaum's and Marcus' abilities to present their perspectives compellingly to people with more philosophical and less technical bent who have the relatively high trust in consensus which tends to accompany technical ability but which is the opposite of Thiel's effective investment strategies.


If you use e.g. the Akaike Information Criterion for model evaluation, you get around the size problem in theory. Model size is then something you score explicitely.

Personally, I still have intuitive problems with this approach though: many phenomenological theories in physics are easier to interpret than Quantum Mechanics, and seem to be intuitively less complex, but are more complex in a formal sense (and thus get a worse AIC score, even if they predict the same thing).


'Irreducible' is a pretty strong stance. I agree that many things will be hard for humans to describe ein a way that other humans find satisfying. But do you think that an epistemically rational entity with unlimited computational power (something like a Solomonov Inductor) would be unable to do that?

Also, WOW I had no idea Arbital's writing was so good. (The Solomonov Inductor link.) In case anyone else didn't click the first time, it's not just a definition, it's a dialogue, probably by Eliezer, and it's super cool.

So, it's usually possible to create "adversarial examples" -- datasets that are so "perverse" that they resist accurate prediction by simple models and actually require lots of variables. (For a very simple example, if you're trying to fit a continuous curve based on a finite number of data points, you can make the problem arbitrarily hard with functions that are nowhere differentiable.) I'm not being that rigorous here, but I think the answer to the question "are there irreducibly complex statistical models?" is yes. You can make models such that any simplification has a large accuracy cost.

Are there irreducibly complex statistical models that humans or animals use in real life? That's a different and harder question, and my answer there is more like "I don't know, but I could believe so."


I agree that there are some sources of irreducible complexity, like 'truely random' events.

To me, the field of cognition does not pattern-match to 'irrecducibly complex', but more to 'We don't have good models. Yet, growth mindset'. So, unless you have some patterns where you can prove that they are irrreducible, I will stick with my priors I guess. The example you gave me,


falls squarely in the 'our models are bad'-category, e.g. the Weierstrass function can be stated pretty compactly with analyitic formulas.

But also, of course I can't prove the non-existence of such irreducible, important processes in the brain.


Fair enough.

Ah, how you think about that example helps clarify. I wasn't even thinking about the possibility of an AI that could "learn" the analytic form of Weierstrass function, I was thinking about the fact that trying to fit a polynomial to it would be arbitrarily hard.

Obviously "not modelable by ANY means" is a much stronger claim than "if you use THESE means, then your model needs a lot of epicycles to be close to accurate." (Analyst's mindset vs. computer scientist's mindset; the computer scientist's typical class of "possible algorithms" is way broader. I'm more used to thinking like an analyst.)

I think you and I are pretty close to agreement at this point.

Yes, I completely agree with the weaker formulation "irreducible using only THESE means", like e.g. Polynomials, MPTs, First-Order Logic etc.

I think thought is irrelevant.

If we have two systems A and B.

For all inputs, A and B produce the same corresponding outputs.

A and B have the same domain and the same image.

Then it doesn't matter if the algorithms of A and the algorithms of B are different, as far as I'm concerned, A and B are functionally equivalent.

@SarahConstantin I think the intricacies of human thought are irrelevant.

I don't think we need machines to approximate human thought.

If you want machines to reach human level intelligence, then you need them to approximate human behaviour.

I don't think the fact that machines don't think the way we do is at all relevant to human level machine intelligence.

It seems like a non issue.

I think the difference in internal algorithmic process is largely irrelevant.

Having matching algorithmic processes is one way to get two systems have the same output for all input, but it's not the only way.

For any function f, it is possible to construct an infinite number of functions f'_i that are functionally equivalent to f, but possess different algorithmic processes.

f'_0(x) = f(x) + 0.

f'_i{x} = f'_{i-1}(x) + i - i. (i > 0, i in N).

The cardinality of the set of such functions is the same as the cardinality of the set of natural numbers.

What we really want is functional equivalence.

We do not need algorithmic equivalence (where algorithmic equivalence is when the algorithms match exactly).

We care about the output, and not the processing.

Algorithmic equivalence is a proper subset of functional equivalence.

If your only criticism of B is that B is not algorithmically equivalent to A even though it is functionally equivalent to A, then I would say that you're missing the point.

I think criticising algorithmic equivalence is missing the point. Unless it is not functionally equivalent, then you don't need to criticise algorithmic equivalence.

If it is functionally equivalent, then you don't need to criticise algorithmic equivalence and can directly criticise the lack of functional equivalence.

In short, there's no need to criticise algorithmic equivalence?

Unless algorithmic equivalence is pursued s the path to functional equivalence.

I was not under the impression that algorithmic equivalence was what is necessarily being pursued.

(Admittedly NN are based on gaining algorithmically similar architectures).

Computer vision is not algorithmically equivalent to human vision.

I think the same is true for speech recognition as well.

I don't think you can prove that algorithmic equivalence is necessary.

I think that's a fucking insane thing to prove.

I mean mention 3 significant ML achievements that are algorithmically equivalent to their human counterparts.

I do think algorithmic similarity matters:

The human algorithm is a very general problem-solving algorithm. If we can solve problems with an algorithm that looks very similar to the human algorithm, at a similar level of competence to the human algorithm, then this is evidence about the generalizability of our algorithm, since we should expect algorithms that are very similar to the human algorithm to also have similar generality.

You'll have to decompose what you mean by "algorithmic similarity"; if I presented you a universal learning algorithm that looked very different from the human universal learning algorithm, would you say that the two algorithms were simiar?

What makes two algorithms similar?

I think algorithm equivalence is unnecessary. We do not want algorithmic equivalence , what we actually want is functional equivalence. If the human universal learning algorithm was random, and implementing it did not produce a human engine of cognition, then you would not want to implement the human universal learning algorithm. Algorithmic equivalence is a subordinate desire to functional equivalence; it is desired only in so much as it produces functional equivalence.

The true desire is functional equivalence (which is itself subordinate to Human Level Machine Intelligence).

If we have functional equivalence we do not need to criticise algorithmic equivalence, and if we do not have functional equivalence we do not need to criticise algorithmic equivalence and algorithmic equivalence is not what we actually desire; it is a subordinate desire to functional equivalence.

If modern ML was pursuing algorithmic equivalence is a route to functional equivalence, then--and only then--woud it make sense to criticise a lack of algorithmic equivalence. However, it has not been my impression that this is the path AI is taking.

Machine natural Language Processing is not algorithmically equivalent to human nature language processing.

Computer vision is not algorithmically equivalent to human vision.

Computer speech recognition is not algorithmically equivalent to human speech recognition.

AlphaGo is not algorithmically equivalent to human Go.

I can not recall a single significant concrete machine learning achievement in a specific problem domain made by gaining algorithmic equivalence with the human engine of cognition. If you can name 3 examples, I would update in favour of algorithmic equivalence.

Neural networks are human inspired, but the actual algorithms those neural networks run are not algorithmically equivalent to the algorithms we run.

FWIW, https://www.edge.org/response-detail/26794 is the best actual argument I have seen for deep learning being a qualitative new capability, but it looks like the capability "analogy/metaphor" not the capability 'awareness' or 'deliberate thought'.

Yep, I'd agree on both counts.