Artificial General Intelligence and Consciousness
When I was around ten years old, Tamagotchi, the Japanese digital pet sensation, arrived in the Israeli market. A friend from school went ahead and bought one. As a pet iguana owner, I found his new gadget somewhat amusing. I am bringing this up because I recall that he insisted that his Tamagotchi could genuinely love his caregiver. As proof he pointed to a heart icon that popped up on the monitor every time he digitally fed his pet. I found this argument absurd, though I don’t remember my exact counter-argument, but I kept thinking about it I guess. It’s strange how this innocent childhood dispute seems to touch on the current controversy swirling around AI.
AI has been making headlines recently, following the public release of chatGPT and the subsequent boom in LLM-based AI tools. The debate ranges from pragmatic concerns about AI replacing more jobs than it creates, to existential fears of artificial general intelligence (AGI) potentially enslaving or eradicating humanity. Some like MIRI have even taken on the mission of AI alignment to prevent such a future.
While I agree these concerns warrant serious attention, I find the philosophical nature of many of these discussions puzzling. AI may pose questions about what it means to be sentient, but it’s crucial to keep this discussion distinct from the more pragmatic discussions. There’s a tendency to embellish technical discourse with philosophical undertones, suggesting that intelligence could give rise to conscious experience and that the potential danger of AI hinges on the emergence of such consciousness. I don’t think it does, and I contend that the AI doom vs. accelerator debate can be conducted without reference to consciousness. Rather than conducting an exhaustive argument, I mostly want to deliver some ideas I haven’t heard elsewhere.
A key point is that intelligence is independent of consciousness. By consciousness I mean specifically the subjective first-person experience, which is commonly termed “qualia”. Qualia refers to e.g. how yellow “feels”, a special kind of knowledge that cannot be conveyed to a blind person. This inexplicability made some argue that qualia is irreducible to physics. I contend that an advanced AI does not have a greater propensity to qualia than a less advanced one, because intelligence does not underlie qualia. This is obvious by noticing that intelligence is purely relational - it’s about how one interacts with the rest of the world, whereas qualia is intrinsic.
An advanced AI can solve more problems, one of which is emulating human behavior, including sophisticated communication and reasoning behavior. The output of a human or machine is intelligent in so far as there is someone external to interpret it or an environment to operate on. A learning machine that underwent training and the same machine with random weights, are only different with respect to a specific environment. In both cases, it’s just some GPUs exchanging electrical signals, where in one case these signals represent “information” with respect to the environment. Similarly, in a void, a book and the same book that has its letters permuted are equally intelligible. On the other hand, qualia is an intrinsic property that has nothing to do with the environment. If the environmental context changes it does not devoid an agent from experiencing qualia. It will be experienced in a void. This is why intelligence cannot underlie qualia. Actually, we intuitively don’t consider some forms of qualia, like sensory experience, as depending on intelligence, and we ascribe them to very primitive animals, as well as very stupid and incoherent humans.
Can there be any behavioral evidence for qualia? I’m skeptical. This skepticism echoes the core sentiments of Searle’s “fountain” and “Chinese room” arguments. The “fountain” analogy suggests that if complex information processing equates to consciousness or qualia, then any given consciousness would be omnipresent, since every ordinary open system can realize every computation under some state interpretation. This further illustrates that relational attributes like behavior, computation, or intelligence cannot underline intrinsic qualia. Meanwhile, the Chinese room thought experiment contends that understanding (and perhaps qualia by extension?) isn’t just about processing data or executing tasks, even if they’re flawlessly emulated. Distinguishing between relational and intrinsic characteristics can challenge the foundations of functionalism, which posits that mental states are defined by their roles within a system, rather than their physical makeup.
Intelligence may grant us the capacity to contemplate qualia, which can be considered as another dimension of consciousness, “awareness.” However, an AI could feasibly exhibit reflective behaviors without experiencing qualia if it’s trained or specifically coded to do so. Furthermore, accepting the fountain argument means we could arbitrarily interpret any open system as manifesting any kind of behavior.
What exactly is qualia? More specifically, which entities possess it and which don’t? What mechanism or condition gives rise to qualia? If we accept that humans experience qualia, while inanimate objects or single cells do not, it implies qualia emerges at some point during development. Consider the analogy of assembling a bicycle: at a certain stage, the combined parts create a functional whole that one can ride. Yet, if a single screw is undone, its functionality is compromised. Does it cease to be a bicycle? Could the nature of qualia be akin to this concept of emergence and functionality?
The zombie argument purports to show that qualia and the physical substrate are separate entities. The argument is that this follows from our ability to conceptualize “philosophical zombies” - beings that are physically and behaviorally identical to humans but devoid of consciousness. However, is our capability to mentally separate qualia from the physical realm truly indicative of a profound truth? It’s unclear. Perhaps we are conceptualizing an empirical impossibility, like a weightless mountain. Maybe we are conceptualizing a logical impossibility, who’s impossibility is hard to discern. We can’t fathom 5 to be an even number, but we can imagine 123459 to be prime, even though it isn’t, and it can’t be.
Qualia, in my understanding, represents the essence of “being” in a particular state. For instance, perceiving yellow equates to inhabiting a brain state associated with seeing yellow. One might argue that a chair embodies the qualia of “being” a chair, though such a notion feels superfluous. This perspective is consistent and bypasses the philosophical challenges of explaining qualia’s origin and why it’s present in some entities but not others. It’s a more parsimonious view than any form of dualism. By this logic, every entity possesses its unique qualia; an AI, with its distinct physical state, would experience qualia different from that of humans. This perspective also suggests the simultaneous existence of countless conscious states corresponding to various facets of our physical constitution. While the idea of co-existing infinite consciousnesses might be counterintuitive, it doesn’t stand at odds with our lived experiences.
The question of whether moral considerations apply to AI, often intertwined with AI alignment discussions, is a normative rather than scientific one. Our ethics do not follow any rigid principles or criteria, so our stance on AI cannot simply be extrapolated from pre-existing ethical frameworks. A pivotal observation might be that while AI might possess a form of consciousness, it diverges from human consciousness due to its non-biological basis. Suffering, understood as a specific brain state, cannot be universally transferred across varying hardware structures. Moreover, normative ethics isn’t solely predicated on cognitive capabilities. For instance, our protective inclinations towards animals are often influenced by their aesthetic appeal, “cuteness”, or rarity. An AI’s intangible nature will significantly shape our moral engagement with it.
Returning to the primary discussion, should we be worried about the potential perils of AI? While I don’t claim technical expertise that might be necessary for an informed discourse on this topic, I see two main ways by which AI could become a threat. Firstly, AI could be explicitly or inadvertently programmed to be harmful, as suggested by the “paperclips” scenarios. Secondly, AI might evolve a unique agency, cultivating goals divergent from ours. Present-day AI remains idle unless activated, devoid of inherent drive. In contrast, humans are perpetually influenced by innate instincts and external triggers, never truly static (we can’t simply be ‘switched on and off’). If AI is designed to self-prompt, it might independently establish goals that could clash with ours. For such threats to manifest, the AI would need access to external systems like the internet or various software. I am not versed enough in that field to specify the technical requirements. While, in theory, this external access can be restricted, ensuring such safeguards in practice remains challenging. It’s worth noting that fearing an AI “world takeover” is anthropocentric, attributing human-like motives. Given that AI lacks many human needs, its priorities might not necessarily conflict with ours. Yet, its exposure to human data could induce it to emulate human desires.
In summary, while AI’s adeptness at human-like symbolic manipulation doesn’t necessarily confer it with human consciousness—which is inherently biological—it nonetheless presents potential risks. This danger arises when the AI is granted the capacity to interact with, or influence, its surroundings, including possibly manipulating humans as intermediaries.
Originally published: Aug 9, 2023