Handicapped Superintelligence

Handicapped Superintelligence

Hi, I'm Tim Tyler - and today I will be discussing the how the possible utility functions of superintelligent agents could adversely affect their long-term survival.

Biological organisms can be thought of as having utility functions, and - counter-intuitively - it appears that the best way to think about these is in terms involving the entropy they generate.

However, it seems that we could synthetize intelligent machines with practically any utility function we like.

While there could be cultural evolution of the machines - and perhaps natural selection between them, if there is a single sufficiently large and powerful agent in charge of everything, it could channel development in whatever direction it chose - for example, by imposing its own fitness function. It appears to be possible that universal cooperation and self-directed evolution could create a system with an arbitrary goal.

Also, there are reasons to expect arbitrarily-chosen goal systems to be reasonably stable over time.

This leads to the possibilty of future goals being systematically different from past ones.

A number of authors have proposed that - in order to prolong the survival of our own species - we should ensure that any superintelligent agents that we make are our slaves - that they are wired to obey humans.

However, there may be some long-term problems associated with wiring a dependency on humanity into a superintelligent agent. One is that such a dependency could well eventually become a handicap.

A similar situation has been dramatised in comic book stories - in particular, in the "Superman" tales. Here is a scene from Superman 2:

[Zod attack scene from Superman 2]

In the scene, an evil alien identifies Superman's feelings towards humans as a weakness - and proceeds to use this weakness to attack him.

Now, we are not currently facing threats from any alien races. However, if we do so in the future, then we probably do not want to have handicapped our entire civilisation. While dependencies on humans may have the effect of postponing the demise of our species, they also have considerable potential to hamper and slow evolutionary progress. For example humans have poor space-travel potential, and any tendency to keep humans around will be associated with remaining stuck on the home world. That is not necessarily a very positive effect in the face of the possibility of impacts by large meteorites.

Others have made similar observations. For example, Richard Hollerith has described deviations from the most fundamental goal as being "moral contaminants".

It seems obvious that systems with such "contaminants" are likely to be find themselves at a competitive disadvantage. A system might be reasonably secure and stable enough to avoid the possibility of fragmentation and internal rebellion - but it is hard to rule out the possibility of encountering aliens with goal systems that are not so badly contaminated.

Almost any deviation from God's utility function is likely to represent a competitive weakness. We do not know exactly what would happen if two powerful superintelligent agents with different and thus incompatible goal systems encountered each other - but it seems likely that one of them would eventually competitively exclude the other.

Of course, the probabilty of us encountering another civilisation anytime soon seems to be low. However, we may well encounter one eventually.

In my opinion, we should do our best to think carefully about the long-term implications before we settle on a utility function for our superintelligent agents. This is especially true if we decide to modify nature's utility function, in order to promote our own ends.

The issue may represent a unique opportunity to permanently mess up our civilization's values - and so curtail its ultimate future potential.



  1. Tim Tyler - God's Utility Function

  2. Richard Hollerith - Danger Level Four - If we build a superhuman AI, what goal do we give it?

  3. Richard Hollerith - Goal System Zero

Tim Tyler | Contact | http://alife.co.uk/