Superintelligence: The Driving Problem

A transcript of the above talk

Driving problems

Driving problems are problems that are used to foster the development of a desirable innovation.

The driving problem is not necessarily the same as the main application which the final invention is used for.

To illustrate with some examples:

• nuclear power was produced as a consequence of the development effort that resulted in the nuclear bomb;

• hydropower was developed by the ancient Greeks as a means of grinding grains into flour in order to produce bread;

• hot air balloons were originally developed for military communications.

Also, not every innovation has associated driving problems. Some things are discovered by accident. For example, the discoveries of penicillin, crisps, fireworks, saccharin, LSD and LEDs were largely the result of chance.

The driving problem for superintelligence

Here we will consider three clues associated with the origin of superintelligence:

• Follow the money

  The first clue relating to the issue involves the common idea of "following the money".

  The development of superintelligent machines will probably involve a considerable quantity of research money.

  Also, researchers with access to super-computers have about a 10 year head start on researchers working with desktops - in terms of when the required hardware becomes available.

  This approach points the finger at governments, large companies and large organisations.

  Within governments, supercomputer power is concentrated mostly in the intelligence agencies - such as the NSA in America. These organisations also have some of the smartest folk on the planet working for them - and face extremely-challenging problems.

  Among companies, server-side computing giants have some of the biggest computers. Search companies have an obvious need for superintelligent machines - and are among the most likely companies to develop them. These companies also have a lot of money and large research budgets.

  Another possibility is that the development of machine intelligence will be driven by investment management organisations who trade on the stock market.

  Both of these areas currently use enormous computers running highly sophisticated algorithms.

• Rapid development

  The second clue relating to the problem considers the issue of build-test cycle timing.

  One desirable property of the driving problem is that it should have a short build-test cycle - to facilitate rapid development.

  Any proposed intelligent machine needs to operate using sensors, processsing units and actuators.

  However, using mechanical sensors, and actuators in the real world would probably be slow and inefficient.

  Fortunately, there is no good reason why the sensors, and actuators of an intelligent machine need to extend very far into the "real" world. Virtual worlds can be highly complex and interesting - and they can also be made to operate rapidly.

  So, for example, a proposed intelligent machine could extend sensors and actuators onto the internet, and then perform operations at broadband speed.

  However, even this approach seems rather likely to be bottlenecked by network bandwidth issues. Why not extend sensors, and actuators into an entirely synthetic universe, and dispense with the network connection entirely?

  An example of this approach involves getting intelligent agents to play the game of go. The universe of the game of go is rich and complex, and apparently requires a deep and sophisticated intelligence to successfully navigate it. However, it is entirely self-contained - and can thus machine players be made to operate at very high speeds entirely within the computer.

• The core problem

  The third and final clue relates to the consideration of the core problem for a prospectively superintelligent machine.

  A machine will not be superintelligent until it is better at designing machines than humans are.

  As with other automation tasks, eliminating humans from the loop typically allows operation to be sped up considerably. So, it seems likely that the development of intelligent machines will accelerate when their development is perfomed primarily by other intelligent machines. So, the problem of developing a superintelligence will be mostly complete - once a designer of intelligent machines that is itself an intelligent machine has been created.

  Since this is - in some respects - the core problem, perhaps resources and attention are best devoted directly to it. The problem is broadly similar to that faced by high-level language designers: given a high-level specification of the problem to be solved, produce circuitry - or other machine code - which then solves the problem.

  From this perspective, tasks such as playing go - and playing the stockmarket - are seen as mostly distractions from the real underlying problem.

Conflicting clues

Playing the stockmarket potentially generates cash. A go-playing program can be sold for something - while developing computer programming language compilers typically produces something which is extremely challenging to sell.

Naturally, cash is important - it can be used to buy more and better hardware, and to pay the wages of researchers, programmers and inventors. Unfortunately, the most financially-attractive paths seem to be those that lead most indirectly towards the goal.

However, my expectation is that the strategy that produces superintelligent machines will heed each of these three clues - with financial considerations strongly influencing the most likely origin of the agents, while the other issues influencing the details of how the development takes place.

Which organisation?

I think it is hard to say very much more on the topic at this stage with much certainty. However, to my eyes, the NSA is well placed in some respects - but I have some difficulty in imagining their budget proposals for constructing a superintelligent agent. Also, if such an organisation constructed a superintelligence, they might do their best to keep it chained up in their underground basements for as long as possible.

DARPA funds might contribute to the development of superintelligent machines - but most of DARPA's projects seem like half-hearted side dishes to me. Their " Deep Green" battle computer is probably the nearest thing on their menu to a superintelligent machine. DARPA don't show many signs of trying hard to me.

The government claims a long history of funding research into intelligent machines. They did effectively invent ARPANET - the precursor to the internet. I'm sure the other funds they have put into the field have also helped. However, my perception is that the government's main influence has been via the funding of academic research. I am inclined to doubt whether that will be enough. It is difficult for me to imagine the scenario in which a superintelligent machine arises within a government department. It seems to me that most development in the IT industry has effectively been done by companies.

So: I currently think that scenarios in which superintelligent machines first arise within companies are the most realistic. Search oracles seem to me to be the most likely suspect. Stockmarket traders are probably the next in line, and perhaps after that conglomerate companies - global giants with fingers in various different pies.

The company scenario

At that point, it will probably have mastered molecular nanotechnology, and will have the ability to create autonomous living systems. Its options are thus likely to be fairly diverse. Then, much will depend on precisely what the intentions and values of its makers are.

Enjoy,