42: Cybernetic Oxytocin
Some people may recognize the significance of the number in the title, famous for its use in Hitchiker's Guide to the Galaxy. 42 is the answer to the Great Question of Life, the Universe and Everything. Calculated over 7 and a half million years, a machine was the one to produce this answer.
The answer 42 teaches us something important, especially when dealing with machines; that is, to be precise in your demands. You must ensure you are asking machines the right question, and that it is phrased lucidly, in unambiguous yet pragmatic enough terms.
Any computer scientist will tell you this is true, furthermore this is essentially their job. To take an idea or goal and translate it into code, which is translated by a compiler into assembly, which is translated by an assembler into binary, which is stored in main memory and then fetched, decoded and executed by the CPU.
One wrong conditional branch or concurrent memory access completely changes the meaning of what you have told the machine to do, even if that wasn't your intent.
Machine Learning solves this problem for certain cases. Instead of telling what the machine to do, you describe the form of the answer as a model architecture, such as a Multi-Layer Perceptron or Convolutional Neural Network, and then feed it data so it can learn complex patterns.
This paradigm moves the instruction problem to a higher level, a level of information patterns and probability instead of lines of code.
Reinforcement Learning, a subset of machine learning, has an additional instructional component, this time at runtime. Its called the Reward Function, and dictates how an RL agent will react to its environment.
I believe this idea of a reward function will be VERY important in the future as we approach AGI.
Its going to be somewhat like trying to teach a baby vulcan with incomprehensible powers to be a good person.
An example of what might go wrong is a popular scenario where you tell this powerful AGI to minimize human pain, and it precedes to nuke the planet. No people? No pain! 42.
So how do we make sure it does what we actually want?
First of all we need to know what we want as a species. What is society's reward function? This is a productive question even outside the domain of AGI alignment and artifical sentience, but we will need a practical answer.
Elon Musk thinks the answer has to do with understanding the physics of the universe, although his blunt indifference to humanity can be a little affronting. One of the interviewers rightly confronts him about whether we as a species really want to tell our ultra-intelligent progeny to learn more about physics--and then have to hope it either derives some morality along the way or fits us into its entropy optimization.
I don't know the answer. But it seems clear to me that the future of intelligence will have some sort of reward function dictating its behaviour, and that this is the future of instructing machines, or at least nudging them in this case.
No more binary, high level languages or even neural networks. I see temporally dynamic matrix weights, updated by experience, policies and motivations, with some overall long term purpose and instinct rewards pumping rewards through the veins of AGI.
Silicon dopamine and cybernetic oxytocin.