“People use laws, social norms and international agreements to reap the benefits of technology while minimizing undesirable things like environmental damage. In aiming to find such rules of behavior, we often take inspiration from what game theorists call a Nash equilibrium, named after the mathematician and economist John Nash. In game theory, a Nash equilibrium is a set of strategies that, once discovered by a set of players, provides a stable fixed point at which no one has an incentive to depart from their current strategy.

To reach such an equilibrium, the players need to understand the consequences of their own and others' potential actions. During the Cold War, for example, peace among nuclear powers depended on the understanding the any attack would ensure everyone's destruction. Similarly, from local regulations to international law, negotiations can be seen as a gradual exploration of all possible moves to find a stable framework of rules acceptable to everyone, and giving no one an incentive to cheat – because doing so would leave them worse off.

But what if technology becomes so complex and starts evolving so rapidly that humans can’t imagine the consequences of some new action? This is the question that a pair of scientists — Dimitri Kusnezov of the National Nuclear Security Administration and Wendell Jones, recently retired from Sandia National Labs — explore in a recent paper. Their unsettling conclusion: The concept of strategic equilibrium as an organizing principle may be nearly obsolete.”

How Technology Might Get Out of Control
https://www.bloomberg.com/view/articles/2017-08-15/how-technology-might-get-out-of-control
via Instapaper