There is tremendous concern that artificial intelligence could eventually become capable of destroying humanity itself. The scenarios vary, but share the same fear: an AI becomes extraordinarily intelligent, pursues an objective that conflicts with human interests, and comes to regard humans as obstacles. It might exploit biological research and gain control of a laboratory to release a deadly virus, or somehow gain access to nuclear weapons. Even if eliminating humanity is not its objective, an AI pursuing some other goal might cripple the internet, disrupt the world’s banking system, or cause catastrophic harm in ways we have not yet imagined.

The concern grows as AI systems become increasingly autonomous. AI “agents” are systems that can independently pursue a goal, perform tasks, make decisions along the way, and take actions. Multiple agents can be arranged to work together, much as people do.