This Visionary Sci-Fi Author Sees the Destruction of Human Civilization: Predatory Capitalism

Chiang is the acclaimed author of "Arrival" (Stories of Your Life and Others). He explains the threat of AI to humanity by riffing on an Elon Musk interview:

In a recent interview with Vanity Fair, Musk imagines a mechanized strawberry picker wiping out the species simply as a means of maximizing its production.

Long story short at Alternet:

https://www.alternet.org/culture/visionary-sci-fi-author-sees-destructio...

Now I'm switching to the full story at Buzzfeed from the Alternet link:

Speaking to Maureen Dowd for a Vanity Fair article published in April, Musk gave an example of an artificial intelligence that’s given the task of picking strawberries. It seems harmless enough, but as the AI redesigns itself to be more effective, it might decide that the best way to maximize its output would be to destroy civilization and convert the entire surface of the Earth into strawberry fields. Thus, in its pursuit of a seemingly innocuous goal, an AI could bring about the extinction of humanity purely as an unintended side effect.

And why AI lacks judgment:

In psychology, the term “insight” is used to describe a recognition of one’s own condition, such as when a person with mental illness is aware of their illness. More broadly, it describes the ability to recognize patterns in one’s own behavior. It’s an example of metacognition, or thinking about one’s own thinking, and it’s something most humans are capable of but animals are not. And I believe the best test of whether an AI is really engaging in human-level cognition would be for it to demonstrate insight of this kind.

I question whether "most humans" are capable of metacognition. Socrates made the observation that "an unexamined life is not worth living", he didn't say it was easy. If introspection was easy there wouldn't be so many people in therapy or so many who still support Hillary.

Chiang provides us with this insight:

Then I realized that we are already surrounded by machines that demonstrate a complete lack of insight, we just call them corporations. Corporations don’t operate autonomously, of course, and the humans in charge of them are presumably capable of insight, but capitalism doesn’t reward them for using it. On the contrary, capitalism actively erodes this capacity in people by demanding that they replace their own judgment of what “good” means with “whatever the market decides.”

Corporations are designed to operate precisely like Musk's automated strawberry picker:

Just as a superintelligent AI ought to realize that covering the planet in strawberry fields isn’t actually in its or anyone else’s best interests, companies in Silicon Valley need to realize that increasing market share isn’t a good reason to ignore all other considerations. Individuals often reevaluate their priorities after experiencing a personal wake-up call. What we need is for companies to do the same — not to abandon capitalism completely, just to rethink the way they practice it.

https://www.buzzfeed.com/tedchiang/the-real-danger-to-civilization-isnt-...

I can only surmise that Chiang limited his critique to Silicon Valley because he naively believes high tech executives will be more receptive to his suggestion than Wall Street financiers and Big Pharma. Actually, the threat of malignant AI is more easily prevented than the threat of malignant corporations due to Asimov's Three Laws of Robotics:

1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.

2. A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.

3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws.[1]

Near the end of his book Foundation and Earth, a zeroth law was introduced:

0. A robot may not injure humanity, or, by inaction, allow humanity to come to harm.