(This post was inspired by listening to Devon Stack’s most recent Insomnia Stream. If you want a layman’s intro to understanding what sorts of things AI can and can’t do, give it a look.)
In China, there are two interesting phenomena:
1. The social credit system
2. AI analyzing physiognomy
I’m also going to assert without proof that humans are incapable of completely ignoring information, even if the only thing they do with it is spend a little cognitive energy suppressing their own knowledge of it. This creates a feedback loop between the social credit system and the AI.
Beginning with a concrete example, let’s imagine the AI can predict pedophilia with 99.999% accuracy (IIRC it’s more like 90%, which is still pretty high). Let’s imagine that the AI has identified someone you know as almost certainly a pedophile. You’re going to act differently toward that person. Whether it’s true or not in that case, the behavior is adaptive. People who are inclined to ignore it get winnowed out of the population. This trains humanity over time to rely on the tool more when making policy decisions.
The subtle point here is this poisons the input data by creating a loop of circular reasoning. Maybe the person you know was the 1/100,000 who looks like a pedophile but isn’t one, but they were locked up for pedophilia because humanity has been trained by selection effects and behavioral conditioning to rely on the useful tool. The predictive power of the AI, as far as the AI knows, is now 100%. Because the AI isn’t really predicting pedophilia, it’s predicting whose photo looks like a mugshot photo of someone in jail for pedophilia. Your friend who isn’t a pedophile now has such a mugshot, despite not being a pedophile. Human behavior, modified by exposure to the AI’s predictions, has reduced the validity of those predictions (making them less useful). Once you feed the two things in China into each other, the social credit system and the AI physiognomy machine, this creates a game which will seek a Nash equilibrium. This demonstrates the tradeoff in the title.
So why shouldn’t that equilibrium be 50% predictive power and 50% usefulness? The prediction in the parentheses of the title is based on an idea that’s analogous to white flight, and it’s predicated on the idea that moral cowardice is adaptive at the individual level. Rather than population-wide soul-searching, I expect an arms race, comparable to white flight, where people begin relying on AIs to tell them which AIs are reliable. I’m going to assert something else without proof: Self-fulfilling AI prophecies will be more common than self-defeating prophecies. This is because people are more likely to be hypocrites (knowing information but saying otherwise) than fanatical (knowing information but acting otherwise). The only limit on this behavior is that the usefulness of tools limits the technological complexity of society, so each AI will be used and abused until its data inputs and outputs become 100% circular or the people using it are in enough economic distress to cast it aside and seek a newer, more useful one.
Here’s a fun exercise for you simulation theory types: prove this hasn’t already happened.
Aeoli Pera 