Predictive AI imposes fundamental limits on predictive systems. Automated systems cannot account for the impact of automated decisions; are trained on preexisting datasets generated by one population but applied to different ones; and, when kept low-cost (human-free), often preserve or intensify discrimination -- encouraging people to game the supposedly fair, objective, and accurate system.

The fixes one might immediately suggest -- collecting more data, developing more innovative algorithms, and integrating humans for oversight and accountability -- not only reveal the point of adopting automated systems is moot (conceding these systems cannot make accurate, cost-saving, and bias-free predictions) but likely won't yield significant improvements. "Despite the widespread use of computational predictions in social settings, you might be surprised to learn that there's remarkably little published evidence of their effectiveness," Narayanan and Kapoor write. When it comes to human behavior, there exist too many unknowns: There's a practical limit to how much data we can collect; there is the possibility that exceeding the limit may not be enough; and there is data we may never think to collect or be able to (such as data on the cumulative advantages or disadvantages of granting bail). Since none of this deters companies "from selling AI for making consequential decisions about people by predicting their future," the authors insist we must resist "the AI snake oil that's already in wide use today" instead of pining for better predictive AI technology.

Many of the limitations of generative AI are familiar. Generative AI chatbots simply "predict" the next word in a sequence using methods that require vast computational resources, data, and labor. While they cannot "think" or "understand" language, they can produce "internal representations of the world through their training process" that allow them to "learn" a language's structure without encoding its grammar. Producing a poem, answering factual questions, beating a human at a game -- all these performances are about learning patterns and intuiting rules, then remixing whatever is in the dataset to generate an output. When playing chess or generating a poem, this is relatively straightforward. When answering questions that concern factual claims, we quickly encounter "automated bullshit": Recall for instance Google's AI-generated advice earlier this year to eat one rock per day, or add a serving of glue to pizza.

Narayanan and Kapoor are clear-eyed about some risks generative AI may pose. Automated bullshit is one thing, but they're also concerned about the swell of AI-generated audio, images, and videos (e.g., deepfakes, the vast majority of which are nonconsensual porn). They spend a chapter arguing that "reframing existing risks as AI risks would be a grave mistake," because that framing paints this technology as all-powerful, leads critics to "overstate its capabilities and underemphasize its limitations," empowers firms "who would prefer less scrutiny," and ultimately obscures our ability to identify and root out AI snake oil.