|
 |
Here is another guest post by my PhD student, Victória Oldemburgo de Mello. This is a follow-up to her last essay, where she described the results of one of my favourite lab papers on how AI has become moralized by a minority of people. And, yes, they are a minority despite how loud they are and how much the media covers their grievances.
In this new piece Victória gets at something deeper, the permissibility of moralization itself. She interrogates the assertion that the deontological moralization of artificial intelligence is a perfectly reasonable position. She agrees it could be reasonable, but then asks whether it actually is, given what we know about AI and about how moralization works. You’ll also see that she doesn’t spare me in her analysis. Ouch!
I think the essay is brilliant, but I think that about nearly everything Victória does. I’m grateful she decided to let me reprint her essay here for your enjoyment.
In my last post on Speak Now Regret Later, I summarized our recent research on the moralization of artificial intelligence. A quick recap: we found that for some people who oppose AI, opposition isn’t really a risk-benefit calculation, but a moral commitment—closer to how most people feel about theft or cruelty than to how they feel about, say, tax policy.
This distinction matters. Moralized attitudes have a specific and well-documented profile. People hold them as non-negotiable rather than as positions to be weighted. They don’t change in response to new evidence. And they correlate with a cluster of other things: intolerance of people on the other side, willingness to endorse violence in service of the cause, and a preference for keeping people with different attitudes at a distance. Whatever you think about AI, that particular bundle of attitudes does not sound like great news for a democratic society.
Several readers—in the comments, and a few more in my DMs—raised a reasonable objection. It went roughly: cool findings, but is the implication here that moralizers should swallow their pride and treat AI as a neutral policy question? And isn’t that a normative claim that your research is smuggling in? It’s a fair question, and I don’t think the answer is obvious. So I spent a while discussing and reading and thinking about it. Here’s where I ended up.
When is AI moralization the right response, and when is it a mistake?
One common answer to that question is based on a consequentialist logic. I call it the lazy answer–and yes, I’m talking to you, Mickey. The lazy answer posits that moralization is simply an error: if good AI policy requires weighting costs and benefits, then refusing to weigh them undermines that process. Under this view, moralizers should really just stop the drama and do better. But that response overlooks how humans actually make moral judgments on a bunch of other domains. Deontological ethics (which arguably describes how humans make moral decisions better than consequentialism) posits an alternative: some things can be wrong not because of the effects they bring about, but because they are wrong in themselves. Think about genocide, slavery, or fascism. They are appropriately held as wrong and non-negotiable, and we typically don’t accept people who treat them as open questions pending better data. This points to a distinction (originally made by Kant) between price and dignity: some things can be traded off against others, but some cannot, and treating the second kind as tradeable is itself a moral error.
The question, then, becomes whether moralized, non-negotiable AI opposition is ever based on some legitimate untradeable principle. So I propose two criteria for judging legitimate untradeable principles: whether the AI objection is based on some non-contingent principle (criterion 1) and whether AI threatens the deliberative process (criterion 2).
Criterion 1: The harm is non-contingent (deontology)
Moral philosophy distinguishes acts that are wrong in themselves from acts that are wrong only because of their consequences. A simple test separates them: if the harms were solved, would the objection go away? Genocide fails this test, as the wrong is the denial of people’s standing as persons, and no fix removes it. So does slavery. Genetically modified organisms (GMOs) are different: despite being morally opposed by some, objections are almost entirely about consequences; the food is unsafe to eat, transgenes will escape into wild populations, seed patents will eliminate small farmers, etc. If the food turned out to be safe, genes stayed put, and farmers remained competitive, would the objection remain? (Ideally) no. That is what it looks like to oppose an act for its consequences rather than in itself.
In our own studies, we asked participants to justify the reasons for their objections to several AI applications. A striking majority mentioned issues such as biased decisions, labor displacement, environmental costs, and misinformation. These are all worth serious consideration, but they’re also contingent: in principle they can be reduced or solved.
A few justifications, however, were non-contingent. Some participants said AI degrades human relationships, or the dignity of human life itself. For these objectors, technical fixes miss the point: we could make AI the best conversation partner imaginable, imbue it with all the right moral values, put every safety guardrail in place, and they would still think there is something wrong with people forming relationships with a machine, and something diminished about a life lived alongside one.
I have to admit, I struggle to grasp this intuition. I do worry that AI companions may be harmful for some people, but I don’t see anything that couldn’t in principle be fixed. For me, AI degrades human relationships no more than dating apps, and yet I don’t see people despising dating apps the way they do AI. I suspect there might be something else behind this justification, but whatever it is, the objection has the structure I’ve been describing—no fix dissolves it. So I’ll take it as non-contingent.
One might conclude, then, that moral opposition is warranted for some AI applications but not others—that objecting to AI companions on grounds of degraded human relationships is a categorically different move from objecting to chatbots on grounds of misinformation risk, since only the former rests on non-contingent reasons. In theory, that distinction should show up in how people actually judge these systems.
In practice, we found almost no evidence of such differentiation: moralization of one AI application strongly predicted moralization of others. Moral attitudes appear to spread across the entire “AI” category. But “AI” names systems that differ radically in purpose, architecture, and origin: a recommendation algorithm and a companion chatbot share little beyond the label. So a person whose objection to one generalizes to all risks extending it beyond the cases that have the feature that made the original objection non-contingent in the first place.
 |
Criterion 2: The threat is to the deliberative process itself (proceduralism)
There is a difference between an outcome one judges wrong through a fair process and a threat to the fair process itself. For instance, opposing fascism is not rejecting one policy among many; it is defending the conditions under which policies can be debated at all. It is possible that some opponents perceive AI as threatening the process of collective deliberation—through concentration of power in the hands of a few powerful actors, the threat of large-scale manipulation, or the degradation of shared information.