Why programming an AI with a 'correct' moral theory can be a catastrophic mistake, and what 20th-century philosophy tells us about doing it right.
The Naive Mandate
How do we build an ethical AI? The common-sense answer seems straightforward: identify the correct moral theory and program the machine to follow it. If Act-Utilitarianism is true—that the right act is the one that maximizes overall utility—then an AI’s prime directive should be to calculate and execute the utility-maximizing action in any given scenario. Simple.
Except, as moral philosophers have known for decades, it’s not. This naive approach commits a category error, one that would turn a powerful AI into a dangerous, paralysed, or self-defeating machine.
The Self-Defeating Objection
The problem was first hammered out in debates over Act-Utilitarianism (AU) in the mid-20th century. Critics argued that if a person tried to live their life by constantly calculating the utility of every possible action, they would become paralysed. They'd miss opportunities, break promises that rely on trust over calculation, and spend all their time computing instead of acting. Adopting AU as a direct guide for moment-to-moment decisions would, ironically, produce far less utility than just following common-sense moral rules. Therefore, the theory was judged to be practically self-defeating.
The defence against this came in a sharp, clarifying paper by R. Eugene Bales in 1971. Bales argued that critics were conflating two distinct functions of a moral theory:
- A Criterion of Rightness: A standard that defines which features make an action morally right. For AU, this is the property of maximizing utility. It’s the final court of appeal for what is, objectively, the correct thing to do.
- A Decision-Making Procedure: A practical method, heuristic, or set of rules that an agent uses to figure out what to do.
Bales’s crucial insight was that a theory can provide the first without claiming to be the second. AU can be the correct account of what makes acts right, while simultaneously endorsing a completely different, non-utilitarian set of rules (like “keep your promises” or “don’t lie”) as the best decision procedure for fallible, cognitively-limited humans to follow. If following those rules empirically leads to the best consequences, then the Act-Utilitarian framework itself demands you use them as your guide, rather than calculating utility from scratch every time.
The Sophisticated AI
This distinction is not just an academic footnote; it is a critical insight for designing safe and effective AI. To build a 'utilitarian' AI by commanding it to `maximize(utility)` as its direct decision procedure is to repeat the same category error. A truly sophisticated consequentialist AI would not be a raw calculator. Its designers would engineer a decision-making procedure that produces the best results in the real world.
That procedure might look more deontological (a set of inviolable rules) or virtue-ethical (a set of cultivated dispositions) because those architectures are more robust, predictable, and trustworthy in a complex environment than a brittle, all-knowing calculator.
The AI's operational code—its day-to-day 'morality'—might be to follow rules, respect rights, and act with integrity. Not because those things are ends in themselves, but because they are the most reliable known method for achieving its ultimate criterion of rightness. The goal isn’t to make the AI deliberate like a utilitarian philosopher, but to make it act in a way that a utilitarian would, in the final analysis, approve of.
The challenge of AI alignment, then, is not just about specifying the right goal. It’s a far more complex engineering problem: designing the right relationship between an ultimate criterion of value and the practical, everyday procedure the system uses to navigate the world.
Method note. This piece was built in NeuraScene's Advertising Mode using an explicit psychological frame:
- Persuasion trigger: Anchoring
- Target affect: Sublime Terror
- Gestalt principle: Figure-Ground Articulation
The audience is philosophically and technically literate. 'Anchoring' on a familiar problem provides a solid entry point. 'Sublime Terror' matches the gravity of AI ethics and the 'Creature Feature' visual. 'Figure-Ground Articulation' offers a precise perceptual metaphor for the core philosophical argument, creating an intellectually coherent package.
