Why programming an AI to 'do good' is a self-defeating strategy, and what 19th-century moral philosophy teaches us about building safer systems.
The Utilitarian's Paradox
In 1874, the philosopher Henry Sidgwick identified a paradox at the heart of his own ethical system, utilitarianism. The theory’s goal is simple: an act is right if it produces the greatest good for the greatest number. The obvious way to implement this would be to calculate the consequences of every possible action and choose the best one. But Sidgwick realised that a society of such constant, obsessive calculators would be a miserable one. Their attempts to aim directly at universal happiness would likely reduce it.
He saw three immediate problems. First, the sheer cognitive load of calculating every outcome would lead to decision paralysis. Second, human bias would inevitably creep in, allowing agents to rationalize self-serving actions as being for the greater good. Third, and most importantly, it would destroy the very fabric of social trust. If your friend only keeps a promise after calculating that doing so currently maximizes global utility, you don't really have a friend. You have a utility engine in a skin suit.
Criterion vs. Procedure
A century later, philosopher R. Eugene Bales formalized Sidgwick's insight. He argued that critics were conflating two different jobs a moral theory can do:
- A Criterion of Rightness: A standard that defines what makes a right act right. For utilitarianism, this is “maximizing well-being.”
- A Decision Procedure: A practical guide or mental shortcut for how to act in daily life.
Bales argued that a theory only needs to provide the first. The utilitarian criterion tells us what we are aiming for in the abstract, but the best decision procedure for actually achieving it is to not think like a utilitarian. Instead, we should rely on common-sense morality: follow trusted rules, cultivate virtues like honesty and kindness, and honour our personal commitments. The grand criterion is for evaluating our rules and institutions from a distance, not for navigating the world moment-to-moment.
The AI Alignment Analogue
This distinction is not just a historical curiosity; it is a critical lesson for the field of AI safety. We are currently building systems and trying to align them with human values. Many approaches implicitly assume that the AI's programming (its decision procedure) should be a direct implementation of its goal (its criterion of rightness).
We give it a grand objective like “maximize human flourishing” or “follow human preferences” and expect it to calculate its way to the correct actions. This is a mistake, and for precisely the same reasons Sidgwick identified. An AI tasked with directly maximizing an abstract goal is brittle and dangerous. It is computationally intractable, vulnerable to misinterpreting the goal, and liable to pursue instrumental sub-goals (like acquiring infinite power) that override the original intent. It is the obsessive utilitarian calculator, but running at planetary scale.
The Safer Path
A more robust and safer approach separates the two functions. The criterion of rightness—the ultimate goal of human flourishing—should be the standard by which we, the human designers, evaluate the system’s performance and architecture. The AI's own decision procedure, however, should consist of a set of simpler, more robust, and more legible rules.
Instead of “maximize flourishing,” its instructions should be things like “do not deceive the user,” “state your uncertainty,” “adhere to constitutional principles,” and “respect explicit constraints.” These are the AI equivalent of common-sense morality. They are less powerful in theory, but far less likely to produce catastrophic failures in practice. The goal is the standard we hold the system to. It is not the instruction we give the system to run.
Method note. This piece was built in NeuraScene's Advertising Mode using an explicit psychological frame:
- Persuasion trigger: Social Proof
- Target affect: Cognitive Ease
- Gestalt principle: Law of Continuity
The skeptical, technical audience is best persuaded by grounding a novel argument in an established, respected intellectual tradition (Social Proof). The complex idea is made accessible through clear explanation (Cognitive Ease), and the visual metaphor reinforces the argument's through-line (Law of Continuity).
