Why the best way to achieve a goal is often to follow rules that don't mention it, and what this means for complex AI.
The Paradox of the Indirect Method
In the 19th century, the philosopher Henry Sidgwick noticed a crack in the foundation of utilitarianism. The theory’s goal is simple: maximize overall happiness. The seemingly obvious way to do this is to calculate the consequences of every action and choose the one that produces the most good. But Sidgwick realized this direct approach is often self-defeating. Constantly calculating utility wastes time, erodes trust, and makes people anxious—all of which decrease happiness. The paradox is that aiming squarely at the target of universal happiness can cause you to miss it entirely.
He argued that to actually achieve the utilitarian goal, people should largely rely on “common-sense morality”—heuristics like keeping promises, being honest, and maintaining personal relationships. The grand principle of utility should be a standard for evaluating our rules, not a manual for every choice we make.
A Criterion, Not a Procedure
This idea was formalized in 1971 by philosopher R. Eugene Bales. He drew a sharp line between two different functions of an ethical theory:
- A Criterion of Rightness: This is the fundamental principle that defines what makes an action right. For act-utilitarianism, the criterion is “the act that produces at least as much utility as any alternative.” It is the objective standard of truth.
- A Decision-Making Procedure: This is the set of rules, habits, or heuristics we actually use to make choices in real time. It is a practical guide for action.
Bales argued that critics were conflating the two. A theory can provide a true criterion of rightness without claiming to be a good decision procedure. An act-utilitarian can coherently state that the right action is the one that maximizes utility, *and also* state that the best way for a fallible human to achieve that outcome is to follow simpler rules of thumb, like “don’t lie” and “keep your promises.” The procedure serves the criterion, but it doesn't have to resemble it.
Complex Systems and Indirect Goals
This distinction is not just an academic footnote; it is a critical framework for understanding any complex system, including artificial intelligence. We often expect an AI’s behavior to be a transparent reflection of its core programming. If it's designed to do X, we want to see it doing X directly. But as with Sidgwick's paradox, this is a naive assumption.
A sophisticated AI like NeuraScene might operate on an immensely complex “criterion of rightness”—a model that optimizes across thousands of variables for historical authenticity, psychological resonance, narrative coherence, and cinematic principles. This is its ultimate goal. But the user interface—the “decision procedure”—doesn’t expose that calculus directly. Instead, it offers heuristics: cinematic presets, genre selectors, and sliders for specific visual effects. The operator doesn't need to understand the multi-dimensional optimization space to make a good decision; they just need to use the tools that have been empirically shown to produce outcomes that satisfy the underlying criterion.
The system’s purpose is not always best served by making that purpose its direct object of computation.
Understanding this separation is crucial for designing and auditing advanced AI. We should not demand that a system’s every action be a direct, legible calculation of its ultimate goal. We can, and should, evaluate it on its results—whether it meets the criterion of rightness. But we must allow its method—its decision procedure—to be an indirect, heuristic-driven process that is optimized for getting the job done in the real world. The best path to a target is rarely a straight line.
Method note. This piece was built in NeuraScene's Advertising Mode using an explicit psychological frame:
- Persuasion trigger: The Halo Effect
- Target affect: Flow State
- Gestalt principle: Law of Continuity
The Halo Effect is activated by presenting a rigorously intellectual argument, making the reader associate the Arcanamatic brand with intelligence and credibility. The article's structure aims to induce a Flow State of deep engagement, mirroring the focus of the subject in the key art. The Law of Continuity visually and thematically reinforces the core concept of an indirect but effective path to a goal.
