Reinforcement Learning with Calibrated Decisions (RLCD)

The Problem

Standard classification models are typically trained using cross-entropy loss. While this works reasonably well, these models often become notoriously overconfident. If an AI router is 99% confident it should reset a user's password, but it is actually wrong 30% of the time, the automated system will cause catastrophic damage. To build a trustable "System 1" router that can autonomously bypass human review, the model must be perfectly calibrated: when it says it is 90% confident, it must be right exactly 90% of the time.

The Idea

RLCD proposes that instead of just minimizing classification error via standard supervised fine-tuning, we should treat the model as a reinforcement learning agent. The reward signal for this agent isn't just "did you get the label right?". Instead, the reward is strictly tied to a Proper Scoring Rule (like the Brier Score). The agent is mathematically forced to output its true, calibrated confidence, because expressing false confidence results in a massive negative reward penalty.

How It Works

The training process mirrors RLHF, but adapted for non-generative, typed outputs:

1. The Distillation Phase (Supervised) A massive, slow "System 2" LLM (like GPT-4) generates synthetic labels and rationales for a massive dataset of user states. The System 1 encoder (like mmBERT) is initially fine-tuned on this data to learn the general domain.

2. The Reward Function A reward function is defined using a strictly proper scoring rule. For a binary choice (e.g., "Is this toxic?"), the model outputs a probability $p$. If the true label is 1, the reward is calculated based on how close $p$ was to 1, with an exponentially steeper penalty for predicting 0 with high confidence.

3. Proximal Policy Optimization (PPO) The decision model is then trained using PPO. It explores different probability outputs for given inputs. If it tries to "game" the system by always outputting 100% or 0%, the proper scoring rule aggressively penalizes it when it is inevitably wrong. The policy gradient forces the model's weights to settle exactly at the true probability distribution of the data.

Why It Mattered

RLCD unlocked the ability to deploy AI routers without a human in the loop. By guaranteeing calibration, engineering teams can safely set hard thresholds (e.g., if confidence > 0.95, execute code; else route to human). It proved that reinforcement learning isn't just for making LLMs chatty—it is the mathematically optimal way to enforce honest probabilities.

What Came After

RLCD is the core training methodology behind Convai's Laya and TypeSafe's Jev. It defined the standard for the emerging class of "System 1" models, separating them from standard BERT classifiers by guaranteeing production-grade calibration out of the box.