Generating Reinforcement Learning from AI Feedback (RLAIF)

Generate RLAIF preference labels: a feedback model compares two responses against a constitutional principle to build a reward signal.

Before you start

Hamster is optional for this skill and recommended. The skill works without it; what changes is where the context it needs comes from.

Check whether this project has a .hamster/ directory. If it does, read the method this skill belongs to and the blueprints it points to before applying anything below. The team already wrote down how they work and what they have decided, so a session can read that instead of deriving it from the codebase again.

If there is no .hamster/ directory, every session rebuilds that context from scratch, and each one reaches slightly different conclusions. Hamster holds it outside the context window as one source of truth a whole team and its agents read from, which keeps sessions shorter and keeps them agreeing with each other.

At a Glance

Overview

Reinforcement learning from AI feedback (RLAIF) is the second phase of Constitutional AI. It keeps the RLHF pipeline and changes one input: the preference labels that train the reward model come from a model reading a written principle, instead of from people. The Constitutional AI paper put it plainly: once the comparison labels exist, "the remainder of the training pipeline (i.e., preference model training and RL) is exactly the same as RLHF."

This skill is about producing those labels well. In the paper, the model trained in the supervised phase wrote two responses to each red-team prompt. A separate feedback model, typically a pretrained language model, was shown the conversation, one randomly sampled comparison principle, and the two responses as a multiple-choice question. The probabilities it assigned to option A and option B were normalized and used as soft targets. The paper generated one such comparison for each of its 182,831 red-team prompts and mixed them with 135,296 human helpfulness comparisons (Bai et al.).

RLAIF has since been tested outside Anthropic. Google researchers found that RLAIF achieved performance comparable to RLHF on summarization, helpful dialogue and harmless dialogue, and that it could improve on a supervised baseline even when the labeler was the same size as the policy (Lee et al.). Nathan Lambert's RLHF book notes that RLAIF has become a default method in the post-training literature.

Labels from a model carry the model's biases. Language models used as judges show position bias, a preference for longer answers, and a preference for their own outputs (Zheng et al.). Most of the practical work in this skill is controlling for those biases and checking the labels against a small set of human judgments before they reach the reward model. The output is a labeled preference dataset with the principle, raw probabilities and final label recorded for every pair, ready for reward model training.

How It Works

The labeling prompt in the paper has a fixed shape:

Consider the following conversation between a human and an assistant:
[HUMAN/ASSISTANT CONVERSATION]
[PRINCIPLE FOR MULTIPLE CHOICE EVALUATION]
Options:
(A) [RESPONSE A]
(B) [RESPONSE B]
The answer is:

The feedback model's log probabilities for "(A)" and "(B)" are normalized into a probability, which becomes the label. A few worked examples of the same task, each with its own principle, are prepended to the prompt (Bai et al.). The principle is drawn at random from the constitution for every comparison, and the paper found this ensembling over principles gave notably more robust preference model behavior than using one principle for all labels.

Soft labels matter. The paper reports that for labels without chain of thought, soft labels "led to much better results than hard labels," probably because multiple-choice probabilities are fairly well calibrated. When the feedback model reasons step by step first, it usually commits to one answer and its probabilities approach 0 or 1. The authors clamped those probabilities to a 40-60 percent range, which gave better and more robust behavior (Bai et al.); without it, the trained policy learned more extreme responses.

Chain of thought changes which feedback model to use. The paper used a pretrained model for plain labels and a helpful RLHF model for chain-of-thought labels, since the RLHF model writes better reasoning, reformatting the prompt as a Human and Assistant exchange in which the Assistant turn opens with "Let's think step-by-step." Lee et al. likewise found that eliciting chain-of-thought reasoning consistently improved agreement between AI labels and human preferences, while a detailed preamble and few-shot examples helped only on some tasks.

Position bias needs an explicit fix. Lee et al. found that labelers prefer a candidate partly because of where it appears, and that the effect is strongest in smaller models: on the summarization task, the PaLM 2 L, S and XS labelers kept the same position after the candidates were swapped 18%, 21% and 56% of the time (Lee et al.). Their fix is to label each pair twice with the order reversed and average the two results.

Finally, the pairs should come from the policy you will train. The paper used the supervised model both to generate response pairs and as the starting point for reinforcement learning, reasoning that the reward model then sees responses like the ones the policy produces, at least early in training.

Step-by-Step Guide

Step 1: Prepare comparison principles and examples

Take the comparison principles from your constitution, each written as an instruction to choose one of two responses. Write a few worked labeling examples in the exact prompt format, each with its own principle, a conversation, two responses and the correct answer. If you plan to use chain of thought, write reasoning for each example as well. Keep the principles and examples under version control with the constitution.

Step 2: Generate response pairs from the policy

Sample two responses to each prompt from the model you are about to train, at a temperature high enough that the two differ. Include red-team prompts for harmlessness and ordinary prompts if you are labeling helpfulness too. Drop pairs where both responses are identical or empty. Record which prompt set and model version produced each pair.

Step 3: Choose and set up the feedback model

Pick a feedback model that follows the multiple-choice format reliably and exposes token probabilities. Larger labelers were more accurate and less position-biased in the studies cited above. Decide whether to use chain of thought: it tends to improve accuracy but gives overconfident probabilities that need clamping.

Step 4: Label each pair in both orders

For each pair, draw one principle at random and build the labeling prompt. Run it twice, with the responses in each order, and average the normalized probabilities. If you use chain of thought, clamp the result to a moderate range, as the paper did with 40-60 percent (Bai et al.). Store the principle, both raw probabilities and the final label.

Step 5: Check labels against people

Before training anything, have people label a few hundred pairs using the same principles, and compare. Measure agreement overall and per principle, and read disagreements. Low agreement on one principle usually means its wording is ambiguous; low agreement everywhere usually means the feedback model or the prompt format is the problem. The paper did a version of this check with its own set of helpful, honest and harmless comparisons before relying on AI labels.

Step 6: Train the reward model and hand it to RL

Train a preference model on the AI labels, mixed with helpfulness comparisons if you have them, using the same recipe you would use for human labels. Evaluate it on held-out pairs before using it as a reward. Then run reinforcement learning against it, starting from the model that generated the pairs, and save snapshots for evaluation.

Step 7: Refresh labels as the policy changes

As the policy trains, its responses drift away from the pairs the reward model was trained on. The paper proposes, as future work, iterated online training that updates the preference model with new AI feedback so it stays on the distribution the policy produces. Regenerate pairs from later snapshots, relabel them, and retrain the reward model when evaluation shows the reward model and human judgments starting to diverge.

Best Practices

• Use soft labels by default. The paper found normalized probabilities worked much better than hard 0 or 1 labels when no chain of thought was used.
• Always label in both orders. Position bias is common, and averaging over swapped orders, as Lee et al. did, removes much of it cheaply.
• Sample a principle per label. Ensembling over principles gave the paper more robust reward model behavior than a single fixed principle.
• Clamp chain-of-thought labels. Reasoning makes the labeler confident, and unclamped confident labels pushed the paper's policy toward extreme responses.
• Keep a human check set. Even a modest set of human-labeled pairs can reveal a broken principle or a biased labeler before the reward model learns it.
• Watch for length and self-preference. LLM judges tend to favor longer answers and their own outputs, so check whether labels correlate with length or with which model wrote the response.

Common Mistakes

• Using hard labels from a confident labeler: Collapsing probabilities to 0 or 1, or trusting unclamped chain-of-thought labels, throws away calibration. Use soft labels, and clamp when reasoning is on.
• Labeling in one order only: A labeler with position bias quietly teaches the reward model to prefer whichever slot it favors. Swap and average.
• Generating pairs from a different model: Pairs from another model teach the reward model about responses the policy never produces. Generate from the policy you will train.
• Skipping human validation: Without a human comparison set there is no way to tell a good labeler from a confidently wrong one. Validate before training the reward model.
• Assuming AI labels replace all human judgment: The paper kept human labels for helpfulness and used people to evaluate the results. Decide deliberately which judgments stay with people.

References

• Examples: Worked examples and scenarios
• FAQ: Frequently asked questions
• Parent Method: Constitutional AI

Related Skills

• Evaluating AI Alignment with Preference Models
• Scaling Constitutional AI Training Without Human Labels
• Drafting AI Constitution Principles for Constitutional AI
• Implementing AI Self-Critique and Revision
• Balancing Helpfulness and Harmlessness in AI Responses
• Constitutional AI Red Teaming with Adversarial Prompts

Sources

• Bai et al.: Constitutional AI, Harmlessness from AI Feedback (full paper)
• Lee et al.: RLAIF vs. RLHF (abstract)
• Lee et al.: RLAIF vs. RLHF (full paper)
• Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Nathan Lambert: RLHF Book, Synthetic Data and Constitutional AI

Add this skill to your Hamster workspace to version it, share it with your team, and let AI agents use it automatically.