• integrated_82#Q3.23

In RAIDT, RLHF/DPO-type controls are preference-based controls that sit in the influence stack alongside prompting, PEFT/LoRA, and RAG. RLHF uses pairwise human judgements to train a reward signal and then optimise the policy towards preferred behaviour; DPO learns directly from preferred versus dispreferred continuations without a separate reward model. The papers therefore treat them as behavioural steering mechanisms rather than knowledge-grounding mechanisms. Their strongest contribution is to Responsibility: they can produce safer tone, better refusal behaviour, clearer escalation cues, and stronger red-flag surfacing in domains such as healthcare, finance, and public policy. They can also improve reviewer-perceived Interpretability when the aligned policy produces more structured rationales.

The same papers are equally clear about the downside. Because preference learning changes behaviour through hidden selection and optimisation choices, it can open audit gaps when reward provenance is weak. If annotator pools, rubric definitions, inter-rater reliability, policy versions, preference datasets, or reward-model assumptions are not logged, the organisation cannot reconstruct why a model preferred one answer over another. DPO simplifies the lineage by removing the reward model, but it does not remove governance risk; it shifts more evidential weight onto the pairwise dataset and annotation protocol. In RAIDT terms, RLHF/DPO often lift Responsibility faster than Auditability or Traceability, so their score profile is asymmetric unless supported by logging, hashing, reviewer forms, and documented preference governance.

In finance, a lender might add DPO to an adverse-action letter generator so that the model prefers explanations that are respectful, regulation-aware, and explicit about uncertainty, while avoiding speculative or inflammatory wording. Reviewers compare better and worse draft letters and record which version better follows the house rubric for fairness caveats and explanation quality.

This improves Responsibility because the final letters are more careful and less likely to mislead or stigmatise applicants. However, an audit gap appears immediately if the bank cannot show the labelled pairs, the reviewer guidance, and the version of the model trained on those pairs. In a complaint or regulator review, it would then be hard to explain why a particular wording pattern emerged, even though the letters look more responsible on the surface.

• 07-RAIDT_RLHF_V1
• 04-RAIDT_Prompt_Eng_V2