Q205 - Over-reliance_definition_example_and_why_it_matters_in_RAIDT
Q205 — Over-reliance — definition, example, and why it matters in RAIDT
← RAIDT · Star S2 - Governance Meaning and Problem Context · primary item: S2.10 · GenAI failure modes
B. Background & Problem | Ordered by mind-map priority: inner circles first, then operational detail.
Appears in sources
workshop_dense_100#slide 28
Answer
Over-reliance in RAIDT is the uncritical acceptance or routine downstream use of GenAI outputs beyond what the available evidence and uncertainty warrant. The listed papers treat this as a socio-technical risk rather than a purely psychological one. The managerial decision-making paper links it to overconfidence in opaque outputs, poor trust calibration and the need to communicate uncertainty explicitly. The evidence-review paper shows that organisational harms often arise through workflow integration, staff incentives and the routinisation of outputs. The human-AI collaboration discussion further shows that hybrid systems need clear role boundaries, override capability and retained accountability; otherwise assemblage can drift into rubber-stamping.
This matters in RAIDT because over-reliance weakens Responsibility even when other controls appear to exist. A human reviewer may be present, yet if the run-level evidence pack shows little more than a rapid approval of a confident output, the governance quality of the run is still low. Over-reliance also interacts with the other pillars: weak Interpretability makes challenge harder, weak Traceability obscures the evidential basis of the recommendation, and weak Dependability means similar tasks may receive varying outputs that staff nonetheless accept as authoritative. The score profile is useful here because a run should not be treated as 5=audit-ready merely because a person approved it; where challenge, uncertainty handling or escalation evidence is thin, the case is closer to 3=partial or even 1=missing from a governance standpoint.
Practical example
Imagine a public-service style resource-allocation exercise in which a manager uses a GenAI assistant to synthesise conflicting forecasts and recommend where limited funds should go first. The output is well written and presented with apparent certainty, so the manager accepts it quickly and circulates it as the basis for a meeting. No one checks how the recommendation was grounded, whether retrieved sources conflicted, or whether the system signalled uncertainty.
That is over-reliance. The risk is not removed by the manager's participation, because the human role was procedural rather than critical. In a RAIDT workflow, reviewers would expect the run-level evidence pack to show source review, uncertainty interpretation, edits or escalation actions. If that evidence is absent, the score profile should expose that the run is not genuinely governance-ready.
Sources in RAIDT papers
01-Responsible_AI_for_Managerial_Decision-Making_Under_Uncertainty-V313-RAIDT-Evidence-Review_M_v1015-RAIDT-IS-Governance_M_v07