S11.07 - Evidence_capture_feasibility

S11.07 ? Evidence capture feasibility

flowchart LR
    A[Closed platforms and missing metadata] --> B[RAIDT
run-level evidence framework]
    H[Prompts, timestamps, outputs,
review notes, wrappers, logs] --> C[[Evidence capture feasibility
can the run be reconstructed?]]
    B --> C
    C --> D[Evidence pack]
    C --> E[RAIDT score profile]
    D --> F[Reviewer reconstruction
and contestability]
    E --> G[Governance readiness
and organisational learning]
    I[Procurement and implementation choices] --> C

? Star S11 - Boundaries, Limitations and Future Questions

Star context: Clarifies a practical boundary of RAIDT by showing that governance quality depends partly on whether a platform, workflow, or organisational setting can actually produce the evidence needed for run-level review.

Academic picture

Definition / background

Evidence capture feasibility is the practical question of whether enough relevant evidence can be recorded for a particular generative AI run to be reconstructed, reviewed, and evaluated. In RAIDT, the issue is not whether an organisation would like better documentation in principle, but whether the technical platform, workflow design, and organisational controls make evidence capture possible at the level of the individual run.

The concept matters because RAIDT treats the run as the unit of governance. A run-level evidence pack and a five-pillar score profile depend on the existence of a usable evidential record. If prompts, settings, source materials, outputs, review actions, or timestamps cannot be captured reliably, then the organisation cannot fully justify its governance claims for that run. In that sense, evidence capture feasibility is a condition of governance visibility.

This concept is different from general logging, transparency, or documentation quality. Logging may exist but still be infeasible for governance purposes if it omits contextual details, human interventions, or output versions. Transparency may be claimed at the vendor or policy level without giving an organisation access to the artefacts needed to reconstruct one concrete run. Evidence capture feasibility therefore sits between infrastructure capability and governance method: it asks whether the environment can support RAIDT's evidential demands.

Within RAIDT, the concept belongs in Boundaries, Limitations and Future Questions because it prevents overclaiming. RAIDT does not solve missing evidence by rhetorical force. If a platform does not expose sufficient metadata, if review steps occur outside the system, or if implementation is weak, RAIDT makes that limitation visible. The framework remains useful precisely because it can show when low Auditability or Traceability scores reflect a real evidence gap rather than a failure of interpretation.

Why this concept matters

Evidence capture feasibility matters because many organisations adopt generative AI tools whose evidential affordances are uneven, opaque, or poorly aligned with governance requirements. A governance framework that ignores this issue risks assuming that evidence can always be produced after the fact. In practice, many disputes arise only once a problematic output, contested decision, or review request forces the organisation to discover what was never captured.

The concept also prevents a common confusion between governance design and governance executability. An organisation may have a strong policy, a clear responsible-use statement, and a well-written assurance narrative, yet still be unable to reconstruct a run because its toolchain does not retain prompts, versioned outputs, or reviewer actions. RAIDT uses evidence capture feasibility to separate aspirational governance from operationally supportable governance.

If this concept is missing, organisations may overestimate their audit readiness, underestimate procurement risk, and misinterpret weak evidence as a minor documentation inconvenience rather than a structural limitation. By foregrounding feasibility, RAIDT helps move governance from principle statements to realistic operational judgement.

Key idea: Evidence capture feasibility matters because RAIDT can govern only what an organisation can meaningfully evidence at the level of the individual run.

What this item explains

Whether a given platform or workflow can capture the artefacts needed for run-level governance.
Why missing metadata is not merely a technical inconvenience but a governance limitation.
How weak evidence capture constrains the quality of the evidence pack and the defensibility of the score profile.
Why low scores on Auditability or Traceability may reveal infrastructure or procurement problems rather than reviewer weakness.
Which parts of a run are most at risk of becoming invisible, such as prompt history, context, tool settings, output revisions, or human review actions.
How organisations can distinguish between what is theoretically desirable to record and what is operationally feasible to capture.
Why feasibility should influence implementation planning, vendor selection, workflow design, and proportional governance expectations.

Practical example / likely audience question

Audience question

What if the platform cannot log everything?

Answer

The concern behind this question is that RAIDT might appear to assume ideal technical visibility. The direct answer is no: RAIDT does not require perfection, but it does require that evidence limitations be made explicit rather than hidden. If a platform cannot log everything, the missing evidence becomes a governance fact about that implementation environment.

For example, an organisation may use a vendor chatbot that stores final outputs but does not retain prompt history, configuration details, or reviewer edits. In that case, RAIDT can still be applied, but the resulting evidence pack will be thinner and the score profile should reflect that limitation, especially in Auditability and Traceability. The issue is not that RAIDT has failed. The issue is that the platform does not support the level of evidence capture needed for stronger governance assurance.

This is where RAIDT is more useful than a generic AI governance approach. A generic approach may stop at recommending better documentation. RAIDT turns the limitation into an assessable governance finding. It shows that the gap may need to be addressed through procurement requirements, wrapper design, workflow redesign, logging infrastructure, or policy constraints on which tools are acceptable for certain classes of work.

Practical example in RAIDT terms

Consider an enterprise productivity setting in which staff use a generative AI assistant to draft contract summaries for internal procurement teams. The use case is attractive because it speeds up first-pass review of supplier terms, but the run-level issue is that the chosen platform only stores the final generated summary and a timestamp. It does not preserve the original prompt, attached contract excerpt, model settings, or the sequence of edits made by the employee before the summary is circulated.

The evidence needed for stronger RAIDT governance would include the task purpose, the source clause text supplied to the model, the prompt or instruction template, the model and version used, the generated draft, any employee edits, review comments from legal staff, and the final decision on whether the summary could be relied upon. Responsibility is affected because accountability for review and sign-off becomes harder to demonstrate. Auditability is affected because a later reviewer cannot reconstruct how the summary emerged. Interpretability is affected because the reasoning context of the output is under-documented. Dependability is affected because recurring output quality problems cannot be analysed properly across runs. Traceability is affected because the chain from source text to generated and approved output is incomplete.

In governance-readiness terms, evidence capture feasibility improves the organisation's position when the tool is wrapped with structured templates and logging controls, or when staff are required to submit source excerpts, prompts, and reviewer notes into an evidence form before relying on the output. RAIDT therefore makes the limitation actionable: it identifies what additional evidence infrastructure is required before the workflow can be treated as strongly governable.

Detailed link to RAIDT

Evidence capture feasibility links to RAIDT in four ways.

First, it tests RAIDT's core idea that responsible governance should be grounded in evidence from actual runs rather than broad claims about tools or policies.

Second, it determines whether the run can function as a meaningful unit of governance, because a run that cannot be evidenced adequately cannot be reviewed in depth.

Third, it shapes the quality of the evidence pack and the confidence with which a RAIDT score profile can be justified across the five pillars.

Fourth, it strengthens reviewability, contestability, audit readiness, and organisational learning by revealing where evidence infrastructure is sufficient and where governance is being constrained by technical or process limitations.

Evidence capture feasibility ? Run-level evidence ? Evidence pack ? RAIDT score profile ? Governance readiness

Link to the five RAIDT pillars

Responsibility

Evidence capture feasibility supports Responsibility by determining whether the organisation can show who initiated, reviewed, approved, or relied on a run and under what authority.