Extraction, generation, retrieval, and agents: ask what the system actually does

“AI-powered” is not a sufficient description of a workflow. A useful vendor explanation should tell you what information enters the system, how it is transformed, which sources it uses, and whether it can take actions outside the screen.

Use the four categories below as an editorial vocabulary for evaluation, not as mutually exclusive product classes. A single application can combine all four.

In a document workflow, ask the tool to identify information that exists in the input and place it into a defined output field. Your evaluation should separate reading the text from assigning it to the right field. Microsoft distinguishes word-level transcription confidence from other extraction confidence in its Document Intelligence guidance; those details are specific to that product and are not a universal scoring standard (Microsoft documentation).

For an insurance test, define whether “premium” means a particular line premium, a package premium, or the total including separately disclosed charges. The test is not well specified until the expected meaning is clear.

A generated summary is not a photocopy of the source. NIST describes confabulation as confidently presented erroneous or false content and recommends checking outputs against known ground truth and verifying their sources (NIST Generative AI Profile).

Ask whether the product distinguishes copied facts, calculated values, and generated interpretation. In a proposed agency workflow, those should be separately reviewable. Fluent prose should not conceal an unsupported inference.

Retrieval-augmented generation, or RAG, adds retrieved external information to the context supplied to a language model; AWS describes ingestion, retrieval, and response generation as distinct parts of that process (AWS documentation).

The buyer's question becomes: which documents were eligible, which were retrieved, and which version was used? A source-grounded answer can still require checking that the source belongs to the right insured, policy period, and task.

OWASP uses “agency” to describe an AI system's ability to invoke functions or interact with other systems through tools or extensions (OWASP Excessive Agency). This is unrelated to the insurance-business meaning of agency.

Distinguish “draft an email” from “send an email,” and “suggest a record update” from “write it into the system.” Ask for the exact permissions and approval step, not simply whether a human is “in the loop.”

Next: Read why source links are not enough. Bring this four-part vocabulary to the vendor evidence worksheet.