Vendor evidence worksheet

Write down the claim before debating the percentage. “Accurate,” “faster,” and “production-ready” are questions to investigate, not interchangeable measures.

Download the blank vendor evidence worksheet. Use one row per claim, even when several claims come from the same vendor; a source page, a vendor, and a performance claim are different counting units.

• Exact statement: Copy the wording, source URL, and source date. Keep any qualification next to the number.
• Task and population: Ask which fields, documents, customers, language, and operating conditions were evaluated.
• Denominator: Ask whether omitted fields, rejected documents, failed runs, and difficult cases were counted.
• Method: Identify who scored outputs, how ground truth was established, and whether reviewers knew which system produced them.
• Error severity: Separate cosmetic errors from incorrect or missing information that could affect an action.
• Time boundary: Ask whether savings include setup, review, correction, exception handling, and reruns.
• Transferability: State which parts of your workflow differ from the disclosed test.
• Evidence status: Mark vendor statement, published method, independently evaluated result, locally observed result, or not disclosed. Do not combine those labels into a single credibility score.

NIST's generative AI profile recommends checking performance claims and cautions against extrapolating capability from narrow or anecdotal assessments; it also calls for third-party due diligence (NIST GenAI Profile). This worksheet is an editorial implementation aid, not an independent evaluation of a vendor.

Record “not disclosed in the material reviewed” and the scope of the material. That is more precise than concluding that no evaluation exists or the product is inaccurate.

Ask whether an authorized pilot can test the unresolved point in your environment. A strong answer may be a narrower, clearly defined claim rather than a larger headline percentage.

Choose proceed to controlled pilot, request clarification, decline, or insufficient evidence. Record your reason separately from the vendor's explanation, and keep commercial preference separate from observed performance.

Read Count missing fields before you quote an accuracy rate and Measure the work left after the AI finishes. Neither a large percentage nor an attractive demonstration removes the need to define what was measured.