How AI agent evidence ratings work

Three separate ratings

Documentation applies to every edition. Repository evidence applies when public source is reviewed. Hosted disclosures apply to hosted editions, including those with public source. Scores are comparable only within the named checklist and version, never against skill scores.

Each checklist has ten binary criteria: supported = 1, reviewed-absent = 0. An unknown criterion means the whole score is withheld, with review coverage shown. Not applicable is different from unknown. No combined score is calculated.

A 10/10 means all ten disclosures or source artifacts were found. It says nothing about how well the agent works or how strong its security policy is.

Documentation

1. Intended tasks. Names concrete tasks the selected edition is intended to perform.
2. Boundaries. States a limitation or condition under which this edition should not be used.
3. Setup instructions. Provides a sequenced install or access route for this edition, not just a product announcement.
4. Prerequisites. Identifies required host software or account prerequisites for this edition.
5. Model requirements. Identifies the model/provider requirements or explains provider selection.
6. Worked example. Shows a concrete request/workflow and describes the expected outcome.
7. Tools and integrations. Identifies tools or integrations available to the selected edition.
8. Permission description. Explains at least one permission boundary or approval behavior and its scope.
9. Failure and recovery. Explains how to handle a concrete error, interruption or recovery case.
10. Support route. Links a documented issue, support or troubleshooting route.

Repository evidence

1. License text. The reviewed source contains substantive license terms applying to this edition.
2. Attribution. Reviewed source supplies copyright or attribution notices; presence is not a legal completeness opinion.
3. Pinned source. Intake records an exact upstream commit and a verified file at that commit.
4. Distribution relationship. Pinned source documents its official installation/distribution route for this edition; this does not prove binary equivalence.
5. Dependency manifest. A reviewed manifest identifies the edition's runtime/build dependencies.
6. Resolved dependencies. A reviewed lockfile or dependency inventory records exact resolved versions for this edition.
7. Test sources. Reviewed source contains substantive tests for this edition. No execution or pass claim is awarded.
8. CI definition. A reviewed CI definition invokes relevant checks. This does not establish that CI passed.
9. Change history. A reviewed release/changelog document records concrete product changes.
10. Security reporting. A reviewed policy specifies how to report a security concern. It is not a security assessment.

Hosted disclosures

1. Service terms. Official service terms identify conditions applying to this hosted edition.
2. Pricing and usage units. Dated official documentation explains pricing/access and the unit of usage charged.
3. Data retention. Official documentation states a retention period or retention/deletion policy for customer data.
4. Training and data use. Official documentation states whether/how customer data is used for model training.
5. Authentication. Official documentation identifies supported sign-in or account authentication controls.
6. Permission controls. Official documentation explains scoped access restrictions or human approvals.
7. Audit and export. Official documentation identifies available audit records or session-data export and access conditions.
8. Deletion and revocation. Official documentation explains how to delete relevant data or revoke access.
9. Service status. An official public status or incident page identifies this service.
10. Product changes. An official changelog describes dated changes to this hosted service.

Evidence you can inspect

Repository receipts identify the exact commit, file location, Git blob and file SHA-256. Hosted receipts retain dated paraphrased observations with a digest; they do not archive the full page or identify deployed code. Publisher claims remain publisher claims. Reports bind the catalog snapshot and rubric with hashes.

Missing, stale and failed reviews

Unknown evidence is never scored as zero. Reviews older than 30 days are overdue and remain visible with their original dates and scores. Failed refreshes keep the previous accepted snapshot; Git history preserves changes. Intake and refresh are manual and read-only.

Performance and security

The launch catalog has no runtime tests or independent security assessments. Test files, CI definitions and security policies only earn evidence points. Known findings and permission warnings stay visible regardless of documentation score. A future test program needs a separately reviewed method, named version, environment and reproducible results.

Download rubric · Catalog schema · Submit a correction