evals@earendil-works/pi-evals · private

运行器、评测定义与镜像：三个目录Runner, eval definitions and image: three directories#

一句话结论Takeawaysrc/ 是运行器：cli.ts 编排一次对比，docker.ts 构建两个镜像并运行一侧，plan.ts 展开任务，report.ts 读 Vitest JSON、配对、算提升，harness.ts 是 vitest-evals 适配器。evals/ 放评测定义和夹具；docker/ 放镜像构建文件。package.json 只有五个脚本：test、eval:host、eval:docs、eval 与 clean。src/ is the runner: cli.ts orchestrates one comparison, docker.ts builds the two images and runs one arm, plan.ts expands tasks, report.ts reads Vitest JSON, pairs arms and computes lift, and harness.ts is the vitest-evals adapter. evals/ holds eval definitions and fixtures; docker/ holds the image build files. package.json has just five scripts: test, eval:host, eval:docs, eval and clean.

公共 APIPublic API#

一句话结论Takeawaypi-evals 是私有包（private: true），没有 exports，不能被别的包导入；它的“API”是 5 个 npm 脚本。src/ 的 5 个模块之间共有 26 个值导出和 17 个类型导出，只给评测文件与运行器内部使用；评测文件只用 createPiCodingAgentHarness、createPiDocumentationEvalHarness 与 DOCUMENTATION_EVAL_TOOLS。pi-evals is a private package (private: true) with no exports, so nothing else can import it; its "API" is five npm scripts. Between them the 5 modules in src/ have 26 value exports and 17 type exports, used only by eval files and the runner itself; eval files use just createPiCodingAgentHarness, createPiDocumentationEvalHarness and DOCUMENTATION_EVAL_TOOLS.

npm 脚本npm scripts

package.json · scriptsL6

参数与环境变量Flags and environment

src/ 各模块的导出（逐行链接）Exports of each src/ module (line-linked)

由构建脚本扫描 src/ 下每个模块的 export 声明生成；每个符号链接到 v1.1.0 中的定义行。Generated at build time by scanning each src/ module's export declarations; each symbol links to its defining line at v1.1.0.

7 个评测：2 个 host，5 个文档提升Seven evals: two host, five documentation-lift#

一句话结论Takeawayhost 评测是普通的 vitest-evals 套件：smoke 不给工具，只问一个固定答案；documentation-audit 用 it.for 遍历 coding-agent 的 40 页文档，让模型只读核对实现并提交一个结构化结论。5 个 docs 评测都是“改配置/写扩展 → reload → 检查结果”的真实任务，由 StructuredOutputJudge、ToolCallJudge 或自定义 createJudge 评分，judgeThreshold: null 让低分成为数据而不是失败。Host evals are ordinary vitest-evals suites: smoke gives no tools and asks for one fixed answer; documentation-audit uses it.for over coding-agent's 40 doc pages, having the model check the implementation read-only and submit a structured verdict. The five docs evals are real tasks — change config or write an extension → reload → inspect the result — scored by StructuredOutputJudge, ToolCallJudge or a custom createJudge; judgeThreshold: null makes a low score data rather than a failure.

eval:docs：先写下实验计划，再逐个运行eval:docs: write the plan down first, then run each arm#

一句话结论Takeawaycli.ts 依次解析参数、确认 auth.json 里有该 provider 的凭据、构建两个镜像（id 必须不同）、在两个镜像里分别发现用例（必须完全相同）、生成任务计划，并把 protocol.json（含 sha256 协议摘要）和 expected-runs.json 写盘；然后对每个任务启动一个新容器，读回 vitest.json，最后汇总成 report.json / report.txt。cli.ts parses arguments, checks auth.json holds the provider's credential, builds the two images (whose ids must differ), discovers cases in both images (which must be identical), builds the task plan, and writes protocol.json (with a sha256 protocol digest) and expected-runs.json; it then starts a fresh container per task, reads the vitest.json back, and finally summarises into report.json / report.txt.

报告：结果 → 配对 → 比较The report: outcomes → pairs → comparison#

一句话结论TakeawayreadTaskObservation() 把每个任务归为 scored、unscored、skipped、pending 或 errored；resolvePair() 要求每一侧在计划里恰好一次、恰好观测到一次且已评分，否则这一对被阻塞并写明原因。每个评测集的通过率按 score ≥ 1 计算，提升 = 实验组 − 对照组；只要有一对被阻塞就不公布通过率。另有 5 个标记和 tokens / 工具 / 耗时 / 成本的配对均值差。readTaskObservation() classifies each task as scored, unscored, skipped, pending or errored; resolvePair() requires each arm to be planned exactly once, observed exactly once and scored, otherwise the pair is blocked with its reasons. Per eval set, pass rate counts score ≥ 1 and lift = treatment − control; if any pair is blocked, no pass rate is published. There are also five flags and paired mean deltas for tokens / tools / latency / cost.

扩展点Extension points#

测试Tests#

一句话结论Takeaway六个测试文件只测运行器本身，不调用模型：report.test.ts（12 个用例）覆盖每一种结果分类与会话快照；harness.test.ts（10）覆盖隔离环境、提示词校验与剪掉 <docs>；configured-runtime.test.ts（8）覆盖评分器读取的运行时状态；acme-server.test.ts（5）覆盖夹具服务器；comparison.test.ts（4）覆盖配对、阻塞与报告格式；plan.test.ts（4）覆盖任务展开与身份校验。评测本身（*.eval.ts）不计入。Six test files test only the runner and never call a model: report.test.ts (12 cases) covers every outcome classification and the session snapshot; harness.test.ts (10) covers the isolated environment, prompt verification and cutting <docs>; configured-runtime.test.ts (8) covers the runtime state judges read; acme-server.test.ts (5) covers the fixture server; comparison.test.ts (4) covers pairing, blocking and report formatting; plan.test.ts (4) covers task expansion and identity checks. The evals themselves (*.eval.ts) are not counted.

用例数 = 源码中 it( / test( 的出现次数（静态统计，不运行测试）。Cases = occurrences of it( / test( in the source (static count; tests were not run).

交叉链接Cross-links#