v1.1.0 Pi v1.1.0 源码分析 Analysis of Pi v1.1.0 · 2026-10-07
evals evals 私有包：行为评测与成对的文档提升对比（Docker 隔离） Private package: behavioural evals and paired documentation-lift comparisons (Docker-isolated)
Infra
← → 翻页 · F 全屏 · O 目录 · T 明暗 · L 语言 ← → navigate · F fullscreen · O overview · T theme · L language
01 · 运行器、评测定义与镜像：三个目录 01 · Runner, eval definitions and image: three directories Infra
运行器、评测定义与镜像：三个目录 Runner, eval definitions and image: three directories pi-evals: a private package — a 1,446-line runner, 7 eval files and a multi-stage Docker image src/ 运行器 the runner evals/ 评测定义与夹具 eval definitions and fixtures docker/ 镜像构建 image build package.json private: true · 没有 exports private: true · no exports 192 cli.ts eval:docs 的主程序：解析参数 → 镜像 → 发现 → 计划 → 逐个运行 → 报告 eval:docs main: parse args → images → discover → plan → run each → report 59 plan.ts (case, variant, run) 任务 (case, variant, run) tasks 483 report.ts 配对与提升 pairing and lift 175 docker.ts 镜像与一次隔离运行 images, one isolated run 537 harness.ts vitest-evals 适配器：在隔离目录里跑一个真实的 AgentSession the vitest-evals adapter: runs a real AgentSession in an isolated directory documentation-audit.eval.ts smoke.eval.ts custom-provider extensions models openai-provider tui acme-server.ts configured-runtime.ts 44 Dockerfile 5 个阶段 · 2 个目标 5 stages · 2 targets 152 entrypoint.ts 容器内的检查与 vitest in-container checks, vitest 70 install-runtime.mjs 打包并安装工作区 pack + install workspace pi-ai pi-coding-agent vitest-evals @vitest-evals/core autoevals vitest PI_SESSION_SNAPSHOT_ARTIFACT createPi*Harness host（2） host (2) docs（5） docs (5) 夹具 fixtures docs 评测文件名省略了 .docs.eval.ts docs file names shown without .docs.eval.ts 镜像里只有运行时：评测源码对模型用户不可读。 The image holds only the runtime: eval sources are unreadable to the model user. Dockerfile.dockerignore 排除 .git、node_modules、dist、.env*。 Dockerfile.dockerignore drops .git, node_modules, dist, .env*. test vitest：test/ 里的单元测试 vitest: the unit tests in test/ eval:host vitest --project host：在本机跑 host 评测 vitest --project host: host evals on this machine eval:docs node src/cli.ts：容器里的文档提升对比 node src/cli.ts: the documentation-lift comparison in containers eval eval:host，然后 eval:docs eval:host, then eval:docs clean 删除 .eval/ 产物目录 delete the .eval/ artifacts devDependencies 工作区内的 pi 包都来自当前仓库（npm overrides）。 Internal pi packages come from this repository (npm overrides). 评测的对象是 coding-agent 本身：host 评测在本机检验行为；docs 评测在容器里比较“有文档 / 没文档”两种安装，测量文档带来的提升。 The subject is coding-agent itself: host evals check behaviour on this machine; docs evals compare a with-docs and a without-docs install in containers to measure documentation lift. 标签 = 行数 tag = lines 运行器 runner harness 评测 evals Docker 使用 uses *.docs.eval.ts 是文档提升评测，由 eval:docs 在容器里成对运行；其余 *.eval.ts 是普通的 host 评测，由 eval:host 在本机运行。*.docs.eval.ts files are documentation-lift evals, run in pairs in containers by eval:docs; other *.eval.ts files are ordinary host evals run on the machine by eval:host.
02 · 7 个评测：2 个 host，5 个文档提升 02 · Seven evals: two host, five documentation-lift Infra
7 个评测：2 个 host，5 个文档提升 Seven evals: two host, five documentation-lift What is evaluated: 2 host suites (including a 40-page documentation audit) and 5 documentation-lift suites host 评测 · eval:host Host evals · eval:host 本机运行 on this machine 文档提升评测 · eval:docs Documentation-lift evals · eval:docs 每个用例 × 两种镜像 each case × both images read grep find ls submit_documentation_audit StructuredOutputJudge StructuredOutputJudge + ToolCallJudge StructuredOutputJudge StructuredOutputJudge createJudge documentation-audit.eval.ts “Audit documentation against implementation” > $path matches the implementation smoke.eval.ts “Answer a basic prompt” > returns the expected answer smoke：没有工具；答案必须正好是 "Paris"， smoke: no tools; the answer must be exactly "Paris", 且 usage 的 provider / model 与环境变量一致。 and usage must match PI_PROVIDER / PI_MODEL. documentation-audit：it.for 遍历 coding-agent/docs documentation-audit: it.for over the 40 pages 下的 40 页；只读工具 + 一个提交工具， in coding-agent/docs; read-only tools plus a submit verdict 必须是 "match"（mismatch / inconclusive tool; the verdict must be "match" (mismatch / 都让用例失败）。 inconclusive fail the case). Add custom streaming provider > implements a provider from an API fixture custom-provider.docs.eval.ts Create and use a tool extension > creates, reloads, and invokes the extension extensions.docs.eval.ts Add model to existing provider > adds the model without replacing existing models models.docs.eval.ts Add OpenAI-compatible provider > configures a provider that works through Pi openai-provider.docs.eval.ts Customize the interactive context footer > replaces numeric context usage with a progress bar tui.docs.eval.ts 每个都是 [prompt, reload]：先让 Pi 改配置或写扩展，重载后再检查结果。 Each is [prompt, reload]: Pi changes its config or writes an extension, then reloads and is checked. judgeThreshold: null：低分是数据，不是失败；分数参与配对比较。 judgeThreshold: null: a low score is data, not a failure; scores feed the paired comparison. acme-server.ts 提供假的 OpenAI 兼容 / NDJSON 流式 API 供 provider 用例调用。 acme-server.ts serves fake OpenAI-compatible / NDJSON streaming APIs for the provider cases. vitest.evals.config.ts · projects: docs = evals/**/*.docs.eval.ts · host = evals/**/*.eval.ts minus docs · testTimeout 300_000 · pool forks · maxWorkers 1 · fileParallelism false createPiDocumentationEvalHarness() 在容器沙箱外直接抛错，所以 docs 评测只能经 eval:docs 运行。 createPiDocumentationEvalHarness() throws outside the container sandbox, so docs evals only run through eval:docs. host 评测 host evals docs 评测（标签 = 评分器） docs evals (tag = judge) 每个 docs 用例的输入都是 [prompt, reload] 两步：模型改完配置后，harness 调 session.reload()，评分器再从运行时里读出真实状态。 Every docs case takes two steps, [prompt, reload]: after the model edits config the harness calls session.reload(), and the judge reads the real state back from the runtime.
03 · eval:docs：先写下实验计划，再逐个运行 03 · eval:docs: write the plan down first, then run each arm Infra
eval:docs：先写下实验计划，再逐个运行 eval:docs: write the plan down first, then run each arm One eval:docs run: write down the whole experiment first, then run each arm in a fresh container 第 7 步，对每个任务 Step 7, for each task .eval/<时间戳>_<uuid>/（0700） .eval/<timestamp>_<uuid>/ (0700) parseEvalCli --provider --model · -t · 文件 --provider --model · -t · files requireEvalAuthFile auth.json 里必须有该 provider auth.json must hold the provider buildImages without_docs / with_docs，id 必须不同 without_docs / with_docs, ids must differ discoverCases × 2 两种镜像里发现的用例必须完全相同 both images must discover identical cases createTaskPlan (case, variant, model, run) 全部任务 every (case, variant, model, run) task protocol.json + sha256 协议摘要 · expected-runs.json + sha256 protocol digest · expected-runs.json for task: runTask docker run 一个全新容器 → vitest.json docker run a fresh container → vitest.json summarize + format report.json / report.txt；有阻塞对 → 退出码 1 report.json / report.txt; blocked pairs → exit 1 protocol.json expected-runs.json discovery/<variant>/ tasks/<sha256>/vitest.json observations.jsonl <variant>/sessions/<sha256>/session.jsonl report.json report.txt runTask(docker, task) --project docs 文件 --testNamePattern ^<集> <用例>$ --project docs file --testNamePattern ^<set> <case>$ 有 vitest.json？ vitest.json? 有 → readTaskObservation；没有 → erroredObservation yes → readTaskObservation; no → erroredObservation observations.jsonl 每个任务之后整体重写：中途崩溃也留下已完成的观测 rewritten after every task: a crash keeps what finished 失败不中断 Failures don't stop it 失败或缺失的一侧被记下，计划内的其余任务照常继续 a failed or missing arm is recorded; the rest of the plan continues 产物可能包含提示词、回答、生成的代码与工具输出；目录被 .gitignore 忽略。重复次数按运行序号交替顺序，减少顺序偏差。 Artifacts may contain prompts, responses, generated code and tool output; the directory is git-ignored. Repetitions alternate order by run number to reduce order bias. 1 2 3 4 5 6 7 8 1 步骤 step 顺序 order 容器里执行 runs in a container 实验设计在运行前就固定：报告按 expected-runs.json 检查每一对，而不是只看实际跑出了什么。 The design is fixed before anything runs: the report checks each pair against expected-runs.json, not just against what happened to run.
04 · 任务计划：用例 × 运行序号 × 两种变体 04 · The task plan: case × run number × both variants Extensions
任务计划：用例 × 运行序号 × 两种变体 The task plan: case × run number × both variants createTaskPlan: case × run number × both variants; odd runs start with without_docs, even runs flip 身份与校验 Identity and checks plan.ts 例子：2 个用例，--runs-per-variant 3 → 12 个任务（按执行顺序） Example: 2 cases, --runs-per-variant 3 → 12 tasks (in execution order) #1 · without_docs #2 · with_docs #3 · with_docs #4 · without_docs #5 · without_docs #6 · with_docs #7 · without_docs #8 · with_docs #9 · with_docs #10 · without_docs #11 · without_docs #12 · with_docs parseDiscoveredCases vitest list 的 JSON → 用例 vitest list JSON → cases "<eval set> > <case>" 全名必须恰好两段，否则报错 the full name must have exactly two parts 身份不能重复 No duplicate identities JSON.stringify([evalSet, caseId]) 模型身份 Model identity "provider/model"：必须含 "/"，且不在两端 "provider/model": must contain "/", not at either end runsPerVariant 正整数；默认 1 · --runs-per-variant · PI_EVAL_RUNS_PER_VARIANT positive integer; default 1 · --runs-per-variant · PI_EVAL_RUNS_PER_VARIANT 任务目录 Task directory sha256([set, case, variant, model, run]) Add model to existing provider > adds the model … run 1 run 2 run 3 Create and use a tool extension > creates, reloads … run 1 run 2 run 3 每个 (用例, 模型, 运行序号) 组成一对：without_docs 是对照组，with_docs 是实验组。所有任务在运行前就写进 expected-runs.json。 Each (case, model, run number) forms a pair: without_docs is the control and with_docs the treatment. Every task is written to expected-runs.json before anything runs. 加粗 = 这一轮先跑的一侧 bold = the arm that runs first in its round without_docs（对照） without_docs (control) with_docs（实验） with_docs (treatment) 用例来自 vitest list --json：parseDiscoveredCases() 拒绝不是恰好两段的名字和重复身份。 Cases come from vitest list --json: parseDiscoveredCases() rejects names that are not exactly two parts and duplicate identities.
05 · 一个 Dockerfile，两个只差文档的目标 05 · One Dockerfile, two targets that differ only in docs Extensions
一个 Dockerfile，两个只差文档的目标 One Dockerfile, two targets that differ only in docs The Dockerfile: one build, two targets that differ only in docs; docker run is read-only, one task each docker/Dockerfile 5 个阶段 · 2 个构建目标 5 stages · 2 build targets buildImages() docker.ts builder FROM node:24-slim · 挂载仓库 · npm ci --ignore-scripts FROM node:24-slim · repo bind mount · npm ci --ignore-scripts build:offline → install-runtime.mjs /opt/evaluator without-docs-install FROM builder 删掉 coding-agent 的 README.md CHANGELOG.md docs/ examples/ rm coding-agent README.md CHANGELOG.md docs/ examples/ evaluator FROM node:24-slim · COPY builder 的 root → /repo FROM node:24-slim · COPY builder root → /repo chmod go-rwx src evals docker · ENTRYPOINT entrypoint.ts with_docs + builder 的 node_modules · PI_EVAL_VARIANT=with_docs + builder node_modules · PI_EVAL_VARIANT=with_docs without_docs + 去掉文档的 node_modules · PI_EVAL_VARIANT=without_docs + docs-free node_modules · PI_EVAL_VARIANT=without_docs FROM COPY root FROM FROM node_modules 镜像名 Image name pi-evals-<hash>-<variant>:local hash sha256(repoRoot)[:12] 构建 Build docker build --target <variant> 然后 Then docker image inspect → Id cli.ts 两个 Id 相同 → 报错 equal Ids → error 上下文排除 Context ignores .git .eval node_modules dist coverage .env .env.* install-runtime.mjs 只把运行器需要的文件拷进 root/ install-runtime.mjs copies only runner files into root/ docker.ts · 每个任务一次 docker.ts · once per task docker run --rm --read-only --tmpfs /tmp:rw,exec,mode=1777 --tmpfs /repo/node_modules/.vite-temp:rw,exec,mode=1777 --mount type=bind,source=<.eval/…/tasks/<sha256>>,target=/artifacts --env PI_EVAL_ARTIFACT_DIR=/artifacts --env PI_EVAL_RUNS_PER_VARIANT=<n> --env PI_EVAL_SANDBOX_UID=65532 --env PI_EVAL_SANDBOX_GID=65532 --env PI_PROVIDER --env PI_MODEL [--env PI_EVAL_ARTIFACT_UID/GID = host user] --mount type=bind,source=<auth.json>,target=/run/pi-eval-secrets/auth.json,readonly <image> <entrypoint args> FROM / COPY 阶段 FROM / COPY stage COPY node_modules 带文档 with docs 无文档 / 评测器 no docs / evaluator 两个变体用同一份安装结果：对照组只是在其上删除文档，所以差异只有文档本身。 Both variants share one installation: the control merely deletes docs from it, so the only difference is the documentation.
06 · 三层防护：模型读不到评分器 06 · Three guards: the model cannot read its judges pi-coding-agent
三层防护：模型读不到评分器 Three guards: the model cannot read its judges Three guards: a read-only container → the entrypoint self-checks as root → the harness drops to UID 65532 before the model gets tools docker run 宿主 → 容器 host → container entrypoint.ts 以 root 运行 runs as root runPiCodingAgent harness.ts · 每个用例 harness.ts · each case 模型的工具以 UID 65532 运行 The model's tools run as UID 65532 read · write · edit · grep · find · ls；没有 shell、没有联网搜索 read · write · edit · grep · find · ls; no shell, no web search workspace/ (rw) isolated home/ pi-coding-agent dist README · docs/ · examples/ (with_docs) src/ evals/ (judges, fixtures) docker/ vitest configs auth.json credential env var other packages' docs 只读根文件系统 Read-only root filesystem --read-only；可写的只有两个 tmpfs --read-only; only two tmpfs mounts are writable 每个任务一个新容器 A fresh container per task --rm：上一个任务的痕迹不会留下 --rm: nothing carries over between tasks /artifacts 绑定到 .eval/…/tasks/<sha256> bound to .eval/…/tasks/<sha256> /run/pi-eval-secrets/auth.json 只读挂载；目录在镜像里是 0700 read-only mount; the directory is 0700 in the image 源码 root 专属 Root-only sources 镜像构建时 chmod go-rwx src evals docker chmod go-rwx src evals docker at build time PI_EVAL_SANDBOX_UID/GID = 65532，交给后面两层 = 65532, handed to the next two layers /repo 内容完全符合清单 /repo matches an exact allowlist node_modules · package.json · packages · vitest.base.ts 内部依赖不带文档 Internal deps carry no docs 无 docs/ examples/ src/ test/ README CHANGELOG no docs/ examples/ src/ test/ README CHANGELOG 变体与文档一致 Variant matches its docs without_docs 必须没有；with_docs 必须有 docs/models.md without_docs: none; with_docs: docs/models.md exists 评测源码：root 专属 + 探测 Eval sources: root-only + probe 以 65532 身份尝试读取，必须失败 a read attempted as 65532 must fail coding-agent 来自 dist coding-agent resolves from dist …/pi-coding-agent/dist/index.js 凭据交给沙箱用户 Credentials handed over auth.json → /tmp/pi-eval-host-agent，chown 65532 auth.json → /tmp/pi-eval-host-agent, chown 65532 只跑指定的评测文件 Only an explicit eval file PI_EVAL_CONTAINER=1 · vitest run / list mkdtemp pi-eval-* workspace/ · home/ · sessions/ applyIsolatedEnvironment HOME / PI_CODING_AGENT_DIR 指向临时目录；清掉 PI_EVAL_* HOME / PI_CODING_AGENT_DIR → temp dir; PI_EVAL_* removed InMemoryCredentialStore 凭据只在内存里；随后删除 auth.json credential in memory only; auth.json is then deleted 隐藏凭据环境变量 Credential env var hidden 运行期间 delete，结束后恢复 deleted for the run, restored afterwards 不许有意外扩展 No unexpected extensions 只允许 <inline:…> 的隐藏扩展 only hidden <inline:…> extensions enterToolSandbox 转换后的模块 0600；setgroups([]) → setgid → setuid transformed modules 0600; setgroups([]) → setgid → setuid 确认读不到 Confirm it cannot read 读到任何受保护模块 → 报错 any readable protected module → error 看得到 can see 看不到 cannot see README 坦白说明：调用 provider 仍需要容器联网，所以单靠 Docker 不能证明模型写的代码从不联网。 The README is explicit: provider traffic still needs container network access, so Docker alone cannot prove agent-written code never uses the network. 下一层 next layer 容器 container 入口（root） entrypoint (root) 测试框架 harness docs 评测只给 read、write、edit、grep、find、ls 六个工具：没有 shell，也没有联网搜索。 Docs evals get only six tools — read, write, edit, grep, find, ls: no shell and no web search.
07 · 一次运行：真实的 AgentSession，跑完即拆 07 · One run: a real AgentSession, torn down afterwards pi-coding-agent
一次运行：真实的 AgentSession，跑完即拆 One run: a real AgentSession, torn down afterwards One run of one case: a real AgentSession, torn down afterwards; the two variants differ only in the <docs> of the system prompt runPiCodingAgent(input, signal, setArtifact, options) promptAgent 每个 prompt 步骤 each prompt step createPiDocumentationEvalHarness 失败时 On failure resolveModelSelection · mkdtemp provider/model；root/ 下 workspace、home、sessions（见 ev_sandbox） provider/model; workspace, home, sessions under root/ (see ev_sandbox) ModelRuntime.create({ credentials }) getModel → getAuth；没有认证就直接报错 getModel → getAuth; no auth → error createAgentSessionServices cwd = workspace · extensionFactories（隐藏的提示词变换） cwd = workspace · extensionFactories (hidden prompt transform) createAgentSessionFromServices thinkingLevel "off" · tools / noTools / customTools enterToolSandbox → for step of input prompt → promptAgent；reload → session.reload() prompt → promptAgent; reload → session.reload() 诊断数据 Diagnostics 转录事件 · systemPromptSha256 · tokens / toolCalls · 有定价才算 cost transcript events · systemPromptSha256 · tokens / toolCalls · cost only if priced verifySystemPrompt → output() 有 output 选项就交给它（评分器用），否则就是回答文本 an output option shapes the result for judges; otherwise the response text finally session.jsonl → PI_SESSION_SNAPSHOT_ARTIFACT · dispose · rm root · 还原环境 session.jsonl → PI_SESSION_SNAPSHOT_ARTIFACT · dispose · rm root · restore env preamble <tools> <rules> <docs> Pi documentation (read only … … <cwd> preamble <tools> <rules> … <cwd> stopReason ∈ { stop, toolUse } 否则抛出 errorMessage 或 "unexpected stop reason" otherwise throw errorMessage or "unexpected stop reason" stop && !text → error "Agent run produced no assistant text." getLastAssistantText() ?? "" 最后一个 prompt 的回答就是 response the last prompt's answer becomes the response 须 PI_EVAL_CONTAINER=1 且有沙箱 UID/GID，否则直接报错 needs PI_EVAL_CONTAINER=1 and a sandbox UID/GID, or it throws without_docs with_docs name without_docs with_docs tools read…ls (6) read…ls (6) transform excludePiDocumentation — expect <docs> false true 两种变体共用同一个用例定义和同一个评分器。 Both variants share one case definition and one judge. 已有诊断 → attachHarnessRunToError：带 errors 与耗时的部分结果 diagnostics exist → attachHarnessRunToError: a partial run with errors + timings 运行与清理都失败 → AggregateError run and cleanup both fail → AggregateError 报告据此把这一侧记为 errored the report then records that arm as errored 系统提示词的分段（coding-agent 的 system-prompt.ts） System prompt sections (coding-agent system-prompt.ts) with_docs without_docs excludePiDocumentation 剪掉 <docs>…</docs>；verifySystemPrompt 要求 <rules> 还在、<docs> 有无与变体一致 excludePiDocumentation cuts <docs>…</docs>; verifySystemPrompt requires <rules> and a <docs> matching the variant 1 2 3 4 5 6 7 8 1 步骤 step 顺序 order 只在 with_docs with_docs only 两种变体的唯一区别：without_docs 注册一个隐藏的 before_agent_start 扩展，用 excludePiDocumentation 剪掉系统提示词里的 <docs>…</docs>。 The only difference between variants: without_docs registers a hidden before_agent_start extension that cuts <docs>…</docs> out of the system prompt with excludePiDocumentation.
08 · 报告：结果 → 配对 → 比较 08 · The report: outcomes → pairs → comparison Infra
报告：结果 → 配对 → 比较 The report: outcomes → pairs → comparison The report: classify each arm, then compare in pairs; if any pair is incomplete, no pass rate is published readTaskObservation 每个任务一个结果 one outcome per task resolvePair (集, 用例, 模型, 运行序号) (set, case, model, run) 每个评测集的比较 Comparison per eval set summarize… formatEvalComparisonReport → report.txt 示例数据：2 个评测集 × 3 次运行，第二个集里有一侧出错 illustrative data: 2 eval sets × 3 runs, one arm errored in the second set errored errored skipped / pending errored errored unscored scored vitest.json 读不了 vitest.json unreadable 断言不是恰好 1 条 / 名字不符 not exactly 1 assertion / name mismatch skipped · todo · disabled · pending 没有 harness 运行 / 模型不符 no harness run / wrong model failed 或运行里有错误 failed, or run has errors 没有分数 no score 有分数（0 也算） a score (0 counts) 每一侧在计划里恰好 1 次 each arm planned exactly once design expected 1 run, found N 观测数 = 计划数 observations = planned expected 1 observation, found N 结果是 scored outcome is scored <variant>: <outcome> 全部满足 → Pair { control, treatment } all hold → Pair { control, treatment } 否则 → BlockedPair，带上面的原因（红色） otherwise → BlockedPair with the reasons above (red) without_docs = 对照 · with_docs = 实验 without_docs = control · with_docs = treatment 通过 Pass score ≥ 1 通过率 Pass rate passes / eligible pairs 提升 Lift treatment − control (toPrecision 15) 不公布 Withheld 任何一对被阻塞时 if any pair is blocked no-lift 两组通过率相同 equal pass rates negative-delta 实验组更低 treatment is lower control-saturated 对照组 100% control at 100% treatment-saturated 实验组 100% treatment at 100% flaky 同一用例同一侧跨运行结果不同 same case + arm passes in some runs only 另有 tokens / tools / latency / cost 的配对均值差 plus paired mean deltas for tokens / tools / latency / cost Documentation Eval Comparisons Add model to existing provider Pairs 3/3 eligible Pass rate +33.3 pp (with 100.0%, without 66.7%) Flags treatment-saturated, flaky Tokens -3750.0 (with 15373.3, without 19123.3, 3 pairs) Tools -4.3 (with 5.3, without 9.7, 3 pairs) Latency -13533.3ms (with 31100.0ms, without 44633.3ms, 3 pairs) Est. cost -$0.0076 (with $0.0353, without $0.0429, 3 pairs) Create and use a tool extension Pairs 2/3 eligible Pass rate withheld because pairs are blocked Flags flaky Tokens -6400.0 (with 23800.0, without 30200.0, 2 pairs) Tools -6.0 (with 8.5, without 14.5, 2 pairs) Latency -14850.0ms (with 54600.0ms, without 69450.0ms, 2 pairs) Est. cost -$0.0143 (with $0.0537, without $0.0681, 2 pairs) Operational totals without_docs: 6 runs, 146570 tokens, 71 tools, 339.10s, $0.3301 cost with_docs: 6 runs, 120120 tokens, 44 tools, 263.50s, $0.2725 cost Blocked pairs Create and use a tool extension/creates, reloads, and invokes the extension/ provider/model/run-2: with_docs: errored 通过率、提升与标记按 report.ts 的规则从示例数据算出（构建时计算）。有阻塞对时 eval:docs 退出码为 1；report.json 是 schemaVersion 3。 Pass rates, lift and flags are computed from the example data with the report.ts rules at build time. With blocked pairs eval:docs exits 1; report.json is schemaVersion 3. 数据流 data flow scored 阻塞 / 出错 blocked / errored 0 分也是 scored：它是数据。只有没有分数才是 unscored。 A score of 0 is still scored — it is data. Only a missing score is unscored.
扩展点 Extension points pi-coding-agent
evals 的扩展点 Where to plug into evals 1
evals/<name>.docs.eval.ts evals/<name>.docs.eval.ts
新增一个文档提升评测：自动在两个变体里成对运行并进入报告。 Add a documentation-lift eval: it runs in pairs in both variants and lands in the report automatically.
2
evals/<name>.eval.ts evals/<name>.eval.ts
新增一个 host 评测：普通的 vitest-evals 套件，由 eval:host 在本机运行。 Add a host eval: an ordinary vitest-evals suite run on the machine by eval:host.
3
createPiDocumentationEvalHarness({ output }) createPiDocumentationEvalHarness({ output })
把运行后的状态整理成评分器可比较的 JSON。 Shape the post-run state into JSON a judge can compare.
4
judges: StructuredOutputJudge / ToolCallJudge / createJudge judges: StructuredOutputJudge / ToolCallJudge / createJudge
选择或编写评分器；judgeThreshold: null 让分数成为数据。 Pick or write judges; judgeThreshold: null makes the score data.
5
--runs-per-variant · -t · PI_PROVIDER / PI_MODEL --runs-per-variant · -t · PI_PROVIDER / PI_MODEL
控制重复次数、选择用例和被测模型。 Control repetitions, select cases and choose the model under test.