From 00f2eab716242b29ca67960959a025f1f074dac3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E4=BD=95=E5=BC=80=E5=85=83?= Date: Sat, 27 Jun 2026 09:27:32 -0700 Subject: [PATCH] feat(gatekeeper): add optional advisory flags (soft layer, default-off, deterministic) Add a complementary soft layer to the hard gates: advisory_flags surface governance hints (e.g. oversized-but-not-blocked PRs -> suggest splitting) in the scorecard WITHOUT affecting score or verdict. Hard gates stay the real teeth; advisory leaves the call to humans. - additive & backward-compatible: default off; policies lacking the section behave exactly as before (verified: default policy on real PR #275 yields zero advisory section, PASS 87/100 unchanged) - deterministic: consumes only already-collected PR data (no wall-clock input), preserving same-input -> same-score -> same-verdict - first flag large_pr_files (changed-files >= threshold) answers the 166-open-PR backlog reality where big PRs should be flagged, not blocked - evaluate_advisory_flags() is separate from score_dimensions/decide_verdict; the 5-dim scoring and verdict logic are untouched - 7 new unit tests (15 total) incl. an invariant that enabling advisory does NOT change total/verdict; verified live on real open PR #275 - REFERENCE.md 1.9 + gatekeeper.yaml + verification.md E + demo scorecard Extends the merged gatekeeper line (#90 skill -> #219 workflow -> #220 issueops). Co-Authored-By: Claude Opus 4.8 --- .../workflows/pr-quality-gatekeeper/README.md | 4 +- .../docs/verification.md | 15 ++++- .../demo-outputs/scorecard-advisory.md | 26 +++++++ .../scripts/gatekeeper_workflow.py | 54 ++++++++++++++- .../tests/test_scoring.py | 67 +++++++++++++++++++ skills/gitlink-gatekeeper/REFERENCE.md | 11 +++ .../examples/gatekeeper.yaml | 9 +++ 7 files changed, 179 insertions(+), 7 deletions(-) create mode 100644 examples/workflows/pr-quality-gatekeeper/examples/demo-outputs/scorecard-advisory.md diff --git a/examples/workflows/pr-quality-gatekeeper/README.md b/examples/workflows/pr-quality-gatekeeper/README.md index 997aa9b..6c824ef 100644 --- a/examples/workflows/pr-quality-gatekeeper/README.md +++ b/examples/workflows/pr-quality-gatekeeper/README.md @@ -55,13 +55,15 @@ python3 scripts/gatekeeper_sweep.py \ | 注入真实审查发现 | 同一 PR + `findings.example.json` | ❌ REQUEST_CHANGES 55/100（裁决翻转，确定性可复算） | | `--apply` 真实回写 | 自有 fork 的演练 PR | 评分卡评论 + tracking issue + 裁决标签全部由 API 回执确认 | | **全仓批量体检** | 本仓库**全部 113 个 open PR** | 113/113 成功：PASS 105 / COMMENT 6 / REQUEST_CHANGES 2，均分 88.5；96% 未关联 issue | -| 单测 | `tests/test_scoring.py` | 全绿（锁定四个权威裁决案例的分值与裁决） | +| **咨询标记（advisory）** | 真实 open PR #275（10 文件） | 触发「建议拆分」提示，但**裁决不变** PASS 87/100（软建议不改判） | +| 单测 | `tests/test_scoring.py` | 15 全绿（8 评分案例 + 7 advisory，锁定确定性） | ## 设计要点 - **确定性评分**：AI 只负责产出「发现列表」（可选注入），扣分与裁决由纯函数完成——同策略 + 同 PR → 同裁决，可逐位手算复现、可审计。 - **安全默认**：默认 dry-run 什么都不写；即便策略开了 `auto_merge`，也必须 `verdict == PASS` 且显式 `--apply` 才会合并；强语义的 approve/reject 始终留给人，自动裁决只以建议性 `common` 评论 + 标签呈现。 - **原生适配 GitLink**：PR 标题/描述取自 `pr +view` 的 `issue.subject/description`；标签挂载走「`label +list` 查 id → Raw API `POST /:owner/:repo/issues/`」；尊重 `common/approved/rejected` 三态 review。 +- **硬门禁 + 软建议两层**：硬门禁命中即拦截（真牙齿）；可选的 `advisory_flags`（默认关闭、向后兼容、确定性）只在评分卡里提示、不改裁决——把「超体量 PR 该提示拆分但不该阻断」这类治理建议留给人工，正回应活跃仓库的 PR 积压实况。 - **零依赖、零常驻**：纯标准库脚本 + `gitlink-cli`，无需部署 webhook 服务或数据库，CI 一条 step 即可接入（见 `ci-example/`）；确定性意味着**大规模治理零 AI 成本**。 ## 许可证 diff --git a/examples/workflows/pr-quality-gatekeeper/docs/verification.md b/examples/workflows/pr-quality-gatekeeper/docs/verification.md index c2d461e..53915f7 100644 --- a/examples/workflows/pr-quality-gatekeeper/docs/verification.md +++ b/examples/workflows/pr-quality-gatekeeper/docs/verification.md @@ -40,14 +40,23 @@ - 完整报告（含全量明细表）：[`../examples/demo-outputs/sweep-report-2026-06-10.md`](../examples/demo-outputs/sweep-report-2026-06-10.md) - 诚实口径：批扫不注入审查发现（review_findings 维未评、按满分计），CI 统一 `--skip-ci`（unknown 半分）——总分代表「除人工/AI 审查外的工程卫生分」，偏乐观 -## E. 单元测试（确定性回归护栏） +## E. 咨询标记（advisory）→ 真实 PR 触发但不改裁决 + +开启 `advisory_flags`（`enabled: true, large_pr_files: 10`）对本仓库真实 open PR `#275`（feat(shortcut): add repo upload，10 个变更文件）跑 dry-run： + +- `large_pr_files` 标记**触发**（10 ≥ 阈值 10），评分卡新增 `### 💡 Advisory` 节提示「建议拆分」 +- **裁决不变**：仍为 **PASS 87/100**——咨询标记不参与评分、不改变裁决（与 `test_scoring.py` 的 `test_advisory_does_not_change_verdict` 不变量一致） +- 这是与硬门禁互补的「软」一层：硬门禁命中即拦截（真牙齿），咨询标记只提示、把判断留给人——正回应 D 节里 166-PR 积压仓库「超体量 PR 该提示拆分但不该阻断」的真实场景 +- 产物：[`../examples/demo-outputs/scorecard-advisory.md`](../examples/demo-outputs/scorecard-advisory.md) + +## F. 单元测试（确定性回归护栏） ```bash $ python3 tests/test_scoring.py -OK +OK # 15 tests（8 评分案例 + 7 advisory） ``` -锁定四个权威裁决案例（PASS / REQUEST_CHANGES / COMMENT / 硬门禁直拒）的**总分与裁决**与 Skill 文档逐位一致；任何改动若破坏「同输入 → 同分 → 同裁决」，测试立即变红。 +锁定四个权威裁决案例（PASS / REQUEST_CHANGES / COMMENT / 硬门禁直拒）的**总分与裁决**与 Skill 文档逐位一致；advisory 测试另锁「默认关闭向后兼容」与「开启触发也不改总分/裁决」两条不变量。任何改动若破坏「同输入 → 同分 → 同裁决」，测试立即变红。 ## 真实运行当场暴露过的问题（透明记录） diff --git a/examples/workflows/pr-quality-gatekeeper/examples/demo-outputs/scorecard-advisory.md b/examples/workflows/pr-quality-gatekeeper/examples/demo-outputs/scorecard-advisory.md new file mode 100644 index 0000000..422bb68 --- /dev/null +++ b/examples/workflows/pr-quality-gatekeeper/examples/demo-outputs/scorecard-advisory.md @@ -0,0 +1,26 @@ +## 🛡️ Gatekeeper Report — PR #275 feat(shortcut): add repo upload + +**Verdict: ✅ PASS** · Score: 87/100 · policy: adv-demo-policy.yaml@v1 + +| Dimension | Weight | Score | Notes | +|-----------|:------:|:-----:|-------| +| Review findings | 40 | 40/40 | 0 blocker / 0 major / 0 minor / 0 nit | +| Test coverage | 20 | 17/20 | 3 src / 2 test files | +| PR hygiene | 15 | 10/15 | desc ✓ / linked issue ✗ / size ✓ | +| Commit quality | 15 | 15/15 | 0/0 conventional | +| CI status | 10 | 5/10 | unknown | + +### 👥 Suggested reviewers (4) +- @doc-maintainer — 3 file(s): README.md, README.zh-CN.md, doc/changes/repo-upload-shortcut.md +- @go-reviewer — 5 file(s): internal/client/client.go, internal/client/client_test.go, shortcuts/common/types.go … +- @qa-reviewer — 2 file(s): internal/client/client_test.go, shortcuts/repo/repo_test.go +- @maintainer — 2 file(s): internal/i18n/locales/en-US.json, internal/i18n/locales/zh-CN.json + +### 💡 Advisory (1) +> 建议性提示，不参与评分、不改变裁决。 +- `large_pr_files`: 变更 10 个文件，达到建议拆分阈值 10（未触发硬门禁，建议拆分为更小的 PR 以便审查） + +### Next steps +1. 满足合并门禁；如策略开启 auto_merge 且操作者带 --apply，可执行合并 +--- +*Generated by gitlink-gatekeeper · policy-as-code PR gate · re-run after changes* diff --git a/examples/workflows/pr-quality-gatekeeper/scripts/gatekeeper_workflow.py b/examples/workflows/pr-quality-gatekeeper/scripts/gatekeeper_workflow.py index 601fc86..ccedeea 100644 --- a/examples/workflows/pr-quality-gatekeeper/scripts/gatekeeper_workflow.py +++ b/examples/workflows/pr-quality-gatekeeper/scripts/gatekeeper_workflow.py @@ -84,6 +84,12 @@ DEFAULT_POLICY: dict[str, Any] = { "auto_merge": False, "merge_method": "squash", }, + # 咨询标记（advisory）：默认全关；开启后仅在评分卡中给出建议性提示， + # 不参与评分、不改变裁决。向后兼容——旧策略无此段时等同关闭。 + "advisory_flags": { + "enabled": False, + "large_pr_files": 0, # 变更文件数 ≥ 此值则提示拆分（0=不启用；建议 < hard_gates.max_changed_files） + }, } VERDICT_EMOJI = {"PASS": "✅", "REQUEST_CHANGES": "❌", "COMMENT": "💬"} @@ -641,6 +647,35 @@ def evaluate_hard_gates(inp: ScoreInput, policy: dict[str, Any]) -> list[dict[st return failures +def evaluate_advisory_flags(inp: ScoreInput, policy: dict[str, Any]) -> list[dict[str, str]]: + """咨询标记（advisory）——建议性、不参与评分、不改变裁决。 + + 与硬门禁互补的「软」一层：硬门禁是真牙齿（命中即 REQUEST_CHANGES）， + 咨询标记只在评分卡里给出提示，把治理建议（如积压仓库里的超体量 PR + 应拆分但不应阻断）留给人工判断。 + + 设计约束： + - 默认全关、向后兼容（策略无 advisory_flags 段时返回空，行为与旧版完全一致）； + - 确定性——仅消费已采集的 PR 数据，不依赖当前时间等非确定性输入， + 保持「同输入 → 同输出」，与评分主逻辑一致可复算。 + """ + cfg = policy.get("advisory_flags") or {} + if not cfg.get("enabled"): + return [] + flags: list[dict[str, str]] = [] + threshold = int(cfg.get("large_pr_files", 0) or 0) + n_files = len(inp.changed_files) + if threshold > 0 and n_files >= threshold: + flags.append( + { + "flag": "large_pr_files", + "detail": f"变更 {n_files} 个文件，达到建议拆分阈值 {threshold}" + f"（未触发硬门禁，建议拆分为更小的 PR 以便审查）", + } + ) + return flags + + def decide_verdict( total: int, hard_gate_failed: bool, policy: dict[str, Any] ) -> str: @@ -667,6 +702,7 @@ def render_scorecard( policy_label: str, routing: dict[str, Any] | None, tracking_issue: Any = None, + advisory: list[dict[str, str]] | None = None, ) -> str: emoji = VERDICT_EMOJI[verdict] total = dims["total"] @@ -719,6 +755,13 @@ def render_scorecard( lines.append(f"- `{f['gate']}`: {f['detail']}") lines.append("") + if advisory: + lines.append(f"### 💡 Advisory ({len(advisory)})") + lines.append("> 建议性提示，不参与评分、不改变裁决。") + for a in advisory: + lines.append(f"- `{a['flag']}`: {a['detail']}") + lines.append("") + must = [f for f in inp.findings if f.severity in ("blocker", "major")] should = [f for f in inp.findings if f.severity == "minor"] nits = [f for f in inp.findings if f.severity == "nit"] @@ -1184,9 +1227,13 @@ def main(argv: list[str] | None = None) -> int: ) dims = score_dimensions(inp, policy) failures = evaluate_hard_gates(inp, policy) + advisory = evaluate_advisory_flags(inp, policy) verdict = decide_verdict(dims["total"], bool(failures), policy) - scorecard = render_scorecard(inp, dims, failures, verdict, policy_label, routing) - print(f" Score: {dims['total']}/100 · 硬门禁失败 {len(failures)} 项 · 裁决: {verdict}") + scorecard = render_scorecard( + inp, dims, failures, verdict, policy_label, routing, advisory=advisory + ) + adv_note = f" · 咨询标记 {len(advisory)} 条" if advisory else "" + print(f" Score: {dims['total']}/100 · 硬门禁失败 {len(failures)} 项 · 裁决: {verdict}{adv_note}") print() # 落盘本地产物（无论 dry-run 与否都生成，便于复核 / 验证记录） @@ -1229,7 +1276,7 @@ def main(argv: list[str] | None = None) -> int: # 把编号补进评分卡 Next steps 后重新落盘（评论将带上回链） scorecard = render_scorecard( inp, dims, failures, verdict, policy_label, routing, - tracking_issue=tracking_issue_no, + tracking_issue=tracking_issue_no, advisory=advisory, ) scorecard_path.write_text(scorecard + "\n", encoding="utf-8") @@ -1289,6 +1336,7 @@ def main(argv: list[str] | None = None) -> int: "scores": {k: v for k, v in dims.items() if k != "total"}, "total": dims["total"], "hard_gate_failures": failures, + "advisory_flags": advisory, "verdict": verdict, "verdict_label": verdict_label, "tracking_issue": tracking_issue_no, diff --git a/examples/workflows/pr-quality-gatekeeper/tests/test_scoring.py b/examples/workflows/pr-quality-gatekeeper/tests/test_scoring.py index b5eb763..fd8471b 100644 --- a/examples/workflows/pr-quality-gatekeeper/tests/test_scoring.py +++ b/examples/workflows/pr-quality-gatekeeper/tests/test_scoring.py @@ -244,5 +244,72 @@ class TestVerdictBoundaries(unittest.TestCase): self.assertEqual(decide_verdict(100, True, DEFAULT_POLICY), "REQUEST_CHANGES") +evaluate_advisory_flags = gw.evaluate_advisory_flags + + +def _copy_policy(**advisory): + """深拷一份默认策略并覆盖 advisory_flags，避免污染共享 dict。""" + p = _json.loads(_json.dumps(DEFAULT_POLICY)) + p["advisory_flags"] = advisory + return p + + +class TestAdvisoryFlags(unittest.TestCase): + """咨询标记（advisory）：加性、默认关闭、不参与评分/裁决、确定性。""" + + def _input(self, n_files: int) -> ScoreInput: + # 一个干净的 PASS 输入（无 findings、有测试、CI 过、描述充分、关联 issue） + return _build( + pr_id="1", title="feat: x", desc_len=60, linked_issue=True, + n_files=n_files, src=1, tests=1, commits=(1, 1), ci="passing", + findings_counts={}, + ) + + def test_default_off_is_backward_compatible(self): + # 默认策略未显式开启 → 返回空，行为与旧版完全一致 + self.assertEqual(evaluate_advisory_flags(self._input(200), DEFAULT_POLICY), []) + + def test_missing_section_returns_empty(self): + # 策略完全没有 advisory_flags 段（旧策略）→ 不报错、返回空 + p = _json.loads(_json.dumps(DEFAULT_POLICY)) + p.pop("advisory_flags", None) + self.assertEqual(evaluate_advisory_flags(self._input(200), p), []) + + def test_enabled_below_threshold_no_flag(self): + p = _copy_policy(enabled=True, large_pr_files=30) + self.assertEqual(evaluate_advisory_flags(self._input(29), p), []) + + def test_enabled_at_threshold_flags(self): + p = _copy_policy(enabled=True, large_pr_files=30) + flags = evaluate_advisory_flags(self._input(30), p) + self.assertEqual(len(flags), 1) + self.assertEqual(flags[0]["flag"], "large_pr_files") + + def test_zero_threshold_disabled_even_if_enabled(self): + # large_pr_files=0 表示该标记不启用，即便 enabled=True + p = _copy_policy(enabled=True, large_pr_files=0) + self.assertEqual(evaluate_advisory_flags(self._input(500), p), []) + + def test_advisory_does_not_change_verdict(self): + # 关键不变量：开 advisory 且触发标记，total 与 verdict 必须与不开时逐位一致 + inp = self._input(50) + base_dims, base_fail, base_verdict = _run(inp) + p = _copy_policy(enabled=True, large_pr_files=30) + adv = evaluate_advisory_flags(inp, p) + self.assertEqual(len(adv), 1) # 确实触发了 + dims = score_dimensions(inp, p) + verdict = decide_verdict(dims["total"], bool(evaluate_hard_gates(inp, p)), p) + self.assertEqual(dims["total"], base_dims["total"]) # 分数不变 + self.assertEqual(verdict, base_verdict) # 裁决不变 + + def test_deterministic(self): + # 同输入两次调用结果完全相同 + p = _copy_policy(enabled=True, large_pr_files=10) + inp = self._input(20) + self.assertEqual( + evaluate_advisory_flags(inp, p), evaluate_advisory_flags(inp, p) + ) + + if __name__ == "__main__": unittest.main(verbosity=2) diff --git a/skills/gitlink-gatekeeper/REFERENCE.md b/skills/gitlink-gatekeeper/REFERENCE.md index b19a54f..b36e665 100644 --- a/skills/gitlink-gatekeeper/REFERENCE.md +++ b/skills/gitlink-gatekeeper/REFERENCE.md @@ -95,6 +95,17 @@ | `auto_merge` | bool | `false` | **仅当 `true` 且裁决=PASS 且显式 `--apply` 时才合并** | | `merge_method` | enum | `squash` | `merge` \| `rebase` \| `squash` | +### 1.9 `advisory_flags`（可选 · 咨询标记 · 默认全关） + +与硬门禁互补的「软」一层。硬门禁是真牙齿（命中即 `REQUEST_CHANGES`）；咨询标记**只在评分卡中给出建议性提示，不参与评分、不改变裁决**，把治理建议留给人工判断。默认关闭、**向后兼容**（策略无此段时等同关闭，旧行为不变）；**确定性**——仅消费已采集的 PR 数据，不依赖当前时间等非确定性输入，与评分主逻辑同样可复算。 + +| 字段 | 类型 | 默认值 | 含义 | +|------|------|--------|------| +| `enabled` | bool | `false` | 总开关，默认关闭 | +| `large_pr_files` | int | `0` | 变更文件数 ≥ 此值则提示拆分（`0`=不启用；建议 `< hard_gates.max_changed_files`，使其落在「偏大但未触硬门禁」区间） | + +> 典型用途：在 PR 积压的活跃仓库里，对「超体量但未触硬门禁」的 PR 提示拆分而**不阻断**——既给出治理信号，又不把判断权从人手里夺走。命中的标记渲染在评分卡的 `### 💡 Advisory` 小节，并写入 `summary.json` 的 `advisory_flags` 字段。 + --- ## 2. 评分算法规格（确定性，可复现） diff --git a/skills/gitlink-gatekeeper/examples/gatekeeper.yaml b/skills/gitlink-gatekeeper/examples/gatekeeper.yaml index 92ff8a6..7930e9a 100644 --- a/skills/gitlink-gatekeeper/examples/gatekeeper.yaml +++ b/skills/gitlink-gatekeeper/examples/gatekeeper.yaml @@ -48,3 +48,12 @@ behavior: apply_label: true # 按裁决打标签 auto_merge: false # 仅当 true 且裁决=PASS 且显式 --apply 时才合并 merge_method: squash # merge | rebase | squash + +# —— 咨询标记（advisory，可选）—— +# 与硬门禁互补的「软」一层：硬门禁命中即 REQUEST_CHANGES（真牙齿）； +# 咨询标记只在评分卡里提示、不参与评分、不改变裁决（建议留给人工判断）。 +# 默认关闭、向后兼容；确定性——仅用已采集数据，不依赖当前时间等非确定性输入。 +# 典型用途：PR 积压的活跃仓库里，对「超体量但未触硬门禁」的 PR 提示拆分而不阻断。 +advisory_flags: + enabled: false # 总开关，默认关闭 + large_pr_files: 0 # 变更文件数 ≥ 此值则提示拆分（0=不启用；建议 < hard_gates.max_changed_files）