ba986de6d6
Some checks failed
repository checks / lockfiles, advisories, tests in CI, workflow hygiene, spec and section references, public text (push) Has been cancelled
WAI breaking changes and the conformance runner / WAI breaking · does this change touch WAI (push) Successful in 12s
WAI breaking changes and the conformance runner / WAI breaking · wire, record and API against main · linux (push) Successful in 46s
WAI breaking changes and the conformance runner / WAI breaking and conform · their tests, and the independent verifiers through wai conform · linux (push) Successful in 3m25s
The workflow step ran scripts/check-frozen-rules.sh from the checkout, so a commit that added `exit 0` to it, together with a coordinated change to a rule, its text and its verifier, passed that run and every later one, though the history still showed the change. The step now takes the runner as the first commit of the first-parent history to add it held, and runs it as well as the checkout's copy; the job fails if either does. A runner edited, emptied or removed, in the rule's commit or an earlier one, hides nothing, and the check goes red on that commit and on the ones after it. A branch made before the runner takes it from the pull request's base, and a history that never held it fails. What is left is the step itself, which is read from the commit under test: a commit that edits it silences the check from then on, until that edit is undone, and the diff in the history records it. The record's page and its changelog entry now say this, and limit "does not hide" to the two scripts. They also say that a merge is read from its first parent, so one that puts another line of commits first changes the history the check reads. scripts/check-frozen-rules-selftest.sh runs the step's own text, from its marker in the workflow, in its scratch repository. It requires each of those cases, and it records the first-parent reading: a merge with main as its second parent pins its own line's rule, and only the comparison with a base shows it. Co-Authored-By: Claude Opus 5.5 <[email protected]>
202 lines
16 KiB
Shell
Executable file
202 lines
16 KiB
Shell
Executable file
#!/usr/bin/env bash
|
|
#
|
|
# check-frozen-rules-selftest.sh — a rule file stays as it first reached the history, and a
|
|
# change to it fails on every later run, whatever happened to the run of its own push.
|
|
#
|
|
# scripts/check-frozen-rules.py pins each research record's rule file
|
|
# (wai/research/<record>/*-rule*.json) at what the first commit of the first-parent history to
|
|
# add it held, scripts/check-frozen-rules.sh runs it as it first reached the history and as head
|
|
# holds it, and the step of .github/workflows/repo-checks.yml runs that runner the same way. This
|
|
# builds a scratch repository laid out as this one is and requires:
|
|
# a change elsewhere, a change to a record's other files and a later rule in a new file pass;
|
|
# one byte of the rule changed fails, and so does the rule changed together with the digest its
|
|
# text quotes and the one its verifier holds; the same change still fails with unrelated commits
|
|
# on top, the run of its own push being gone; removing, renaming or changing the mode of the rule
|
|
# fails, and so does removing it and adding other content at the path; a rule that entered by a
|
|
# rename is pinned; restoring a changed rule passes; a
|
|
# rule revised on its own branch fails until the branch adds it once, and a merge commit brings
|
|
# only what it merged; a change landed by a merge fails; a branch made before the rule reached
|
|
# main fails at a pull request if it holds another file at the rule's path; the script changed
|
|
# or removed together with the rule fails through its first copy, and so does the script changed
|
|
# in an earlier commit than the rule, while a later copy's failure fails too; a branch made before
|
|
# the check reached main runs it from its base; through the workflow step's own text, the runner
|
|
# changed, emptied or removed together with the rule fails, on that commit and on later ones, and
|
|
# so does the runner changed earlier than the rule, removed and added back, or added first on a
|
|
# side line, while a later runner's failure fails too and a branch made before the runner takes it
|
|
# from its base; a merge is read from its first parent, and only the base shows another rule it
|
|
# brought; and a shallow clone, a commit the clone lacks and a wrong command line never pass.
|
|
set -uo pipefail
|
|
|
|
here=$(cd "$(dirname "$0")" && pwd)
|
|
tmp=$(mktemp -d)
|
|
trap 'rm -rf "$tmp"' EXIT
|
|
R="$tmp/r"
|
|
mkdir -p "$R/scripts" "$R/wai/research/rec" "$R/wai/tools" "$R/jcp"
|
|
git -C "$R" init -q -b main
|
|
git -C "$R" config user.email [email protected]
|
|
git -C "$R" config user.name t
|
|
echo a > "$R/jcp/SPEC.md"
|
|
printf '{"format": "x/1"}\n' > "$R/wai/research/rec/run.json"
|
|
git -C "$R" add -A; git -C "$R" commit -qm "before the check"
|
|
NOCHECK=$(git -C "$R" rev-parse HEAD)
|
|
cp "$here/check-frozen-rules.py" "$here/check-frozen-rules.sh" "$R/scripts/"
|
|
git -C "$R" add -A; git -C "$R" commit -qm "before the rule"
|
|
PRE=$(git -C "$R" rev-parse HEAD)
|
|
# The rule reaches main, with its digest quoted by the text and held by the verifier.
|
|
printf '{"format": "rule/1", "go": -10}\n' > "$R/wai/research/rec/decisive-rule.json"
|
|
sha() { python3 -I -c 'import hashlib, sys; print(hashlib.sha256(open(sys.argv[1], "rb").read()).hexdigest())' "$1"; }
|
|
d=$(sha "$R/wai/research/rec/decisive-rule.json")
|
|
printf 'Its SHA-256 is `%s`.\n' "$d" > "$R/wai/research/rec.md"
|
|
printf 'RULE = "%s"\n' "$d" > "$R/wai/tools/verify.py"
|
|
git -C "$R" add -A; git -C "$R" commit -qm "the rule"
|
|
BASE=$(git -C "$R" rev-parse HEAD)
|
|
|
|
fails=0
|
|
expect() { # expect <0|1|2> <what> <text the output must hold or ""> -- <command...>
|
|
local want=$1 what=$2 needle=$3 out rc
|
|
shift 4
|
|
out=$(cd "$R" && "$@" 2>&1); rc=$?
|
|
if [ $rc -ne "$want" ]; then echo "FAIL $what: exit $rc, not $want"; printf '%s\n' "$out" | sed 's/^/ /'; fails=$((fails+1)); return; fi
|
|
if [ -n "$needle" ] && ! grep -qF -- "$needle" <<<"$out"; then echo "FAIL $what: the output does not say '$needle'"; printf '%s\n' "$out" | sed 's/^/ /'; fails=$((fails+1)); return; fi
|
|
echo "ok $what"
|
|
}
|
|
branch() { git -C "$R" checkout -q -B "$1" "${2:-$BASE}"; }
|
|
commit() { git -C "$R" add -A; git -C "$R" commit -q -m "${1:-change}"; git -C "$R" rev-parse HEAD; }
|
|
C="python3 -I scripts/check-frozen-rules.py"
|
|
L="$here/check-frozen-rules.sh"
|
|
rule="$R/wai/research/rec/decisive-rule.json"
|
|
runner="$R/scripts/check-frozen-rules.sh"
|
|
# The workflow step's own text, from its marker to the end of the step, run as the job runs it.
|
|
awk '/# frozen rules: the runner as it first reached/ { on = 1 } on && !NF { exit } on' \
|
|
"$here/../.github/workflows/repo-checks.yml" | sed 's/^ *//' > "$tmp/step.sh"
|
|
S="bash -euo pipefail $tmp/step.sh"
|
|
|
|
branch outside; echo b > "$R/jcp/SPEC.md"; H=$(commit)
|
|
expect 0 "a change outside the records passes" "the 1 rule file(s) of" -- $C "$H"
|
|
branch record; printf '{"format": "x/1", "more": 1}\n' > "$R/wai/research/rec/run.json"; echo more >> "$R/wai/research/rec.md"; H=$(commit)
|
|
expect 0 "a change to a record's other files passes" "as they first reached it" -- $C "$H"
|
|
branch second; printf '{"format": "rule/1", "go": -12}\n' > "$R/wai/research/rec/decisive-rule-2.json"; H=$(commit)
|
|
expect 0 "a later rule in a new file passes, and is pinned" "the 2 rule file(s) of" -- $C "$H"
|
|
|
|
branch byte; printf '{"format": "rule/1", "go": -11}\n' > "$rule"; H=$(commit)
|
|
expect 1 "one byte of the rule changed fails, naming it" "wai/research/rec/decisive-rule.json is changed" -- $C "$H"
|
|
branch coordinated; printf '{"format": "rule/1", "go": -5}\n' > "$rule"
|
|
d=$(sha "$rule")
|
|
printf 'Its SHA-256 is `%s`.\n' "$d" > "$R/wai/research/rec.md"; printf 'RULE = "%s"\n' "$d" > "$R/wai/tools/verify.py"; H=$(commit)
|
|
expect 1 "the rule changed with the digest its text quotes and the one its verifier holds fails" "is changed" -- $C "$H"
|
|
echo b > "$R/jcp/SPEC.md"; H=$(commit)
|
|
expect 1 "that change still fails one unrelated commit later, its own push's run gone" "is changed" -- $C "$H"
|
|
echo c > "$R/jcp/SPEC.md"; commit >/dev/null; echo d > "$R/jcp/SPEC.md"; H=$(commit)
|
|
expect 1 "and on every later run" "is changed" -- $C "$H"
|
|
branch removed; git -C "$R" rm -q "$rule"; H=$(commit)
|
|
expect 1 "removing the rule fails" "is removed or renamed" -- $C "$H"
|
|
branch renamed; git -C "$R" mv "$rule" "$R/wai/research/rec/old-rule.json"; H=$(commit)
|
|
expect 1 "renaming the rule fails" "decisive-rule.json is removed or renamed" -- $C "$H"
|
|
branch mode; chmod +x "$rule"; git -C "$R" update-index --chmod=+x wai/research/rec/decisive-rule.json; H=$(commit)
|
|
expect 1 "changing the rule's mode fails" "changes its mode from 100644" -- $C "$H"
|
|
branch readded; git -C "$R" rm -q "$rule"; commit >/dev/null; printf '{"format": "rule/1", "go": -1}\n' > "$rule"; H=$(commit)
|
|
expect 1 "removing the rule and adding other content at its path fails" "is changed" -- $C "$H"
|
|
branch renamedin; git -C "$R" mv "$R/wai/research/rec/run.json" "$R/wai/research/rec/moved-rule.json"; commit >/dev/null; printf '{"format": "x/1", "more": 2}\n' > "$R/wai/research/rec/moved-rule.json"; H=$(commit)
|
|
expect 1 "a rule that entered by renaming another file of the record is pinned too" "moved-rule.json is changed" -- $C "$H"
|
|
branch restored; printf '{"format": "rule/1", "go": -11}\n' > "$rule"; commit >/dev/null; printf '{"format": "rule/1", "go": -10}\n' > "$rule"; H=$(commit)
|
|
expect 0 "a changed rule restored to what first reached the history passes" "as they first reached it" -- $C "$H"
|
|
|
|
# A rule on its own branch. Its commits are the history once they land, so it enters once.
|
|
branch revised; printf '{"format": "rule/1", "go": -9}\n' > "$R/wai/research/rec/next-rule.json"; commit >/dev/null
|
|
printf '{"format": "rule/1", "go": -8}\n' > "$R/wai/research/rec/next-rule.json"; H=$(commit)
|
|
REVISED=$H
|
|
expect 1 "a rule revised by a later commit of its branch fails" "next-rule.json is changed" -- $C "$H"
|
|
branch once; printf '{"format": "rule/1", "go": -8}\n' > "$R/wai/research/rec/next-rule.json"; H=$(commit)
|
|
expect 0 "the same rule added once, in its final form, passes" "the 2 rule file(s) of" -- $C "$H"
|
|
|
|
# Landing by a merge commit: the first parent sees the merge alone.
|
|
branch merged; git -C "$R" merge -q --no-ff --no-edit "$REVISED" >/dev/null; H=$(git -C "$R" rev-parse HEAD)
|
|
expect 0 "a merge brings the rule as it merged it, and the branch's revisions are not on the first parent" "the 2 rule file(s) of" -- $C "$H"
|
|
printf '{"format": "rule/1", "go": -7}\n' > "$R/wai/research/rec/next-rule.json"; H=$(commit)
|
|
expect 1 "a rule that came in by a merge is pinned as the merge brought it" "next-rule.json is changed" -- $C "$H"
|
|
branch sidechange; printf '{"format": "rule/1", "go": -3}\n' > "$rule"; SIDE=$(commit)
|
|
branch landed; git -C "$R" merge -q --no-ff --no-edit "$SIDE" >/dev/null; H=$(git -C "$R" rev-parse HEAD)
|
|
expect 1 "a change that lands by a merge commit fails" "decisive-rule.json is changed" -- $C "$H"
|
|
branch mergedin "$PRE"; echo b > "$R/jcp/SPEC.md"; commit >/dev/null; git -C "$R" merge -q --no-edit "$BASE" >/dev/null; printf '{"format": "rule/1", "go": -1}\n' > "$rule"; H=$(commit)
|
|
expect 1 "a branch that merges main in cannot change main's rule" "decisive-rule.json is changed" -- $C "$H"
|
|
|
|
# A pull request from a branch made before the rule reached main.
|
|
branch stale "$PRE"; echo b > "$R/jcp/SPEC.md"; H=$(commit)
|
|
expect 0 "a branch made before the rule, without it, passes" "the 0 rule file(s) of" -- $C "$H" "$BASE"
|
|
branch stalerule "$PRE"; printf '{"format": "rule/1", "go": -2}\n' > "$rule"; H=$(commit)
|
|
expect 0 "another file at the rule's path passes without the base, whose history it is not on" "the 1 rule file(s) of" -- $C "$H"
|
|
expect 1 "and fails with the base" "is another file than the one the base" -- $C "$H" "$BASE"
|
|
|
|
# The check itself. Head's copy has been weakened in these; the first copy has not.
|
|
branch weakened; sed -i.bak 's/-rule\[/-rules-never[/' "$R/scripts/check-frozen-rules.py"; rm -f "$R/scripts/check-frozen-rules.py.bak"
|
|
printf '{"format": "rule/1", "go": -6}\n' > "$rule"; H=$(commit)
|
|
expect 0 "the rule and the check's pattern changed in one commit pass the check as head holds it" "the 0 rule file(s) of" -- $C "$H"
|
|
expect 1 "and fail through the check as it first reached the history" "decisive-rule.json is changed" -- bash "$L" "$H"
|
|
branch weakenedfirst; sed -i.bak 's/-rule\[/-rules-never[/' "$R/scripts/check-frozen-rules.py"; rm -f "$R/scripts/check-frozen-rules.py.bak"; commit >/dev/null
|
|
printf '{"format": "rule/1", "go": -6}\n' > "$rule"; H=$(commit)
|
|
expect 1 "the check weakened in an earlier commit than the rule's change fails the same way" "decisive-rule.json is changed" -- bash "$L" "$H"
|
|
branch deleted; git -C "$R" rm -q scripts/check-frozen-rules.py; printf '{"format": "rule/1", "go": -6}\n' > "$rule"; H=$(commit)
|
|
expect 1 "the check removed with the rule's change fails" "decisive-rule.json is changed" -- bash "$L" "$H"
|
|
branch nochange; echo b > "$R/jcp/SPEC.md"; H=$(commit)
|
|
expect 0 "both copies pass a change that leaves the rule alone" " holds it" -- bash "$L" "$H"
|
|
branch stricter; printf 'import sys\nprint("a later copy finds a fault")\nsys.exit(1)\n' > "$R/scripts/check-frozen-rules.py"; H=$(commit)
|
|
expect 1 "a later copy of the check is run too, and its failure fails the runner" "a later copy finds a fault" -- bash "$L" "$H"
|
|
branch stalerule2 "$PRE"; printf '{"format": "rule/1", "go": -2}\n' > "$rule"; H=$(commit)
|
|
expect 1 "both copies take the base" "is another file than the one the base" -- bash "$L" "$H" "$BASE"
|
|
|
|
# A pull request from a branch made before the check reached main: it holds no copy of the check.
|
|
branch nocheck "$NOCHECK"; echo b > "$R/jcp/SPEC.md"; H=$(commit)
|
|
expect 0 "a branch made before the check runs it from the base" "the 0 rule file(s) of" -- bash "$L" "$H" "$BASE"
|
|
branch nocheckrule "$NOCHECK"; printf '{"format": "rule/1", "go": -2}\n' > "$rule"; H=$(commit)
|
|
expect 1 "and fails there too if it holds another file at the rule's path" "is another file than the one the base" -- bash "$L" "$H" "$BASE"
|
|
|
|
# The workflow step. The runner as head holds it has been weakened in these; the first has not.
|
|
weak() { mkdir -p "$R/scripts"; printf '#!/usr/bin/env bash\nexit 0\n' > "$runner"; }
|
|
changed() { printf '{"format": "rule/1", "go": -6}\n' > "$rule"; }
|
|
branch runnerweak; weak; changed; H=$(commit)
|
|
expect 0 "the rule and the runner changed in one commit pass the runner as head holds it" "" -- bash scripts/check-frozen-rules.sh "$H"
|
|
expect 1 "and fail through the workflow step, as it first reached the history" "decisive-rule.json is changed" -- $S "$H"
|
|
echo b > "$R/jcp/SPEC.md"; H=$(commit)
|
|
expect 1 "and still fail one unrelated commit later" "decisive-rule.json is changed" -- $S "$H"
|
|
echo c > "$R/jcp/SPEC.md"; H=$(commit)
|
|
expect 1 "and two" "decisive-rule.json is changed" -- $S "$H"
|
|
branch runnerempty; : > "$runner"; changed; H=$(commit)
|
|
expect 1 "the runner emptied with the rule's change fails" "decisive-rule.json is changed" -- $S "$H"
|
|
branch runnerearlier; weak; commit >/dev/null; changed; H=$(commit)
|
|
expect 1 "the runner weakened in an earlier commit than the rule's change fails the same way" "decisive-rule.json is changed" -- $S "$H"
|
|
branch runnergone; git -C "$R" rm -q scripts/check-frozen-rules.sh; changed; H=$(commit)
|
|
expect 1 "the runner removed with the rule's change fails" "decisive-rule.json is changed" -- $S "$H"
|
|
branch runnerback; git -C "$R" rm -q scripts/check-frozen-rules.sh; commit >/dev/null; weak; changed; H=$(commit)
|
|
expect 1 "the runner removed, then added back weakened with the rule's change, fails" "decisive-rule.json is changed" -- $S "$H"
|
|
branch runnerside "$NOCHECK"; weak; SIDE=$(GIT_AUTHOR_DATE=2001-01-01T00:00:00Z GIT_COMMITTER_DATE=2001-01-01T00:00:00Z commit)
|
|
branch runnermerged; git -C "$R" merge -q --no-ff -X ours --no-edit "$SIDE" >/dev/null; weak; changed; H=$(commit)
|
|
expect 1 "a weakened runner added earlier on a side line is not the first copy" "decisive-rule.json is changed" -- $S "$H"
|
|
branch runnerok; echo b > "$R/jcp/SPEC.md"; H=$(commit)
|
|
expect 0 "the step passes a change that leaves the rule alone" "as this checkout holds it" -- $S "$H"
|
|
branch runnerstrict; printf 'echo "a later runner finds a fault"\nexit 1\n' > "$runner"; H=$(commit)
|
|
expect 1 "a later runner is run too, and its failure fails the step" "a later runner finds a fault" -- $S "$H"
|
|
branch norunner "$NOCHECK"; echo b > "$R/jcp/SPEC.md"; H=$(commit)
|
|
git -C "$R" checkout -q main
|
|
expect 0 "a branch made before the runner takes it from the base, in a checkout that holds it" "as this checkout holds it" -- $S "$H" "$BASE"
|
|
expect 1 "and a history that never held it, with no base, fails saying so" "adds scripts/check-frozen-rules.sh" -- $S "$H"
|
|
expect 1 "a commit the clone lacks fails the step, saying so" "adds scripts/check-frozen-rules.sh" -- $S 0000000000000000000000000000000000000000
|
|
|
|
# A merge is read from its first parent: a line that held another rule pins that rule.
|
|
branch swapped "$PRE"; printf '{"format": "rule/1", "go": -2}\n' > "$rule"; commit >/dev/null
|
|
git -C "$R" merge -q -X ours --no-edit "$BASE" >/dev/null; H=$(git -C "$R" rev-parse HEAD)
|
|
expect 0 "a merge with main as its second parent is read from its first, and pins that line's rule" "as they first reached it" -- bash "$L" "$H"
|
|
expect 1 "and the base shows it" "is another file than the one the base" -- bash "$L" "$H" "$BASE"
|
|
|
|
# It never passes unchecked.
|
|
git -C "$R" checkout -q main
|
|
expect 1 "a commit the clone lacks fails, saying so" "is not a commit of this clone" -- $C 0000000000000000000000000000000000000000
|
|
expect 1 "a base the clone lacks fails, saying so" "is not a commit of this clone" -- $C "$BASE" 0000000000000000000000000000000000000000
|
|
expect 2 "a wrong command line fails" "usage" -- $C
|
|
expect 2 "and the runner's" "usage" -- bash "$L"
|
|
expect 1 "a history that never held the check fails through the runner" "the check cannot run from it" -- bash "$L" "$NOCHECK"
|
|
git clone -q --depth 1 "file://$R" "$tmp/shallow" 2>/dev/null
|
|
out=$(cd "$tmp/shallow" && python3 -I "$here/check-frozen-rules.py" "$(git -C "$tmp/shallow" rev-parse HEAD)" 2>&1); rc=$?
|
|
if [ $rc -eq 1 ] && grep -qF "this clone is shallow" <<<"$out"; then echo "ok a shallow clone fails, saying so"; else echo "FAIL a shallow clone: exit $rc"; printf '%s\n' "$out" | sed 's/^/ /'; fails=$((fails+1)); fi
|
|
|
|
if [ $fails -ne 0 ]; then echo "check-frozen-rules-selftest: $fails failure(s)"; exit 1; fi
|
|
echo "check-frozen-rules-selftest: the check passes what it should and fails each way it should."
|