2557c4c58e
- BENCHMARKS.md / joulecontract diff: replace "16 tests" / "64 tests"
count phrasings with shape descriptions, per the no-vanity-metrics rule.
- check-no-vanity-metrics.sh: require the count to be a standalone number
(start-of-line or a non-identifier char before it), so an identifier like
"JR-2 tests" — where "tests" is a verb — is no longer flagged. Genuine
counts (" 16 tests", "64 tests") still match.
Scanner passes clean (exit 0).
Co-Authored-By: Claude Opus 4.8 <[email protected]>
97 lines
3.6 KiB
Shell
Executable file
97 lines
3.6 KiB
Shell
Executable file
#!/usr/bin/env bash
|
|
#
|
|
# check-no-vanity-metrics.sh
|
|
#
|
|
# Blocks codebase-state numbers from landing in Markdown surfaces.
|
|
# Run by the no-vanity-metrics CI workflow and (optionally) by a
|
|
# local pre-commit hook (`scripts/install-hooks.sh`).
|
|
#
|
|
# Why: counts of crates / tests / LOC / adapters drift on every code
|
|
# change and create permanent sync debt between the website, the
|
|
# GitHub READMEs, and any memory document. The open-standards posture
|
|
# is "describe the shape, not the size."
|
|
#
|
|
# Opt-out: append `<!-- allow-vanity: <reason> -->` to the END of a
|
|
# line whose number is genuinely structural (a fact about an EXTERNAL
|
|
# spec, not a count of TX-side artefacts). The marker MUST give a
|
|
# reason. Misuse is caught by a separate audit.
|
|
#
|
|
# What gets flagged (the regex catches every count we've actually had
|
|
# to fix; tuned to avoid false positives on codec / RFC / spec
|
|
# references):
|
|
#
|
|
# - "N crates", "N+ crates"
|
|
# - "N tests passing", "N tests"
|
|
# - "N LOC", "~Nk LOC", "Nk LOC"
|
|
# - "N+ samplers"
|
|
# - "N+ adapters"
|
|
# - "N+ backends"
|
|
# - "N+ integrations"
|
|
#
|
|
# Identity-class numbers (spec versions like "JCR-1 v1", algorithm
|
|
# names like "Ed25519", RFC numbers, codec generation tags like
|
|
# "MPEG-4 Part 2") are NOT matched by the regex.
|
|
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
|
cd "$REPO_ROOT"
|
|
|
|
# The vanity-metric pattern. Anchored to the keyword set with a
|
|
# trailing punctuation/space/EOL requirement so it doesn't fire on
|
|
# "Test-vector", "lockstep", "MPEG-4", RFC numbers, or version tags.
|
|
# The leading (^|[^[:alnum:]_-]) requires the count to be a standalone
|
|
# number, so a digit inside an identifier (e.g. "JR-2 tests", where
|
|
# "tests" is a verb) is not mistaken for a metric.
|
|
PATTERN='(^|[^[:alnum:]_-])[0-9][0-9,]*[+kKMG]?[+ ]*([Cc]rates?|[Tt]ests?|LOC|loc|[Ss]amplers?|[Aa]dapters?|[Bb]ackends?|[Ii]ntegrations?)([ .,;:!?)$]|$)'
|
|
|
|
# Build the file list — marketing / spec / governance surfaces only.
|
|
# Historical progress notes (docs/*-progress.md), changelogs, and
|
|
# external corpora are EXEMPT by design (they're snapshots, not the
|
|
# message we're keeping in sync). Research and notes dirs also exempt.
|
|
# Portable: read into an array without `mapfile` so macOS bash 3.2 works.
|
|
files=()
|
|
while IFS= read -r f; do
|
|
case "$f" in
|
|
*-progress.md|*/docs/*-progress.md|*/CHANGELOG.md|*/conformance/*|*/research/*|*/notes/*|*/notes-*/*)
|
|
continue ;;
|
|
esac
|
|
files+=("$f")
|
|
done < <(git ls-files '*.md')
|
|
|
|
if [ ${#files[@]} -eq 0 ]; then
|
|
echo "no Markdown files to scan"
|
|
exit 0
|
|
fi
|
|
|
|
# Scan. A hit-line is exempted iff it ends with an `allow-vanity:`
|
|
# marker with a non-empty reason.
|
|
hits=0
|
|
while IFS= read -r line; do
|
|
# `line` is "path:lineno:content"
|
|
content="${line#*:}"
|
|
content="${content#*:}"
|
|
if echo "$content" | grep -qE 'allow-vanity:[[:space:]]*[^[:space:]]'; then
|
|
continue
|
|
fi
|
|
if [ $hits -eq 0 ]; then
|
|
echo "::error::Vanity metric(s) found — see scripts/check-no-vanity-metrics.sh for the policy and opt-out marker."
|
|
echo ""
|
|
fi
|
|
echo "$line"
|
|
hits=$((hits + 1))
|
|
done < <(grep -nE "$PATTERN" "${files[@]}" 2>/dev/null || true)
|
|
|
|
if [ $hits -gt 0 ]; then
|
|
echo ""
|
|
echo "Total hits: $hits"
|
|
echo ""
|
|
echo "Fix: replace the count with a shape description"
|
|
echo " (e.g. 'a multi-crate Rust workspace', 'covered by a test suite',"
|
|
echo " 'the diffusion sampler catalog'), OR if the number is genuinely"
|
|
echo " structural (describes an external spec, not TX state) append"
|
|
echo " '<!-- allow-vanity: <reason> -->' to the end of the line."
|
|
exit 1
|
|
fi
|
|
|
|
echo "✓ No vanity metrics found across $(echo "${files[@]}" | wc -w | tr -d ' ') Markdown files"
|