Transaction-Science/open-standards
Watch
1
0
Fork
You've already forked open-standards
 0
Code Issues Pull requests Projects Releases 15 Packages Wiki Activity Actions 87
open-standards/aac/aac-rs/prerender-static.sh
Transaction Science 8b736713dc feat(aac first letters): abbreviation expansion on WebGPU, GPU only 
abbrev.rs is first letters' engine, built like the listener: a
content-named catalogue read and kept for offline use, files downloaded one
at a time into the model store and SHA-256 checked, install / stop /
continue / remove, a first-run check (it finds sentences once), a newer
model kept only once it has run here, and other tabs followed.

- Runtime: the listener's onnxruntime-web 1.30.0 JSPI build, stored once for
  both and never removed while the other's record uses it (listen.rs now
  keeps what first letters uses, and the other way round). Without JSPI,
  it says the browser cannot run it yet. Its own worker and Hugging Face's
  @huggingface/tokenizers 0.2.0 (the latest; Apache-2.0) are vendored and
  pinned by third_party/hf-tokenizers/vendor.sh into
  vendor/hf-tokenizers-0.2.0-aac1/, so a change to the worker never makes
  anyone download the 17 MB runtime again.
- The worker is the prototype's engine: one session, every node on WebGPU,
  session.disable_cpu_ep_fallback; the weights read one tensor at a time
  from the model store (a file stored in two parts read as one); a beam
  search that keeps only words starting with the letters typed, in order,
  with what the partner said in the prompt, and slips to a neighbouring key
  forgiven when asked. A later request supersedes one in flight at its next
  model run.
- fit() is the listener's: a hardware adapter that is not refused, with
  shader-f16, and JSPI.
- Sharing the graphics chip with the sentence helper and the listener: no
  two load at once (each waits for the others' loads and first-run checks;
  the helper looks at both again if it had to wait), and where they take
  turns (every device that does not say it has 8 GB, so every iPad), each
  lets the others go first. It lets go after 10 minutes unused.

prerender-static.sh vendors it (and the runtime, when the listener is not in
the build) and ships ae/ after tools/ae/check_packs.py, or says AAC_AE=none.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
2026-10-02 00:51:00 -04:00

264 lines
15 KiB
Shell
Executable file
Raw Permalink Blame History

#!/usr/bin/env bash
# ============================================================================
# prerender-static.sh — build aac.transaction.science as a self-contained
# static site.
#
# AAC ships as a static CSR-hydrate SPA, but it's a Leptos SSR app: the
# index.html the browser loads is the SSR server's *rendered* output (it carries
# the hydration island markers and the pre-rendered board). A plain
# `cargo leptos build` never emits that index.html, and it doesn't copy the
# PWA assets that live in `static/`. This script closes both gaps and makes the
# result safe to cache forever:
#
# build -> target/site/{pkg/, icons/, symbols/} (cargo-leptos)
# render -> target/site/index.html (capture SSR `/`)
# hash -> pkg/aac-web.<sha256:12>.{js,wasm,css} (and rewrite references)
# vendor -> target/site/vendor/mediapipe-<ver>-<rev>/ (eye gaze, same origin)
# worker -> target/site/sw.js (static/sw.js + precache list)
#
# Output: a self-contained `target/site/` ready to rsync to prod. Deploy it with
# no_cache_bust — the names are already content-hashed, and the service worker
# lists them. Idempotent.
#
# Hosting under a sub-path, or under another name, is a build-time choice; see
# AAC_BASE_PATH, AAC_SW_URL, AAC_APP_TITLE and AAC_HIDE_ENERGY in
# crates/aac-web/src/app.rs. This script builds the root-mounted default.
# ============================================================================
set -euo pipefail
cd "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PORT="${AAC_PRERENDER_PORT:-39207}"
SITE="target/site"
CACHE="${XDG_CACHE_HOME:-$HOME/.cache}/aac-rs"
# wasm-bindgen's CLI must match the crate version exactly (Cargo.toml pins it).
# Use a private copy of the right version rather than the machine's global one.
WBG_VER="$(awk '/^name = "wasm-bindgen"$/{getline; gsub(/version = |"/, ""); print; exit}' Cargo.lock)"
if ! wasm-bindgen --version 2>/dev/null | grep -qx "wasm-bindgen $WBG_VER"; then
WBG_ROOT="$CACHE/wasm-bindgen-$WBG_VER"
if [ ! -x "$WBG_ROOT/bin/wasm-bindgen" ]; then
echo "▸ installing wasm-bindgen-cli $WBG_VER into $WBG_ROOT"
cargo install --quiet --locked --root "$WBG_ROOT" wasm-bindgen-cli --version "$WBG_VER"
fi
export PATH="$WBG_ROOT/bin:$PATH"
fi
echo "▸ cargo leptos build --release"
rm -rf "$SITE/pkg"
cargo leptos build --release
find "$SITE/pkg" -name '*.d.ts' -delete
# Symbol sprites and their licences (cargo-leptos copies public/; a clean copy
# drops sprites of earlier builds, whose names carry another hash).
rm -rf "$SITE/symbols"
cp -R crates/aac-web/public/symbols "$SITE/symbols"
# Language packs for word prediction (data/lang: content-hashed, with their
# licence) under models/lang/, where the app's model store downloads a
# language's pack once, verifies it and keeps it (aac-web lang_packs.rs).
rm -rf "$SITE/models/lang"
mkdir -p "$SITE/models/lang"
cp data/lang/*.bin data/lang/index.json data/lang/LICENSE.md "$SITE/models/lang/"
echo "▸ starting SSR server on 127.0.0.1:$PORT to prerender /"
LOG="$(mktemp)"
LEPTOS_OUTPUT_NAME=aac-web \
LEPTOS_SITE_ROOT="$SITE" \
LEPTOS_SITE_PKG_DIR=pkg \
LEPTOS_SITE_ADDR="127.0.0.1:$PORT" \
LEPTOS_ENV=PROD \
target/release/aac-web > "$LOG" 2>&1 &
SRV=$!
trap 'kill "$SRV" 2>/dev/null || true' EXIT
# Wait for the listener (up to ~30s).
ready=false
for _ in $(seq 1 120); do
if curl -sf "http://127.0.0.1:$PORT/" -o /dev/null 2>/dev/null; then ready=true; break; fi
sleep 0.25
done
if ! $ready; then
echo "✗ SSR server did not come up; log:" >&2; tail -5 "$LOG" >&2; exit 1
fi
echo "▸ capturing rendered / -> $SITE/index.html"
curl -sf "http://127.0.0.1:$PORT/" -o "$SITE/index.html"
kill "$SRV" 2>/dev/null || true
trap - EXIT
# Sanity-check the capture is the hydrating SSR shell, not an error page.
test -s "$SITE/index.html"
grep -q 'mod.hydrate' "$SITE/index.html" || { echo "✗ captured index.html is not the hydrating shell" >&2; exit 1; }
grep -q '<title>AAC' "$SITE/index.html" || { echo "✗ captured index.html missing expected title" >&2; exit 1; }
echo "▸ content-hashing pkg/"
hash12() { shasum -a 256 "$1" | cut -c1-12; }
# Portable in-place sed (BSD and GNU disagree about -i): sed_i <args…> <file>
sed_i() { local f="${!#}"; sed "${@:1:$#-1}" "$f" > "$f.tmp" && mv "$f.tmp" "$f"; }
P="$SITE/pkg"
[ -f "$P/aac-web_bg.wasm" ] || mv "$P/aac-web.wasm" "$P/aac-web_bg.wasm"
W="$(hash12 "$P/aac-web_bg.wasm")"; mv "$P/aac-web_bg.wasm" "$P/aac-web.$W.wasm"
C="$(hash12 "$P/aac-web.css")"; mv "$P/aac-web.css" "$P/aac-web.$C.css"
# The glue may name its wasm; point it at the hashed file before hashing the glue.
sed_i -e "s#aac-web_bg\\.wasm#aac-web.$W.wasm#g" -e "s#aac-web\\.wasm#aac-web.$W.wasm#g" "$P/aac-web.js"
# wasm-bindgen names the snippet directory after the crate, not its contents,
# so a snippet keeps its URL from build to build, and Cloudflare gives every .js
# a 4-hour browser cache whatever Caddy sends. A returning visitor would then
# pair a new aac-web.js with an old snippet, and the app would not start (seen
# on 2026-09-30). Name each snippet directory by its contents too, before
# aac-web.js (whose imports name it) is hashed.
for d in "$P"/snippets/*/; do
d="${d%/}"; name="$(basename "$d")"
S="$(cd "$d" && find . -type f | LC_ALL=C sort | while IFS= read -r f; do printf '%s %s\n' "$f" "$(shasum -a 256 "$f" | cut -c1-64)"; done | shasum -a 256 | cut -c1-12)"
mv "$d" "$P/snippets/$name-$S"
# Only where the script loads it (`from "…"`, `import("…")`): the keys of
# the WebAssembly import object keep the old path, which the .wasm names.
sed_i -e "s#\(from *[\"']\)\./snippets/$name/#\1./snippets/$name-$S/#g" \
-e "s#\(import( *[\"']\)\./snippets/$name/#\1./snippets/$name-$S/#g" "$P/aac-web.js"
grep -q "snippets/$name-$S/" "$P/aac-web.js" || { echo "✗ aac-web.js does not import snippets/$name-$S" >&2; exit 1; }
if grep -qE "(from *[\"']|import\( *[\"'])\./snippets/$name/" "$P/aac-web.js"; then
echo "✗ aac-web.js still loads snippets/$name by its old name" >&2; exit 1
fi
done
J="$(hash12 "$P/aac-web.js")"; mv "$P/aac-web.js" "$P/aac-web.$J.js"
sed_i \
-e "s#/pkg/aac-web_bg\\.wasm#/pkg/aac-web.$W.wasm#g" \
-e "s#/pkg/aac-web\\.wasm#/pkg/aac-web.$W.wasm#g" \
-e "s#/pkg/aac-web\\.js#/pkg/aac-web.$J.js#g" \
-e "s#/pkg/aac-web\\.css#/pkg/aac-web.$C.css#g" \
"$SITE/index.html"
# Leptos's hydration loader builds its URLs from arguments:
# (root, pkg_path, output_name, wasm_output_name)
LOADER='("", "pkg", "aac-web", "aac-web")'
grep -qF "$LOADER" "$SITE/index.html" || { echo "✗ hydration loader call not found — has Leptos changed its loader?" >&2; exit 1; }
sed_i "s#(\"\", \"pkg\", \"aac-web\", \"aac-web\")#(\"\", \"pkg\", \"aac-web.$J\", \"aac-web.$W\")#" "$SITE/index.html"
for f in "aac-web.$W.wasm" "aac-web.$J.js" "aac-web.$C.css" "\"aac-web.$J\", \"aac-web.$W\""; do
grep -q "$f" "$SITE/index.html" || { echo "✗ index.html does not reference $f" >&2; exit 1; }
done
# Eye gaze loads MediaPipe from vendor/mediapipe-<version>-<revision>/ on this
# origin (crates/aac-web/src/bindings/mediapipe_init.js), so turning on the
# camera makes no third-party request and the app stays cross-origin isolated.
rm -rf "$SITE/vendor"
bash third_party/mediapipe/vendor.sh "$SITE" "$CACHE"
MP_DIR="$(cd "$SITE/vendor" && echo mediapipe-*)"
[ -d "$SITE/vendor/$MP_DIR" ] || { echo "✗ vendor.sh left no vendor/mediapipe-* directory" >&2; exit 1; }
# On-device voices, downloaded only when the person asks (neural_voice.rs):
# the Moonshine WASM runtime, vendored and pinned like MediaPipe, and the
# voice packs, built outside git by tools/voices/build_voices.py, under
# voices/ (cloned, not copied, on APFS), once tools/voices/check_packs.py
# has matched them to the committed catalogue. A build without them must say
# so (AAC_VOICES=none): a deploy would otherwise delete the voices it serves.
rm -rf "$SITE/voices"
if [ "${AAC_VOICES:-}" = "none" ]; then
echo "▸ AAC_VOICES=none: no better voices in this build; the app offers the device's voices only"
else
bash third_party/moonshine-wasm/vendor.sh "$SITE" "$CACHE"
VOICES="${AAC_VOICES_DIR:-$(cd ../../.. && pwd)/aac-voice-packs}"
python3 tools/voices/check_packs.py "$VOICES" \
|| { echo "✗ build the packs (tools/voices/build_voices.py), or set AAC_VOICES=none to build without better voices" >&2; exit 1; }
cp -cR "$VOICES" "$SITE/voices" 2>/dev/null || cp -R "$VOICES" "$SITE/voices"
# index.json and LICENSES.md keep their names from one catalogue to the next,
# and the builder stamps every file with one fixed time: give these two this
# build's own, or rsync's size-and-time check would keep prod's old pointer,
# to a catalogue its --delete has just removed.
touch "$SITE/voices/index.json" "$SITE/voices/LICENSES.md"
fi
# The sentence helper, downloaded only when the person asks (ai_helper.rs):
# Google's LiteRT-LM web runtime, vendored and pinned like Moonshine, and the
# model's parts, built outside git by tools/ai/build_ai.py, under ai/ (cloned,
# not copied, on APFS), once tools/ai/check_packs.py has matched them to the
# committed catalogue. A build without them must say so (AAC_AI=none).
rm -rf "$SITE/ai"
if [ "${AAC_AI:-}" = "none" ]; then
echo "▸ AAC_AI=none: no sentence helper in this build"
else
bash third_party/litert-lm/vendor.sh "$SITE" "$CACHE"
AI="${AAC_AI_DIR:-$(cd ../../.. && pwd)/aac-ai-packs}"
python3 tools/ai/check_packs.py "$AI" \
|| { echo "✗ build the model parts (tools/ai/build_ai.py), or set AAC_AI=none to build without the sentence helper" >&2; exit 1; }
cp -cR "$AI" "$SITE/ai" 2>/dev/null || cp -R "$AI" "$SITE/ai"
touch "$SITE/ai/index.json" "$SITE/ai/LICENSES.md"
fi
# The listener, downloaded only when the person asks (listen.rs):
# onnxruntime-web, vendored and pinned like LiteRT-LM, and the model's files,
# built outside git by tools/asr/build_asr.py, under asr/ (cloned, not
# copied, on APFS), once tools/asr/check_packs.py has matched them to the
# committed catalogue. A build without them must say so (AAC_ASR=none).
rm -rf "$SITE/asr"
if [ "${AAC_ASR:-}" = "none" ]; then
echo "▸ AAC_ASR=none: no listener in this build"
else
bash third_party/onnxruntime-web/vendor.sh "$SITE" "$CACHE"
ASR="${AAC_ASR_DIR:-$(cd ../../.. && pwd)/aac-asr-packs}"
python3 tools/asr/check_packs.py "$ASR" \
|| { echo "✗ build the listener's files (tools/asr/build_asr.py), or set AAC_ASR=none to build without the listener" >&2; exit 1; }
cp -cR "$ASR" "$SITE/asr" 2>/dev/null || cp -R "$ASR" "$SITE/asr"
touch "$SITE/asr/index.json" "$SITE/asr/LICENSES.md"
fi
# First letters, downloaded only when the person asks (abbrev.rs): it runs
# on the listener's onnxruntime-web (vendored above, or here when the
# listener is not in this build), with its own worker and Hugging Face's
# tokenizer library, vendored and pinned like it, and the model's files,
# built outside git by tools/ae/build_ae.py, under ae/ (cloned, not copied,
# on APFS), once tools/ae/check_packs.py has matched them to the committed
# catalogue. A build without them must say so (AAC_AE=none).
rm -rf "$SITE/ae"
if [ "${AAC_AE:-}" = "none" ]; then
echo "▸ AAC_AE=none: no first letters in this build"
else
[ "${AAC_ASR:-}" = "none" ] && bash third_party/onnxruntime-web/vendor.sh "$SITE" "$CACHE"
bash third_party/hf-tokenizers/vendor.sh "$SITE" "$CACHE"
AE="${AAC_AE_DIR:-$(cd ../../.. && pwd)/aac-ae-packs}"
python3 tools/ae/check_packs.py "$AE" \
|| { echo "✗ build first letters' files (tools/ae/build_ae.py), or set AAC_AE=none to build without first letters" >&2; exit 1; }
cp -cR "$AE" "$SITE/ae" 2>/dev/null || cp -R "$AE" "$SITE/ae"
touch "$SITE/ae/index.json" "$SITE/ae/LICENSES.md"
fi
# The loaders of the sentence helper, the listener and first letters must
# never reach another site.
AI_SNIP="$(grep -rl 'ensure_engine' "$SITE/pkg/snippets" || true)"
[ -n "$AI_SNIP" ] || { echo "✗ the sentence helper's snippet is not in pkg/snippets" >&2; exit 1; }
if grep -nE 'https?://' $AI_SNIP; then echo "✗ the sentence helper's snippet names another site" >&2; exit 1; fi
LISTEN_SNIP="$(grep -rl 'listen_once' "$SITE/pkg/snippets" || true)"
[ -n "$LISTEN_SNIP" ] || { echo "✗ the listener's snippet is not in pkg/snippets" >&2; exit 1; }
if grep -nE 'https?://' $LISTEN_SNIP; then echo "✗ the listener's snippet names another site" >&2; exit 1; fi
AE_SNIP="$(grep -rl '__aacLettersEngine' "$SITE/pkg/snippets" || true)"
[ -n "$AE_SNIP" ] || { echo "✗ first letters' snippet is not in pkg/snippets" >&2; exit 1; }
if grep -nE 'https?://' $AE_SNIP; then echo "✗ first letters' snippet names another site" >&2; exit 1; fi
SNIP="$(grep -rl 'initialize_face_landmarker' "$SITE/pkg/snippets" || true)"
[ -n "$SNIP" ] || { echo "✗ eye-tracking snippet not found in pkg/snippets" >&2; exit 1; }
if grep -nE 'https?://' $SNIP; then echo "✗ eye-tracking snippet loads from a third party" >&2; exit 1; fi
echo "▸ bundling PWA assets and the service worker"
cp crates/aac-web/static/manifest.json "$SITE/manifest.json"
# Precache: the shell, the manifest, every file in pkg/ (including wasm-bindgen
# snippets such as the eye-tracking loader), the icons, and every symbol sprite
# the prerendered board draws (the default set and the sets it falls back to),
# so symbols show offline from the first visit, and their LICENSES.md. Other sprites are cached by the
# worker the first time they are drawn.
PRECACHE=("/" "/manifest.json")
while IFS= read -r f; do PRECACHE+=("/${f#"$SITE"/}"); done < <(find "$SITE/pkg" "$SITE/icons" -type f | LC_ALL=C sort)
DRAWN="$(grep -oE '/symbols/[a-z]+\.[0-9a-f]{12}\.svg' "$SITE/index.html" | LC_ALL=C sort -u || true)"
[ -n "$DRAWN" ] || { echo "✗ the prerendered board draws no symbols" >&2; exit 1; }
for u in $DRAWN; do
[ -f "$SITE$u" ] || { echo "✗ index.html draws $u, which the build does not ship" >&2; exit 1; }
PRECACHE+=("$u")
done
# Settings links the symbols' licences; the worker serves it offline too.
[ -f "$SITE/symbols/LICENSES.md" ] || { echo "✗ the build does not ship symbols/LICENSES.md" >&2; exit 1; }
PRECACHE+=("/symbols/LICENSES.md")
SYMBOL_FILES="[$(cd "$SITE/symbols" && printf '"%s",' *.svg | sed 's/,$//')]"
VERSION="$(for u in "${PRECACHE[@]}"; do
if [ "$u" = "/" ]; then f="$SITE/index.html"; else f="$SITE$u"; fi
printf '%s %s\n' "$u" "$(shasum -a 256 "$f" | cut -c1-64)"
done | shasum -a 256 | cut -c1-12)"
LIST="[$(printf '"%s",' "${PRECACHE[@]}" | sed 's/,$//')]"
# The worker keeps only the current vendor directory in its vendor cache.
sed -e "s#__VERSION__#$VERSION#" -e "s#__PRECACHE_URLS__#$LIST#" -e "s#__VENDOR_DIRS__#[\"$MP_DIR/\"]#" \
-e "s#__SYMBOL_FILES__#$SYMBOL_FILES#" \
crates/aac-web/static/sw.js > "$SITE/sw.js"
grep -q "__" "$SITE/sw.js" && grep -nE "__[A-Z_]+__" "$SITE/sw.js" && { echo "✗ sw.js has an unfilled placeholder" >&2; exit 1; }
echo "✓ self-contained dist ready at $SITE/ (service worker $VERSION, ${#PRECACHE[@]} files precached)"
ls -1 "$SITE" "$SITE/pkg"
Reference in a new issue View git blame Copy permalink