c8db7f4efa
Drives all four seams over a real WebRTC room against a live livekit-server: join under a minted token, gate each turn OUTSIDE the model (served vs refused for unheld-capability / over-ceiling), meter the model calls, seal one efficiency/1 that verifies under the steward key. examples/live_room.rs is the runnable proof (data-plane turns → gate/meter/seal, receipt verified). Clears two upstream blockers so `--features livekit` builds + runs on macOS: - vendor/libwebrtc: libwebrtc 0.3.38 omits user_data that webrtc-sys 0.3.36 requires (broken upstream release pair); patched additively + [patch.crates-io]. - build.rs: emit -ObjC so libwebrtc's ObjC categories force-load, else PeerConnectionFactory init crashes on an unrecognized selector. Default build + tests are unaffected (the feature stays off by default). Co-Authored-By: Claude Opus 4.8 <[email protected]>
82 lines
4.8 KiB
Markdown
82 lines
4.8 KiB
Markdown
# jouledesk-livekit — the LiveKit `room` binding
|
|
|
|
The reference [`room`](../../SPEC.md) binding class of Joule Desk: `@joule` joins a **live
|
|
voice/video room** and drives the four real-time seams over the call's lifecycle.
|
|
|
|
A live room is a *session*, not a mention. So the loop is not task-per-message — it is:
|
|
|
|
1. **join** — the room access token *is* the capability grant (LiveKit `RoomAgentDispatch`);
|
|
2. **turn** — each user turn is gated **outside the model** (LiveKit `on_user_turn_completed`
|
|
→ `StopResponse`) — an unheld capability or an over-ceiling turn never reaches generation;
|
|
3. **meter** — an allowed turn's model call is metered at the model-call boundary (LiveKit
|
|
`llm_node` / OTel `gen_ai.usage.*`);
|
|
4. **receipt** — one `efficiency/1` seals the whole call at shutdown (LiveKit `session.usage`),
|
|
anchored to the room id — a verifiable record of what `@joule` did, under which capability,
|
|
at what metered energy.
|
|
|
|
The **joule ceiling is shared across the session**, so it holds whether `@joule` is typing or
|
|
talking: the same meter that bounds a text session bounds the room.
|
|
|
|
## Two layers, on purpose
|
|
|
|
| Layer | What it is | Build |
|
|
|---|---|---|
|
|
| **`RoomAgent`** (crate root) | The four seams as a **transport-agnostic** driver over `RealtimeSession`. Gate-before-generate, meter, seal — no SDK. | Default. **Fully unit-tested** (`cargo test`). |
|
|
| **`adapter`** (`src/adapter.rs`) | Wires `RoomAgent` onto the **real LiveKit Rust SDK**: `Room::connect` with the token, the room event loop → `on_turn` / `shutdown`. | Behind `--features livekit`. Needs libwebrtc + a live server. |
|
|
|
|
The driver is where the standard lives, and it is verifiable without a network, a GPU, or a
|
|
running media server. The adapter is thin glue on top.
|
|
|
|
```rust
|
|
let mut agent = RoomAgent::join("standup", &["answer", "summarize"], 5_000_000);
|
|
|
|
// A turn: gated outside the model; run_model runs ONLY if allowed, then is metered.
|
|
agent.on_turn(&turn, |t| my_stt_llm_tts_pipeline(t)); // → Served
|
|
agent.on_turn(&delete_turn, |t| unreachable(t)); // → Refused (unheld cap): model never runs
|
|
|
|
let receipt = agent.shutdown(); // one efficiency/1, verifiable by the whole room
|
|
```
|
|
|
|
See [`../jouledesk/examples/room.rs`](../jouledesk/examples/room.rs) for a full runnable call.
|
|
|
|
## The `livekit` feature (real SDK) — runs live
|
|
|
|
`src/adapter.rs` is reference wiring against the [`livekit`](https://crates.io/crates/livekit)
|
|
Rust SDK — the lower-level client (Room, tracks, data). The high-level `AgentSession` /
|
|
`llm_node` framework is Python/Node, so this crate owns the turn/gate/meter/seal loop itself
|
|
and hands the transport the four call-points; `adapter::run` maps the room event loop onto them.
|
|
|
|
The feature is **off by default** (the SDK is heavy — it pulls libwebrtc), but it **builds and
|
|
runs end-to-end**. Two upstream blockers are worked around in-tree:
|
|
|
|
1. **Native-dep skew.** `livekit 0.7.49` requires `libwebrtc ^0.3.38`, which pulls the newest
|
|
`webrtc-sys` (0.3.36); but `libwebrtc 0.3.38`'s code omits the `user_data` field / a 4th arg
|
|
that `webrtc-sys 0.3.36` now requires. [`vendor/libwebrtc`](../../vendor/libwebrtc) is that
|
|
crate with the three additive lines restored, wired via `[patch.crates-io]` in the workspace
|
|
`Cargo.toml`. (A workaround for a broken upstream release — drop it once upstream ships a
|
|
consistent pair.)
|
|
2. **macOS ObjC categories.** [`build.rs`](build.rs) emits `-ObjC` at link so libwebrtc's
|
|
Objective-C categories force-load — without it, `PeerConnectionFactory` init crashes with
|
|
`unrecognized selector +[NSString stringForAbslStringView:]`.
|
|
|
|
```bash
|
|
cargo test # the tested core (default) — no SDK, no network
|
|
cargo build -p jouledesk-livekit --features livekit # builds the real adapter (+ vendored patch)
|
|
|
|
# a real end-to-end run against a live server:
|
|
brew install livekit && livekit-server --dev & # devkey/secret on ws://127.0.0.1:7880
|
|
cargo run -p jouledesk-livekit --features livekit --example live_room
|
|
```
|
|
|
|
[`examples/live_room.rs`](examples/live_room.rs) mints a real access token, JOINs a real WebRTC
|
|
room as `@joule`, has a second participant publish turns on the room's data plane, gates each one
|
|
**outside the model** (served vs refused for unheld-capability / over-ceiling), meters the model
|
|
calls, and seals one `efficiency/1` that verifies under the steward key — the four seams over real
|
|
transport (ICE + DTLS + SCTP). Bring your own `parse_turn` and `run_model` (STT → LLM → reply →
|
|
`Usage`) closures; the seam logic is identical to the fully-tested core.
|
|
|
|
## Licence
|
|
|
|
Apache-2.0 for reference code; CC-BY-4.0 for the specification text. Part of the
|
|
[Transaction Science open standards](../../../README.md); Joule Desk is the collaboration
|
|
binding of [Joule Table](../../../jouletable).
|