121 lines
11 KiB
Markdown
121 lines
11 KiB
Markdown
# Provenance of flowrs-transport
|
||
|
||
Everything in this crate is written for flowRS and licensed MIT OR Apache-2.0. Per-file origins
|
||
are listed in `tools/runtime-licence.tsv`, which `tools/runtime-licence-audit.sh` checks. This
|
||
file records the clean-room rewrite below in more detail than one register row can.
|
||
|
||
## The ESP8266 ROM erase size: `src/esptool/esp8266_rom.rs` (clean room, 2026-10-27)
|
||
|
||
### Why it was rewritten
|
||
|
||
The ESP8266 mask-ROM loader erases more flash than FLASH_BEGIN asks for, so a host must hand it
|
||
a different number from the image size. flowRS used to carry that rule as
|
||
`crates/flowrs-boards/src/esp8266_rom.rs`. That file was a transcription of
|
||
`ESP8266ROM.get_erase_size` in esptool.py, which is GPL-2.0-or-later. flowRS's porting policy
|
||
lets a GPL tool be used as an oracle, never as a source. Commit e4a336ba removed the file. The
|
||
module here replaces it. Nobody who had read the GPL code or its transcription wrote it.
|
||
|
||
### How it was written
|
||
|
||
The implementer was an AI agent (Claude Opus 5.5) working on branch `work/cleanroom`. It
|
||
started from a specification written by a coordinator who had read the old file. The
|
||
specification held no code: the function's signature and purpose, and the seven assertions of
|
||
the old tests. Under the clean-room rules, the implementer did not open the old file, its git
|
||
history, esptool.py's source, or esptool's documentation. It used only the sources below.
|
||
|
||
| Source | Licence | What was taken |
|
||
|---|---|---|
|
||
| The coordinator's specification | — | Signature `erase_size(offset: u32, size: u32) -> u32`, the purpose, and seven pinned values |
|
||
| ESP8266_RTOS_SDK, commit `858c7c2e`: `components/spi_flash/include/spi_flash.h` | Apache-2.0 | `SPI_FLASH_SEC_SIZE` is 4096 |
|
||
| ESP8266_RTOS_SDK: `components/esp8266/include/esp8266/rom_functions.h`, `components/esp8266/ld/esp8266.rom.ld`, `components/bootloader/subproject/main/esp8266.bootloader.rom.ld` | Apache-2.0 | The ROM's flash descriptor carries `block_size` and `sector_size`; the ROM exports `SPIEraseSector` |
|
||
| ESP8266_RTOS_SDK: `components/bootloader_support/src/bootloader_init.c` | Apache-2.0 | The ROM flash descriptor is configured with 64 KiB blocks and 4 KiB sectors (in the file's ESP32 branch; it confirms the descriptor's convention, not an ESP8266 value) |
|
||
| ESP8266_RTOS_SDK: `components/spi_flash/src/spi_flash.c`, `spi_flash_raw.c`, `include/esp_spi_flash.h`, `components/esp8266/source/chip_boot.c` | Apache-2.0 | Searched for erase and size constants; nothing beyond the above |
|
||
| ESP8266_NONOS_SDK, commit `8e138654`: `ld/eagle.rom.addr.v6.ld`, `include/spi_flash.h` | ESPRESSIF MIT License (MIT terms, limited to use on Espressif's ESP8266) | The ROM has `SPIEraseArea`, `SPIEraseBlock`, `SPIEraseSector` and `SPIEraseChip`; its `SpiFlashChip` carries `block_size` and `sector_size`. Facts only; no code is in flowRS |
|
||
| esptool.py 5.6.0 and 4.8.1 from PyPI, installed in throwaway virtual environments on the build host | GPL-2.0-or-later | Run as a black-box oracle through `tools/esp8266_erase_oracle.py`: inputs and outputs only, with tracebacks off so no source line could print. The method's interface, `(self, offset, size)`, was read with `inspect.signature` |
|
||
| `src/esptool.rs` in this crate (the version after commit b1b3f743) | MIT OR Apache-2.0 | ESP32-family ROMs and the flasher stub take the plain image size; `FLASH_SECTOR_SIZE` |
|
||
|
||
Espressif does not document the miscount. One web search (`ESP8266 ROM SPIEraseArea sector
|
||
block erase espressif`, with docs.espressif.com excluded) returned a summary that paraphrased
|
||
an esptool issue and a Cesanta blog post: an erase that crosses a sixteen-sector block boundary
|
||
erases extra head sectors. The implementer did not open those pages. The model below was fitted
|
||
to the oracle's outputs, not to that summary.
|
||
|
||
### The rule, derived from the oracle
|
||
|
||
The implementer tabulated the oracle's output for every start sector within a block (and a few
|
||
in later blocks) against image sizes of 0 to 40 sectors. It also probed byte-granular sizes and
|
||
unaligned offsets. The table fits one model exactly.
|
||
|
||
Count in 4 KiB sectors. Let `start` be the sector that holds the offset, `head` the sectors from
|
||
`start` to the end of its 64 KiB block (1 to 16), and `handed` the sectors FLASH_BEGIN asks for.
|
||
The ROM then erases `min(2 × handed, handed + head)` sectors. To cover an image of `n` sectors,
|
||
hand it the least `handed` for which that reaches `n`, which is `max(⌈n / 2⌉, n − head)`. A
|
||
partial sector counts as a whole one, and an unaligned offset counts as the start of its sector.
|
||
The outputs repeat every sixteen sectors, which confirms the 64 KiB block.
|
||
|
||
### Verification
|
||
|
||
- **Sweep:** 52,348,517 inputs, 0 disagreements. Each of esptool.py 5.6.0 and 4.8.1 was piped
|
||
into `examples/esp8266_erase_check.rs`, which calls flowRS's rule. The inputs were:
|
||
- every 4 KiB-aligned offset from 0 to 16 MiB (4,097 offsets) against size 0 and every size
|
||
one below, on, and one above each 4 KiB multiple up to 16 MiB (12,289 sizes);
|
||
- 22 × 22 corner values up to 2^32 − 1;
|
||
- one million seeded random inputs anywhere in the 32-bit space;
|
||
- one million seeded random inputs inside 16 MiB.
|
||
- **Pinned vectors:** `tests/data/esp8266_erase_oracle.tsv` holds 396 oracle vectors (18
|
||
offsets by 22 sizes), generated by `tools/esp8266_erase_oracle.py fixture`. The test
|
||
`tests/esp8266_rom_oracle.rs` checks every one.
|
||
- **Unit tests** in the module: the seven specified values. They also prove that the result is
|
||
the least whole-sector count whose modelled erase covers the image, that the ROM over-erases by
|
||
one sector only for an odd image no longer than twice `head`, and that the largest inputs do
|
||
not overflow.
|
||
- **Mutation check:** `tools/mutate-esp8266-rom.sh`.
|
||
|
||
### What the implementer was exposed to, stated plainly
|
||
|
||
- **The old file, through a search mistake.** While looking for the rule's callers, the
|
||
implementer ran a recursive grep and tried to filter the old file out of the results. The
|
||
filter expected `./`-prefixed paths, which this shell's grep does not print, so it failed. As
|
||
a result, 15 matching lines of the old file were shown, as follows:
|
||
- three lines of its provenance header. They said the file was GPL-derived, and that the
|
||
flasher stub's erase size is the plain size (a fact `src/esptool.rs` also states);
|
||
- the first line of the function's doc comment, and its signature (already in the
|
||
specification);
|
||
- the test's name, and a test comment naming a sector-size constant of 0x1000;
|
||
- the seven test assertions (already in the specification);
|
||
- one comment line inside the function body. Cut off mid-sentence, it said the number sixteen
|
||
is a literal in esptool.py's code.
|
||
|
||
No other line of the function body was shown. Every fact in those lines is in the
|
||
specification, in `src/esptool.rs`, or in Espressif's SDKs (4 KiB sectors, 64 KiB blocks, so
|
||
sixteen sectors to a block). The implementer then deleted the file without opening it and
|
||
excluded it from every later search.
|
||
- **Training data.** The implementer is a language model. Its training data may have included
|
||
esptool.py's public source, and it cannot certify otherwise. In this session the rule came from
|
||
the oracle's outputs and is expressed differently: a model of the ROM (`rom_erased_sectors`)
|
||
and the closed-form least cover of that model, `max(⌈n / 2⌉, n − head)`, which the tests check
|
||
against the model. esptool.py was run, not read.
|
||
|
||
## Re-cited to the primary specifications: `src/avr.rs` and `src/dfu.rs` (2026-10-27)
|
||
|
||
The licence audit found that these files cited GPL tools as the place their protocol facts
|
||
were read: avrdude (GPL-2.0-or-later) for the AVR bootloader protocols, and WebDFU (ISC,
|
||
itself ported from dfu-util, GPL-2.0-or-later) for DFU. Facts are not copyrightable, but the
|
||
porting policy wants them cited to the specifications that publish them. Each constant and
|
||
layout was checked against the documents below only, never against the GPL tools. Every value
|
||
matched. Each builder emits one byte sequence per command, in the order the specification's
|
||
command table gives. That table dictates the structure; nothing in it follows a GPL tool's
|
||
code. The comments now cite the specifications.
|
||
|
||
| Source | Licence | Checked in this crate |
|
||
|---|---|---|
|
||
| Atmel AVR061, *STK500 Communication Protocol* (2525B, 04/03), and the `command.h` Atmel published with it (`AVR061.zip`, 2002) | Atmel/Microchip documentation; facts only | `avr::stk500v1`: `Sync_CRC_EOP` 0x20; `Resp_STK_OK` 0x10, `FAILED` 0x11, `INSYNC` 0x14, `NOSYNC` 0x15; commands 0x30, 0x41, 0x50, 0x51, 0x52, 0x55, 0x64, 0x74, 0x75; `Parm_STK_SW_MAJOR` 0x81. Also the address low byte first, the page size high byte first, and the `F` memory type |
|
||
| Atmel AVR068, *STK500 Communication Protocol* (2591C, 06/06), and its `command.h` (`AVR068.zip`, 2005) | Atmel/Microchip documentation; facts only | `avr::stk500v2`. Table 3-1 gives the frame: `MESSAGE_START` 0x1B, sequence, size MSB first, `TOKEN` 0x0E, body, and an XOR checksum over every byte. The commands are 0x01, 0x06, 0x10 to 0x14 and 0x1B; `STATUS_CMD_OK` is 0x00 and `STATUS_CMD_FAILED` 0xC0; the answer body is `[answer id, status, …]` |
|
||
| Atmel AVR109, *Self Programming* (1644G, 06/04), Table 2, and its reference bootloader `main.c` (`AVR109.zip`) | Atmel/Microchip documentation and sample code; facts only | `avr::avr109`: the command characters and the carriage-return acknowledgement. `main.c` shows addresses and block sizes read high byte first, and flash addresses counted in words |
|
||
| USB-IF, *Device Class Specification for Device Firmware Upgrade*, revision 1.1 (2004) | USB-IF specification; facts only | `dfu`: `bmRequestType` 0x21/0xA1 (Table 3.1) and request codes 0 to 6 (Table 3.2). The block number goes in `wValue` for DNLOAD (section 6.1.1) and UPLOAD (section 6.2.1). Section 6.1.2 gives the 6-byte GETSTATUS reply with a 3-byte `bwPollTimeout`, the status codes 0x00 to 0x0F and the states 0 to 10 |
|
||
| ST AN3156, *USB DFU protocol used in the STM32 bootloader* | ST documentation | Cited as DfuSe's specification. **Not read in this session:** st.com did not answer requests from the build network, and no mirror was reachable. Someone should confirm the DfuSe values against AN3156 itself |
|
||
| ST's STM32 USB device library, DFU class (`STMicroelectronics/stm32-mw-usb-device`, `Class/DFU/Inc/usbd_dfu.h` and `Class/DFU/Src/usbd_dfu.c`) | SLA0044; facts only, no code in flowRS | `dfu` DfuSe: the commands ride on DNLOAD with `wBlockNum` 0 (GET_COMMANDS 0x00, SET_ADDRESS_POINTER 0x21, ERASE 0x41) and carry a 32-bit little-endian address. Data blocks start at 2, at `(wBlockNum − 2) × transfer size` past the address pointer. The DFU status and state values agree with DFU 1.1 |
|
||
|
||
The programmer ids in `avr.rs`'s table (`arduino`, `urclock`, `wiring`, `avr109`, …) are the
|
||
names boards use in Arduino's `upload.protocol`, which are avrdude's programmer ids. They are
|
||
named for interoperability; nothing else from avrdude is used.
|