alp@vilnius: ~ up since 2019 · Vilnius Download cv.pdf

← ALP-OS

alp ~ $ work watch-harness

71 tests that run on a real watch

71 tests run on a real smartwatch over BLE, and any new crash fails the test, even if the test’s own checks pass. Together with a memory gate in CI, the setup has found 16 bugs.

16 bugs · 71 testsbugs found, and device tests that run on the watch

A test on a real device can pass while the device crashes in the background. On a watch with little RAM, a change can also pass every test and still eat memory that a later feature needs.

So I built the harness to judge the whole watch, not just the feature under test, and to put a memory cost on every change.

The setup

A pytest harness drives a production smartwatch over BLE and a debug probe. It’s 75 files and 8.1k lines.

A crash guard around every test

Before each test, a guard reads the coredumps, the uptime and the BLE link. After the test, it checks for a new coredump, an unexpected disconnect or an uptime reset. If it finds one, the test fails, even if all its own checks passed.

The watch harness: every test wrapped by a crash guard, every change through a memory gate The rig: a pytest harness of 75 files and 8.1k lines, with 71 device tests, drives a production smartwatch, a dual-core BLE MCU running a watch RTOS, over BLE (SMP) and over a debug probe. Every test is wrapped by the crash guard. Before the test, the guard reads the coredumps, the uptime and the BLE link. Then the test body drives the watch and runs its own asserts. After the test, the guard asks three questions: is there a new coredump, was there an unexpected disconnect, did the uptime reset. Outcome table: own asserts pass and guard clean gives pass. Own asserts pass and guard trips gives fail, a case only the guard catches. Own asserts fail gives fail. Every change also goes through a memory gate in CI with three checks: a RAM gate with a byte budget, a static stack bound computed from the call graph, and a stack and heap peak recorder that writes to flash. The RAM gate caught a stack-size reduction that crashed over-the-air updates. Result: 71 device tests pass with 0 new crashes on the current build, and the harness found 16 bugs, each fixed in its own commit. THE RIG · bench and CI pytest harness 71 device tests 75 files · 8.1k lines BLE · SMP debug probe production smartwatch dual-core BLE MCU watch RTOS EVERY TEST · wrapped by the crash guard guard, before reads coredumps, uptime, BLE link test body drives the watch runs its own asserts guard, after new coredump? unexpected disconnect? uptime reset? own asserts crash guard result pass clean PASS pass trips FAIL only the guard catches this fail either FAIL EVERY CHANGE · memory gate in CI change RAM gate byte budget static stack bound from the call graph peak recorder stack and heap, to flash the RAM gate caught a stack-size reduction that crashed over-the-air updates RESULT 71 device tests pass · 0 new crashes · 16 bugs found
The watch harness: every test wrapped by a crash guard, every change through a memory gate

A memory gate in CI

Every change goes through a RAM budget in bytes, a stack limit worked out from the call graph, and a stack and heap peak recorder that writes to flash. The RAM gate caught a stack-size cut that crashed over-the-air updates.

What it found

The harness turned up 16 bugs, each fixed in its own commit. As a side effect, font loading went from 180 ms to 29 ms.

Font load time, a side result

I also built a binary event journal for the watch. It showed that every ECG abort had been reported with the wrong reason, because of an enum cast. I fixed it with an explicit mapping and a static_assert.

Conclusion

All 71 device tests pass with no new crashes on the current build, and the harness has found 16 bugs.

The crash guard is the part that pays off most. It turns every test into a crash test too.

Tags testing ble wearable tooling