alp@vilnius: ~ up since 2019 · Vilnius Download cv.pdf

← ALP-OS

alp ~ $ work one-dump

A weekly crash and one memory dump

A crash showed up about once a week on our stability stand, and never on a desk. One memory dump showed two tasks freeing the same MQTT buffers. I made the race happen on demand, then fixed it.

about 1/week  to 0abnormal restarts on the stability stand, one-week run

In 2026, devices on our stability stand crashed about once a week. They were sending records over MQTT with TLS to a cloud broker. The RTOS reported a failed buffer release from the record-send task and reset the device.

Once a week is awkward. You can’t reproduce it on a desk, and you can’t prove a fix by waiting a week and hoping.

Working from one dump

With no way to reproduce it, I had one memory dump to work from. I traced the failed release back to link teardown. Two tasks, the MQTT close path and the record-send close path, had both torn down the same link at the same moment.

The double free

Both paths released the link’s connection buffers. The first release returned them to the pool. The second asked the RTOS to free buffers that were already free. The RTOS treats that as fatal and resets. The two closes rarely line up, so it happened about once a week.

Making it happen on demand

I added a test command that forces both closes to run at the same time. Now I could trigger the race whenever I wanted.

Then the fix: link teardown is now guarded, so the buffers are released exactly once, whichever task gets there first. The second one sees they’re gone and skips.

Two tasks closed the same link and freed the same buffers twice Before: the MQTT close task and the record-send task both tear down the same link at the same moment. The MQTT close task releases the link’s connection buffers, and they go back to the pool. Then the record-send task releases the same buffers again. The record-send task asks the RTOS to free a buffer that is already free; the RTOS treats that as fatal and resets the device. The two closes rarely overlap, so this happened about once a week. After: link teardown is guarded. Whichever task closes first releases the link’s connection buffers once. The second task checks, sees they are already released, and skips, so the buffers go back to the pool and the device keeps running. A test command forces both closes to run together, so the race can be triggered on demand. Proof on the stability stand with MQTT over TLS: before, about one abnormal restart a week. With the fixed firmware, seven days in a row with zero abnormal restarts. BEFORE · two tasks close the same link at the same moment MQTT close link buffers record-send close link release buffers held back in pool, free close link release again device reset The second release asks the RTOS to free a buffer that is already free. That is fatal. AFTER · guarded teardown, buffers released once MQTT close link buffers record-send close link release once held back in pool, device runs on close link already freed Whichever task closes first frees the buffers. The other one skips. A test command forces both closes together, so the race runs on demand. PROOF · stability stand, MQTT over TLS, one week before about 1 abnormal restart a week fixed day 1 day 2 day 3 day 4 day 5 day 6 day 7 0 abnormal restarts
Two tasks closed the same link and freed the same buffers twice

Conclusion

The fixed firmware ran a full week on the stand with no abnormal restart. To be fair, one clean week is weak proof for a bug that showed up once a week.

What made this solvable was the test command. A rare race doesn’t need a longer soak. It needs a way to force it.

Tags mqtt tls rtos memory field-bug