alp@vilnius: ~ up since 2019 · Vilnius Download cv.pdf

← ALP-OS

alp ~ $ work tls-buffers

TLS ate the RTOS buffer pool

A tracker rebooted every 20 s during the TLS handshake, but not on every boot. Every boot that crashed was running a vehicle-bus scan. TLS and the scan were drawing from the same RTOS buffer pool, so I gave TLS its own.

42 blockstaken by one TLS handshake

A tracker connecting to a cloud MQTT broker kept rebooting every 20 s, in the middle of the TLS handshake. But not every boot crashed.

That was the useful part. If some boots survive, the difference between them is the clue.

What the crashing boots had in common

I lined up each crash against everything else the device was doing. Boots 1 to 7 all crashed, and all of them were running a vehicle-bus scan. The one that survived had the ignition off, so no scan ran. And at every crash, all 97 of the larger buffers in the RTOS message-buffer pool were in use.

One pool, two users

The TLS library took its small memory blocks from the same RTOS pool that tasks use to message each other. One handshake took 42 of them. The vehicle-bus scan was using the pool too. Together they emptied it, and the RTOS treats that as fatal.

TLS and the vehicle-bus scan drew from the same RTOS message-buffer pool Evidence: boots 1 to 7 were running a vehicle-bus scan and each one rebooted during the TLS handshake, about 20 seconds apart. The one boot that survived had the ignition off, so no scan ran. Before: task messages, the vehicle-bus scan and the TLS handshake all took blocks from the RTOS message-buffer pool, and one handshake alone took 42 of its small blocks. At each crash, all 97 of the pool’s larger buffers were in use. With no free buffer the RTOS treats the request as fatal and the device reboots. After: TLS takes its blocks from its own private fixed-block area, so the RTOS message-buffer pool only serves task messages and the scan. Before connecting, the device counts the free buffers in the pool. If there are enough it runs the handshake. If there are too few it reports a resource error and retries later instead of crashing. EVIDENCE · each boot against what the device was doing boot 1 scan on reboot boot 2 scan on reboot boot 3 scan on reboot boot 4 scan on reboot boot 5 scan on reboot boot 6 scan on reboot boot 7 scan on reboot one boot ignition off survived Each crash came in the TLS handshake, about 20 s after the one before. BEFORE · messages, scan and TLS share the RTOS message-buffer pool task messages vehicle-bus scan TLS handshake larger buffers at each crash: 97 of 97 no free buffer → RTOS fatal → reboot One TLS handshake took 42 of the pool’s small blocks. AFTER · TLS has its own memory, and a check before connecting task messages vehicle-bus scan message-buffer pool no TLS blocks free buffers ok? yes TLS handshake private TLS block area no resource error, retry later The scan and the handshake no longer draw from the same buffers. The reboot loop stopped.
TLS and the vehicle-bus scan drew from the same RTOS message-buffer pool

The fix

TLS now has its own private fixed-block memory, so it can’t drain the RTOS pool. Before connecting, the device also counts the free buffers. If there aren’t enough, it reports a resource error and tries again later instead of crashing. QA also got a way to create low-memory conditions on the bench.

Conclusion

The reboot loop stopped, and the customer confirmed that MQTT and vehicle-bus scanning now work together.

The crash happened in TLS, but TLS wasn’t really the problem. Sharing the pool was. When two things share a pool, the one that asks last is the one that crashes.

Tags tls rtos memory mqtt