alp@vilnius: ~ up since 2019 · Vilnius Download cv.pdf

← ALP-OS

alp ~ $ work not-the-device

Lost records, lost commands, no device bug

Records missing from a download, and commands that stopped reaching trackers. Both looked like firmware bugs. One was an old reader tool, the other a carrier NAT that forgot devices after about 15 seconds of quiet.

201 of 201records found by my parser; the old tool read 101

When data goes missing between a tracker and a server, everyone looks at the device first. It’s the part we build.

These two tickets were both about lost data, and in both the device turned out to be fine. The work was proving where the data really stopped.

Missing records

The first ticket estimated that the “Read records” download in the configuration tool was missing about 30% of the records on the device. If true, that’s a firmware data-loss bug.

The desktop tool for opening the downloaded .bin file had been discontinued. So I wrote my own parser in Python. It decodes the device’s record protocol straight from the dump and counts every record.

On the file I checked, the old tool showed 101 of 201 records. My parser found all 201. The old tool just didn’t read them.

Records decoded from the same .bin download

I published the parser in our internal tools docs, so anyone can check a download themselves. QA retested the download and it passed.

Missing commands

In the second ticket, trackers sent data to the server over UDP, and the commands the server sent back stopped arriving.

I reproduced it first. The modem was the obvious suspect, so I opened a case with the modem vendor. Then I suspected the forwarding on our office network. Neither explained it. The ticket stayed open from 2025 until I found the cause in February 2026.

A NAT with a short memory

On this operator’s APN, the device had no public address. It sat behind the operator’s carrier-grade NAT, which keeps a port mapping open only while traffic flows both ways. On this network, the mapping closed after about 15 seconds of quiet. Any command after that went to a port the NAT had forgotten, and was dropped before it reached the modem.

The fix was a setting: the existing network ping now runs every 10 s, so the NAT never sees 15 seconds of quiet. I also proposed a default ping interval for UDP links in the firmware, and updated the test automation script.

The lost commands: the carrier NAT forgets the device after about 15 s of quiet; a ping every 10 s keeps it open Path: the tracker has no public address and sits behind the operator's carrier-grade NAT, which maps a public port to the device. Data goes up through the NAT to the server, and server commands come back down through the same mapping. The mapping lives only while traffic flows both ways. Timeline drawn to scale on an axis from 0 to 30 seconds. Before: after the last data at 0 s the link goes quiet, the NAT mapping stays open until about 15 s, then closes, and a server command sent after that is dropped at the NAT. After: with the network ping timeout set to 10 s, the device sends the last data at 0 s and then a ping after every 10 s of quiet, at 10, 20 and 30 s. Each send opens a new window of about 15 s, the next ping arrives inside it with about 5 s to spare, the mapping stays open the whole time, and the server command is delivered. PATH · one UDP mapping carries both directions tracker no public address operator carrier-grade NAT public port → device server data, ping command The mapping lives only while traffic flows both ways. Here it closed after ~15 s of quiet. BEFORE · data, then quiet last data command dropped NAT mapping open closed, port forgotten ~15 s without two-way traffic AFTER · network ping timeout set to 10 s last data ping after 10 s quiet command delivered NAT mapping open the whole time each send: ~15 s window the next ping lands ~5 s before it closes 0 10 15 20 30 s Every ping lands inside the window the last one opened, so the NAT never forgets the port.
The lost commands: the carrier NAT forgets the device after about 15 s of quiet; a ping every 10 s keeps it open

Conclusion

Records: all 201 were in a file the old tool read as 101. Commands: with a ping every 10 s, they reach the trackers again. The 201 is from the one file I checked, and 15 s is this operator’s timeout, not a universal number.

Neither fix touched firmware code. The NAT case took long because the first two suspects were reasonable ones.

Tags records network python field-bug