alp@vilnius: ~ up since 2019 · Vilnius Download cv.pdf
alp ~ $ whoami

Alp Sezer Orak

Senior Embedded Software Developer, Vilnius

Firmware for cellular GNSS trackers, BLE sensors and a health smartwatch.

C C++ Python RTOS LTE GNSS BLE MQTT

alp ~ $ cv

Career

2022 – now
Senior Embedded Software Developer · Teltonika Telematics, Vilnius
  • Platform maintainer and release owner for cellular GNSS tracker firmware: about 140 release tickets driven (firmware and app), base updates across about 15 customer branches.
    how: When it was not the firmware

    This is the work around the code at Teltonika since 2022. In that time I’ve made 903 commits, implemented about 567 tickets, investigated about 313 more, and driven about 140 release tickets.

    Tickets by my role, 2022–2026

    On the platform maintainer rotation, I answer the questions that QA and support bring to the firmware team. Plenty of those end with “the firmware is fine”. That answer only counts if there’s data behind it.

    Releases

    I own releases on the tracker platform: five base lines, about 15 customer branches, experimental builds and mobile app builds. Each release comes with notes telling QA what to test. I carry base updates and cherry-picks across the customer branches, and I cancel and redo a release when QA results call for it. One base release carried 53 tickets. I also added a CI check that blocks a customer release if its firmware entry is missing.

    Support

    I’ve answered about 150 support cases with a root cause found in device logs.

    In one case an ACK went missing. I rebuilt the test traffic against my own TCP server. The device sent two messages, and the server received them as one 1,388 B block. The server’s parser was the problem, and I suggested splitting messages by their length field.

    In another, a reported 1.5–2 minute clock error wouldn’t reproduce on two modem types and many SIMs. The evidence pointed to the operator’s network time.

    Networks and vendors

    A carrier NAT that forgets devices after 15 quiet seconds became half of another write-up. One escalation covered thousands of devices on one network, and forcing 4G with manual band selection solved it.

    When a problem might be inside the modem, I build the case. I captured the modem’s internal logs on the bench and sent them to the vendor. One case took months of back and forth, and the fix reached the customer in a base update. I also turn vendor advice into firmware work, like a work item for faster switching between 2G and 4G.

    Estimates

    I’ve written at least 25 research documents with effort estimates, from 30 h to 332 h. The dual APN estimate came to 116 h. It showed that only one TLS session fits in RAM, and the customer accepted a design built around that.

    Conclusion

    About 150 support cases with a root cause, about 140 release tickets, at least 25 estimates, and about 567 tickets implemented.

    “Not our bug” is a finding too, and it needs the same proof as a fix: a rebuilt test, a trace, a modem log.

    full write-up →

    alporak.com/work/the-job
  • Redesigned modem recovery and the RTOS task watchdog as one staged recovery, tested on five modem types.
    how: Hung modems that recover on their own

    Trackers in the field lost data whenever their modem stopped working. Nobody is around to power-cycle a tracker in a vehicle, so the firmware has to notice and fix it by itself.

    Restarting a modem is the easy part. The hard part is knowing when to do it, how hard to push, and making sure there’s no path where the tracker just gives up.

    Escalating step by step

    Recovery is driven by how long the tracker has sent no data. At 10 minutes it restarts the modem radio with CFUN. At 20 it resets the modem through its hardware pin. At 30 it resets the whole device. Cheapest step first. I tested it on five modem types.

    One watchdog instead of two

    I wrote the user story to merge this recovery into the old watchdog and remove the duplicate code: 137 lines added, 183 removed. The watchdog period now grows with the configuration, so a tracker that sends rarely isn’t mistaken for a hung one. Only one recovery can fire per 10 minutes. I documented it with flowcharts and a table for every counter.

    What the merge broke

    The merge led to about 12 follow-up bugs, and I fixed them. The watchdog counted per server, even though all servers share one modem, so I made it one counter. A CFUN restart was clearing the no-GPRS counter. A record period longer than the watchdog period caused pointless modem restarts. And one modem type has no pin reset, so that step only runs where it’s supported.

    Short drops and deep sleep

    A short network drop went unnoticed, so the tracker waited out a long timeout and took 1 min 10 s to react. I added a “suspicious” state: after a short drop, the next communication error starts a full recovery right away.

    For LTE-M I built a step-by-step sequence: check registration with CEREG?, check and reactivate the data context, then restart the radio. I made these work in deep sleep too, where records had been arriving 6–7 minutes late instead of 2.

    A scan that could strand a tracker

    The operator scan ignored errors, so an empty result left the tracker stuck for good. On top of that, a dropped data context banned the operator, started no new search, and reset the recovery counter.

    Now the scan result is checked, and retries happen at 30, 60 and 120 s instead of every 5 minutes. After 3 failures the modem network restarts, a dropped context starts a search right away, and a ban no longer resets the counter.

    First retry after a failed operator search

    Conclusion

    The client confirmed the LTE-M sequence works. QA could no longer reproduce the deep-sleep delay. The operator-search fix passed a test with a roaming SIM that had data suspended.

    Most of the work came after the first version. Recovery code only runs when things go wrong, so its bugs show up late, one field case at a time.

    full write-up →

    alporak.com/work/modem-recovery
  • Cut MQTT record latency p90 from 4.06 s to 0.12 s for a large fleet.
    how: Why did locking a car take 10 seconds?

    A fleet customer locks and unlocks cars over MQTT. The server sends a command, the tracker does it and replies. Over time, the p90 for a lock crept past 10 seconds. That’s a long time to stand next to a car, waiting for the door.

    Measuring first

    I wrote a script that pulls the round-trip time of every publish out of the device logs, and added latency probes at each stage. Then I changed one thing at a time, each behind its own flag, and measured again.

    One task doing everything

    There was no single bug. It was a traffic jam. Commands, record publishing and the protocol engine all ran in one task. That task waited up to 60 s for a socket send, and with no limit at all for an MQTT ACK. One slow write and everything behind it stalled, including a lock command stuck behind a queue of records. On top of that, every record was built twice.

    Splitting it up

    I split MQTT into three tasks: commands, publishing and the protocol engine. Now a task waiting on the network doesn’t hold up the others.

    They still share one TLS session, so writes have to be serialized somewhere. I first tried the shared network layer. It deadlocked. It works in the layer that owns the TLS session.

    Both waits now have a cap. The transport answers ok, busy or fatal, so a busy modem no longer kills the TLS session. The duplicate record build is gone. I also fixed a record-storage mode that an earlier update had broken.

    Conclusion

    Record send p90 went from 4.06 s to 0.12 s, and the median from 1.03 s to 0.05 s. The time one reply held the connection layer went from 495–1,144 ms to 32–240 ms. A calibration command went from 1,664 ms to 4 ms.

    Record send time, p90

    One caveat: lock replies use the same publish path, so record send time is what I measured. I don’t have the end-to-end lock time after the fix. The delay still visible in the customer’s logs came from sockets dropping while roaming.

    Nothing here was slow on its own. The problem was waiting without limits, and it only showed up once each stage was measured.

    full write-up →

    alporak.com/work/mqtt-latency
  • Designed signed device unlock, salted password hashes and downgrade protection. Moved TLS to TLS 1.3, and built the EU CRA SBOM pipeline.
    how: 3,505 files compiled, 597 shipped

    Tracker firmware needed stronger device security, a modern TLS stack and a software bill of materials (SBOM) for the EU Cyber Resilience Act. This page covers the design only. The details stay private.

    Which files actually ship?

    Standard SBOM tools can’t read this kind of RTOS firmware image, so I worked from the linked ELF instead. Most of what the compiler sees never ends up in the image, and listing all of it would make the SBOM longer and less accurate.

    Files in the build

    I compared four ways of counting the same build. The compiler saw 3,505 files. The linker’s “unused” list was misleading. File symbols in the ELF over-count by about 2x. Counting only files that own a sized symbol in a loaded section gives 597, and that’s what the SBOM lists. The CI pipeline handles the scanner’s rate limit and uploads to Dependency-Track with CPE identifiers.

    Unlock and passwords

    Unlocking is challenge-response, and the device verifies a signature. Passwords are stored only as salted PBKDF2 hashes and compared in constant time. Attempt counters survive a power cut. If the lock’s record gets damaged, the lock stays closed.

    Updates and TLS

    Firmware updates got downgrade and tamper protection, designed together with five new unit-test suites. I shipped it as release owner.

    I moved TLS to the library’s current long-term-support line, with TLS 1.3. MD5, SHA-1, DES, renegotiation and DTLS are off. Only AEAD and ECDHE cipher suites are left.

    Conclusion

    Fail-first unit tests, with fakes for crypto and storage, caught 12 of 12 deliberate mutations. I also ran bench loops with reboots and power cuts.

    For security code, a test suite that has never failed doesn’t prove much.

    full write-up →

    alporak.com/work/security
  • Built dual-APN networking and LTE-M recovery sequences together with modem vendors.
    how: Three features against one RAM budget

    Tracker firmware runs on a fixed amount of RAM, and most of it is already taken. Three requests each needed more of it. A client wanted 400 geofences, each a 15-point polygon. Another project needed the driver ID list to hold 10,000 entries. A customer needed two data connections on different APNs, open at the same time.

    I wrote an estimate for each before writing any code: 154 h of firmware work plus 40 h for the configuration tool, 60 h, and 116 h. Each one hit a different wall: free RAM, boot time, and room for only one TLS session.

    Geofences: measure, then cut

    Every polygon point lives in RAM. With 400 zones, I measured 42 kB of free RAM at 10 points per polygon, and 24 kB at 15. That was an early measurement, and it was dangerously low.

    Free RAM with 400 zones, preliminary measurement

    Coordinates come with six decimal places, about 10 cm, and were stored as doubles. I pushed for 32-bit floats instead: half the bytes per coordinate, and a little less precision. I also pushed back on per-zone names as scope creep. It still didn’t fit, so I removed reserved features the build never used, and shortened a rarely used list to free 2.4 kB.

    What the cuts broke

    Trimming had removed some base parameters by mistake, so the immobilizer re-locked after unlock. I restored them. The #ifndef blocks I added to save RAM overlapped and changed an IO default, so an alarm event was always on. I patched that. And the configuration tool still treated coordinates as doubles, so every config showed unsaved changes. I updated it to float.

    10,000 driver IDs: boot was the problem

    A driver touches an ID key to the 1-Wire reader, and the firmware checks it against the list for the immobilizer. With 10,000 entries the feature worked, but boot took about 100 s, against a 30 s limit. I made the list load faster at boot, and cut RAM and BSS use so the list fit.

    Two connections, one TLS session

    The firmware assumed there was only ever one data connection. The estimate is where the real limit showed up: there was RAM for one TLS session, not two. So I reworked the core network API to handle two PDP contexts. Both connections stay up, and secure traffic goes to one server at a time, so one TLS session is enough.

    In testing, the backup connection was torn down and rebuilt on every send. The network rejected that, and the modem got stuck in a restart loop. I fixed the backup handling in several places and removed the settings-refresh loop.

    Conclusion

    Geofences: QA confirmed “400 zones configurable and polygons working”, and the build fits in RAM. Driver IDs: boot went from about 100 s to under 30 s, on two tracker models. Two connections: the customer accepted the design, which touches about 30 files across the network API and two modem drivers.

    On a device this full, every feature costs something else. Here it was precision, unused features and a second TLS session. And cutting RAM breaks things too, so cuts need testing like any new feature.

    full write-up →

    alporak.com/work/ram-budget
  • Traced 700 trackers sending zero coordinates to a fix flag that moved before NMEA was stable.
    how: 700 trackers on Null Island

    In 2024, after a firmware update, 700 trackers in the field started sending records with zero coordinates. The records said they had a GPS fix.

    0, 0 is a spot in the Gulf of Guinea, known as Null Island. A record without a fix is just a gap. A record with a “fix” at 0, 0 is worse, because the server puts it on the map.

    The flag moved too early

    I reproduced it. The GPS fix signal switched to “fix” before NMEA stabilization had finished. Any record created in that window had a fix but no real position.

    So I made the fix signal wait until NMEA is stable. In factory test mode the wait is skipped, so production tests stay fast.

    A flag that never reset

    While wiring that in, I found that the unstable-fix flag was never reset on sleep or on GPS loss, so it carried over into the next fix. I reworked it. The unstable-fix flag now only starts NMEA stabilization, and a separate “NMEA stable” flag feeds the fix filter, the same way the speed and altitude filters do.

    What the wait exposed

    The new delay uncovered an ordering bug. “Update good coordinates” ran before “update GPS fix state”, so coordinates changed while the status still said no fix, and went stale after ultra-deep sleep. I swapped the order, so coordinates only update on a stable fix, and the fix status now resets on sleep.

    I also added GNSS status 4, “filter active”, so testers can see why coordinates are being held.

    Conclusion

    Test drives on 3 device families produced no zero-coordinate records. All three fixes touch the same state, so I put them in one test firmware, and QA verified them together.

    Adding a delay changes the order things happen in, and that brings hidden ordering bugs to the surface. The first fix was small. Most of the work was in what it uncovered.

    full write-up →

    alporak.com/work/zero-coords
  • Found a GNSS race that left the receiver off for 5.5 h. GNSS commands went from 45 s to 3 s.
    how: GNSS off for 5.5 hours, and a cold start that lied

    A tracker drove out of an underground car park and didn’t get a fix for 5.5 hours. The logs showed the GNSS module had never been powered on, even though the firmware said it was.

    Separately, the firmware’s fix-time test was reporting cold starts of 1–3 s. That’s far too good to be true.

    Two different bugs, same problem: the firmware was reporting something about the GNSS that wasn’t real.

    Reproducing the race

    I suspected a race in the restart sequence. I added a test hook that floods the GNSS request path, and that reproduced it on the bench: the firmware says GNSS is on, the module stays off.

    How a request got lost

    A restart sends “off”, then “on” 2 s later. Requests were de-duplicated against the module’s current state. When “on” arrived, “off” hadn’t been applied yet, so the module still looked on, and “on” was dropped as a duplicate. Then “off” completed, and nothing turned it back on.

    The fix

    I added a pending-request counter and a FIFO, so requests can’t be lost or reordered while an earlier one is still being applied. As a backup, a check runs every second and resends the request if the wanted and actual states differ for 30 s.

    Switching GNSS on or off by command went from 45 s to 3 s, with no reboot. In QA, two trackers got their fix back quickly in driving and long-term tests.

    Time to switch GNSS on or off by command

    The test that passed too fast

    With tick logging, the test “passed” after about 215 ticks. The raw NMEA showed why. A valid $GNGGA line from before the reset arrives after the reset command but before the module’s start message, and the firmware took it as a fresh fix.

    I added a 300-tick guard window, about 1.3 s, that ignores GNSS status right after a reset. A fix-holdout feature was also keeping the old status alive, so I cleared it in the two places where the test forces a fix loss. I also tried clearing the assistance data. That made no difference.

    Conclusion

    GNSS now switches in 3 s instead of 45, and requests no longer get dropped. The test reports real numbers: cold start 30–65 s, warm about 3–5 s with assistance data, hot 3–4 s.

    A result that looks too good deserves the same suspicion as one that looks bad. The cold-start number was off by more than ten times, and nothing about it looked broken.

    full write-up →

    alporak.com/work/gnss-honesty
  • Bisected offline record loss on the device to a merge commit, and fixed an older delete edge case. Delivery went from 1 of 11 packets to 11 of 11.
    how: 1 packet out of 11 survived the outage

    In RS232/RS485 buffered mode, the tracker stores data from a serial sensor and sends it when the server is reachable again. But after an outage, only the last packet was left.

    Finding the change

    Buffered mode had worked before, so something had broken it. The record database stores records by sequence number in a nested folder tree. I bisected on the device across releases and merges, and landed on a merge from an unrelated feature: a rewrite of the FIFO record database.

    The flag nobody read

    The multi-save call still took a sequence flag, but the rewritten code never read it. So every packet was written as sequence 0. The database then decided the file was corrupted and erased it, leaving only the newest packet.

    Two more problems on the same path

    Buffered mode always used the TCP path, because only the non-buffered path checked the server type. So a device set up for MQTT never sent its buffered data at all. And in the JSON codec for the cloud, serial data over 18 B was dropped and 0x0A/0x0D bytes were changed. There was also an old edge case in delete-all.

    The fixes: each buffered packet gets its own storage slot, which also fixed delete-all. Buffered mode now uses the same server-type check as the rest, so it works over TCP and MQTT, ASCII and binary. The JSON codec sends the full serial data as a hex string.

    Conclusion

    After an offline period, 11 of 11 packets now arrive, instead of 1 of 11.

    Serial packets delivered after an offline period

    Nothing caught the original break because the record database had no unit tests. I wrote the first ones, with an in-memory fake file system, a 269-line fake record-manager harness and a 5-step test plan.

    The symptom was in one feature and the cause was in another. Bisecting is what connected them.

    full write-up →

    alporak.com/work/offline-loss
  • Fitted 400 polygon geofences and a 10,000-entry driver ID list into trackers short of RAM. With the full list, boot went from about 100 s to under 30 s.
  • Co-developed a React Native BLE sensor app with 14 store releases.
    how: 30,000 records over BLE in 5–6 minutes

    BLE sends data in small packets. Both sides agree on a maximum size, the MTU, and anything bigger doesn’t get through.

    I ran into this limit from two sides: a phone app that downloads logs from a cold-chain temperature recorder, and tracker firmware that iPhones couldn’t talk to.

    The app: every record has to arrive

    I co-developed the React Native app for a family of BLE sensors, on iOS and Android: about 140 commits, about 85 tickets and 14 releases to Google Play and TestFlight.

    For the recorder’s cold-chain certification, every record has to arrive, and you have to be able to show it did. I added a CRC check on every chunk, so each row in the export says whether its chunk arrived intact. I also fixed a page-header shift, where the header sometimes arrived one slot late and the app read a record chunk as the header. Then I raised the MTU to 247 and the chunk size from 15 to 60 records.

    To test full-size downloads, I added a fake-record generator to the app. 30,000 records now download in 5–6 minutes.

    The firmware: iPhones and 512-byte chunks

    An iPhone can only reach the tracker over BLE. The tracker isn’t MFi certified, so iOS can’t use its Bluetooth Classic serial link. And our firmware sent BLE data in fixed 512 B chunks, which iOS can’t take.

    In 2024 I wrote up how MTU negotiation really worked on the platform. The chip vendor’s Bluetooth layer negotiated the MTU, and our code sent fixed 512 B chunks on top of it. I set the chunk payload to 182 B, tested on iOS and Android, and released it as an experimental iOS-compatible firmware. That fixed the iPhone link.

    When the vendor said no

    The vendor SDK had no MTU negotiation in its GATT server layer. In 2025 I opened a support case, and they declined to add it. So I built the MTU exchange in the GATT client layer myself, for links where the tracker is the client. I tested it in the cold-chain firmware, where the tracker reads temperature recorders as a client, and updated the BLE MTU docs.

    Conclusion

    In the app, 30,000 records download in 5–6 minutes, and every chunk is CRC-checked. In the firmware, QA verified 182 B chunks on two tracker models.

    On the firmware side, the write-up mattered more than the code. Once it was clear who negotiated the MTU and what our code did with it, the iPhone fix was small.

    full write-up →

    alporak.com/work/cold-chain
  • Built and own the device inventory app for R&D and QA test hardware, used company-wide since June 2026.
    how: The inventory app that outgrew 50k reads

    R&D and QA have a lot of test hardware, and every device is identified by its IMEI. I built and run the inventory for it: Expo / React Native for Android, iOS and web, an Express API, MongoDB, a C++ scanner service, Docker and GitLab CI. So far that’s at least 33 tickets and about 356 commits.

    It started as a small tool and grew into a company-wide service.

    Outgrowing Firebase

    The first version went live on GitLab Pages the same day it was asked for. Then it moved to Firebase. When it hit Firebase’s 50k read limit, it needed a new home.

    Before rebuilding, I priced the alternative. Moving to an off-the-shelf open-source asset manager came to 64 hours. Moving my app to its own backend came to 40. So I moved it to MongoDB behind my own Express API.

    Scanning etched codes

    The app reads QR and DataMatrix codes with zxing-cpp in three places: a Kotlin module on Android, WASM in the browser, and the scanner service on the server. Laser-etched codes are hard to read, so the decoders try several scales and inverted images.

    I also added a server-side photo decoder (OpenCV, Tesseract OCR, zxing). It reads every code on a PCB from one photo, and does it better than the live scan.

    Everyday use

    Search supports AND/OR, exact (=) and partial (~) matching, suggestions and shareable filter links. Excel import and export puts each location on its own sheet. Device details fill in automatically, duplicate IMEIs are rejected, and devices can be reserved and released. There’s a daily backup, and admin mode has a git-like change history with revert.

    An API for CI

    The firmware CI pipeline needed the device list, but the app only had interactive single sign-on. So I built a separate machine API for CI, with its own credentials. CI firmware builds now use it to fetch their device list.

    Webhooks are signed with HMAC, responses carry ETags for 304s, and a test catches the OpenAPI docs falling behind the routes.

    Conclusion

    Sorting 2,640 devices went from 487 ms to 92 ms. The app has 352 Jest tests and 1,371 screenshot comparisons. In June 2026 it launched company-wide, with company single sign-on.

    List sort, 2,640 devices

    Pricing the off-the-shelf option was worth it, even though I didn’t take it. 40 hours against 64 made it a decision, not a preference.

    full write-up →

    alporak.com/work/inventory-app
  • Smartwatch (C++): motion-tolerant heart rate, an event-log platform and an on-device test harness.
  • Firmware for a heavy-vehicle tracker line, and tracker firmware that reads EN 12830 cold-chain recorders for their certification.
  • About 567 tickets implemented, about 313 investigated and at least 25 research estimates. Vendor cases with modem and chip makers. Regularly stood in as team lead.
2019 – 2022
Electronics Engineer · DvM IoT Solutions, Ankara
  • Ultra-low-power temperature monitor on ESP8266 that runs a year on one battery, using tuned deep sleep and I2C sensors.
  • NB-IoT vibration analysis on LPC1517 with on-device FFT, for predictive maintenance.
  • Modular low-level drivers and sensor integration for LPC1769, and a move of the legacy communication stack to Mbed OS with multithreaded Ethernet and a cloud link.
  • Took part in CE certification, and designed the enclosure and a breakout board for its safety and EMI requirements.
  • Firmware updates over the air and over USB.
    how: Three years of building whole devices

    From 2019 to 2022 I was an electronics engineer at DvM IoT Solutions. We built IoT devices, and I worked on every layer: drivers, firmware, updates and the enclosure.

    When you build the whole device, everything is connected. Battery life depends on what the firmware does between readings. The enclosure decides whether the product passes CE.

    A freezer monitor that lasts a year

    I designed an ultra-low-power temperature monitor for freezers, on a Wi-Fi MCU. It reads its sensors over I2C and spends the rest of the time in deep sleep. I tuned that sleep until one battery lasted a year.

    A vibration monitor that does its own FFT

    For predictive maintenance, I built a vibration and temperature monitor on a Cortex-M MCU with NB-IoT. It runs an FFT on its own vibration data, checks the result against thresholds, and sends an alert over NB-IoT when one is crossed.

    Drivers and an RTOS

    On another Cortex-M MCU, I wrote modular low-level drivers and integrated analog and digital sensors for industrial automation. I moved the old communication stack onto an embedded RTOS, which made the Ethernet communication multithreaded and connected it to the cloud.

    Updates and CE

    The devices can take new firmware two ways: over the air, or over USB with the device in USB client mode.

    I took part in CE certification, and designed the enclosure and a breakout board to meet the safety and EMI requirements.

    Conclusion

    A freezer monitor that runs a year on one battery, a vibration monitor that analyzes its own data, an old stack moved onto an RTOS, and an enclosure designed for CE.

    Those years taught me to start from the hardware. Power, buses and radio set the limits the code has to live within.

    full write-up →

    alporak.com/work/dvm
2015 – 2019
B.Sc. Electrical & Electronics Engineering · Eskişehir Technical University
  • Thesis, in a team of four: a ground station that detects drones in no-fly zones with YOLOv3, and a drone that catches them. It won 1st place at the university project fair.
    how: A drone that hunts drones

    Cheap drones are a risk near sensitive places like government buildings and crowds. For our B.Sc. graduation project, as a team of four, we set out to take control of a drone that flies into a restricted area.

    It had to be fast and autonomous, and leave the drone undamaged. That broke down into four steps: detect it, confirm it, share its position, and catch it.

    Detecting and confirming

    The ground station picks up the radio signals the drone sends. A camera running YOLOv3, an object detector that looks at a whole image in one pass, then confirms that it’s a drone.

    Sharing and catching

    A PyQt program shares the results between the ground computers over UDP broadcast. The ground control station sends the drone’s GPS position to our own predator drone, which we designed and built. A guidance algorithm steers it to track and catch the target. I wrote the system in C and Python.

    Conclusion

    The project won 1st place at the university project fair. The full report is on ResearchGate.

    I finished my B.Sc. in Electrical and Electronics Engineering at Eskişehir Technical University in 2019, after spending 2017–18 on Erasmus+ at VSB Technical University of Ostrava. The same year, I started as an electronics engineer at DvM.

    full write-up →

    alporak.com/work/thesis
  • Erasmus+ exchange at VSB Technical University of Ostrava, 2017–18.

Daily

  • C
  • C++17/20
  • Python
  • RTOS: Nucleus, Zephyr, Keil RTX
  • Cellular modems, AT, PDP, FOTA
  • GNSS, NMEA
  • BLE
  • MQTT, TLS
  • J-Link, GitLab CI

Regular

  • CAN, OBD-II, tachograph
  • RS232/485, Modbus, 1-Wire
  • I2C, SPI, UART, ADC
  • mbedTLS, TLS 1.3
  • CppUTest, Ceedling, pytest
  • Ghidra, Wireshark, sniffers
  • Docker

Used on projects

  • TypeScript, React Native
  • Kotlin, C#/.NET
  • FastAPI, Node.js
  • Mbed OS, FreeRTOS
  • Altium, KiCad
  • Fusion 360

more: all 24 write-ups, tools I built

alp ~ $ work

Work

Twenty-four write-ups from real hardware and real releases.Each one is written up in full, with the method and the proof, at www.alporak.com.

Crashes & resets

21  to 0One reset, two bugs, one of them in a closed libraryresets in a weekend A/B testTrackers sometimes reset during firmware downgrades and left no crash record. I fixed a watchdog bug, and a weekend test went from 21 resets to 0. QA still saw failures, though. The rest was hiding in a closed vendor library. about 1/week  to 0A weekly crash and one memory dumpabnormal restarts on the stability stand, one-week runA crash showed up about once a week on our stability stand, and never on a desk. One memory dump showed two tasks freeing the same MQTT buffers. I made the race happen on demand, then fixed it. 10  to 20  to 30 minHung modems that recover on their ownwithout data before modem restart, pin reset, device resetWhen a modem hung, trackers lost data, and a failed operator scan could leave them stuck for good. I built recovery in steps at 10, 20 and 30 minutes without data, tested it on five modem types, then fixed what broke along the way. 42 blocksTLS ate the RTOS buffer pooltaken by one TLS handshakeA tracker rebooted every 20 s during the TLS handshake, but not on every boot. Every boot that crashed was running a vehicle-bus scan. TLS and the scan were drawing from the same RTOS buffer pool, so I gave TLS its own. 100+One SMS, 100 replies, and a reboot loopSMS replies to one command, then an endless reboot loopOne SMS command got over 100 replies. One modem event put trackers in a reboot loop. Both times the modem said something the firmware didn’t expect. I capped the resends, taught the parser the new format, and five blocked tickets were cleared.

Wrong or lost field data

5.5 hGNSS off for 5.5 hours, and a cold start that liedGNSS module off after a lost requestOne lost “on” request kept GNSS off for 5.5 hours while the firmware said it was on. Separately, a test claimed 1–3 s cold starts because it read a stale NMEA line. I fixed both. Real cold starts are 30–65 s. 700 units700 trackers on Null Islandsent a GPS fix with zero coordinatesAfter a firmware update, 700 trackers sent positions at 0, 0 while claiming a GPS fix. The fix flag switched on before the NMEA data was stable. I made it wait, then fixed two older bugs that the wait exposed. 1/11  to 11/111 packet out of 11 survived the outagebuffered serial packets delivered after an offline periodAfter an outage, only 1 of 11 buffered serial packets reached the server, because every packet was saved as sequence 0. I bisected it to a merge from an unrelated feature, fixed it, and wrote the record database’s first unit tests. 201 of 201Lost records, lost commands, no device bugrecords found by my parser; the old tool read 101Records missing from a download, and commands that stopped reaching trackers. Both looked like firmware bugs. One was an old reader tool, the other a carrier NAT that forgot devices after about 15 seconds of quiet. 3 h  to 1.5 hThe battery gauge that took 3 hours to say 10%for the gauge to show the first 10%Three battery complaints turned out to be three different bugs. One gauge took 3 hours to show its first 10%, and now takes 1.5. Another got stuck at 99%; it now reaches 100%, proven with a simulated −40 mV ADC offset.
alp ~ $ work the-job

When it was not the firmware

A lot of firmware work happens around the code. I’ve traced about 150 support cases to a root cause, often showing with data that the firmware wasn’t at fault, and driven about 140 release tickets.

about 140release tickets driven, from base firmware to app builds

This is the work around the code at Teltonika since 2022. In that time I’ve made 903 commits, implemented about 567 tickets, investigated about 313 more, and driven about 140 release tickets.

Tickets by my role, 2022–2026

On the platform maintainer rotation, I answer the questions that QA and support bring to the firmware team. Plenty of those end with “the firmware is fine”. That answer only counts if there’s data behind it.

Releases

I own releases on the tracker platform: five base lines, about 15 customer branches, experimental builds and mobile app builds. Each release comes with notes telling QA what to test. I carry base updates and cherry-picks across the customer branches, and I cancel and redo a release when QA results call for it. One base release carried 53 tickets. I also added a CI check that blocks a customer release if its firmware entry is missing.

Support

I’ve answered about 150 support cases with a root cause found in device logs.

In one case an ACK went missing. I rebuilt the test traffic against my own TCP server. The device sent two messages, and the server received them as one 1,388 B block. The server’s parser was the problem, and I suggested splitting messages by their length field.

The device sent two messages and the server read one 1,388 B block, so its parser caused the missing ACK A support case I rebuilt with my own test: a record every 30 s and a command every 5 s, sent to a test TCP server. As tested: the device sends message 1 and message 2 as two separate sends. The server reads both in one 1,388 B block. Its parser takes the block as one message, the ACK goes missing and the link closes. Proposed fix: the parser reads the length field and splits the block into message 1 and message 2, and each one gets its ACK. The device was not at fault. QA fixed 2 problems on their side and the tests passed. The widths of the two messages inside the block are not drawn to scale. TEST LOAD · a record every 30 s, a command every 5 s, test TCP server DEVICE SENDS SERVER READS PARSER AS TESTED · one read taken as one message message 1 send 1 message 2 send 2 1,388 B in one read message 1 + message 2 parsed as one message no ACK, link closed PROPOSED · split the block by its length field message 1 send 1 message 2 send 2 msg 1 msg 2 1,388 B, cut where the length field says message 1 → ACK message 2 → ACK The device sent both messages separately. QA fixed 2 problems on their side and the tests passed. Message widths inside the 1,388 B block are not drawn to scale.
The device sent two messages and the server read one 1,388 B block, so its parser caused the missing ACK

In another, a reported 1.5–2 minute clock error wouldn’t reproduce on two modem types and many SIMs. The evidence pointed to the operator’s network time.

Networks and vendors

A carrier NAT that forgets devices after 15 quiet seconds became half of another write-up. One escalation covered thousands of devices on one network, and forcing 4G with manual band selection solved it.

When a problem might be inside the modem, I build the case. I captured the modem’s internal logs on the bench and sent them to the vendor. One case took months of back and forth, and the fix reached the customer in a base update. I also turn vendor advice into firmware work, like a work item for faster switching between 2G and 4G.

Estimates

I’ve written at least 25 research documents with effort estimates, from 30 h to 332 h. The dual APN estimate came to 116 h. It showed that only one TLS session fits in RAM, and the customer accepted a design built around that.

Conclusion

About 150 support cases with a root cause, about 140 release tickets, at least 25 estimates, and about 567 tickets implemented.

“Not our bug” is a finding too, and it needs the same proof as a fix: a rebuilt test, a trace, a modem log.

Tags release modem network field-bug

alp ~ $ work dvm

Three years of building whole devices

For three years I built complete IoT devices: drivers, firmware, updates and the enclosure for CE. One Wi-Fi freezer monitor runs for a year on its battery.

1 yearbattery life, Wi-Fi freezer monitor in deep sleep

From 2019 to 2022 I was an electronics engineer at DvM IoT Solutions. We built IoT devices, and I worked on every layer: drivers, firmware, updates and the enclosure.

When you build the whole device, everything is connected. Battery life depends on what the firmware does between readings. The enclosure decides whether the product passes CE.

A freezer monitor that lasts a year

I designed an ultra-low-power temperature monitor for freezers, on a Wi-Fi MCU. It reads its sensors over I2C and spends the rest of the time in deep sleep. I tuned that sleep until one battery lasted a year.

A vibration monitor that does its own FFT

For predictive maintenance, I built a vibration and temperature monitor on a Cortex-M MCU with NB-IoT. It runs an FFT on its own vibration data, checks the result against thresholds, and sends an alert over NB-IoT when one is crossed.

Drivers and an RTOS

On another Cortex-M MCU, I wrote modular low-level drivers and integrated analog and digital sensors for industrial automation. I moved the old communication stack onto an embedded RTOS, which made the Ethernet communication multithreaded and connected it to the cloud.

Updates and CE

The devices can take new firmware two ways: over the air, or over USB with the device in USB client mode.

I took part in CE certification, and designed the enclosure and a breakout board to meet the safety and EMI requirements.

Conclusion

A freezer monitor that runs a year on one battery, a vibration monitor that analyzes its own data, an old stack moved onto an RTOS, and an enclosure designed for CE.

Those years taught me to start from the hardware. Power, buses and radio set the limits the code has to live within.

Tags power dsp network rtos

alp ~ $ work thesis

A drone that hunts drones

Team of four. A ground station spots a drone in a no-fly zone, YOLOv3 confirms it, and we guide our own drone to catch it. 1st place at the university project fair.

1st placeuniversity project fair, B.Sc. thesis

Cheap drones are a risk near sensitive places like government buildings and crowds. For our B.Sc. graduation project, as a team of four, we set out to take control of a drone that flies into a restricted area.

It had to be fast and autonomous, and leave the drone undamaged. That broke down into four steps: detect it, confirm it, share its position, and catch it.

Detecting and confirming

The ground station picks up the radio signals the drone sends. A camera running YOLOv3, an object detector that looks at a whole image in one pass, then confirms that it’s a drone.

Sharing and catching

A PyQt program shares the results between the ground computers over UDP broadcast. The ground control station sends the drone’s GPS position to our own predator drone, which we designed and built. A guidance algorithm steers it to track and catch the target. I wrote the system in C and Python.

Conclusion

The project won 1st place at the university project fair. The full report is on ResearchGate.

I finished my B.Sc. in Electrical and Electronics Engineering at Eskişehir Technical University in 2019, after spending 2017–18 on Erasmus+ at VSB Technical University of Ostrava. The same year, I started as an electronics engineer at DvM.

Tags python

alp ~ $ work krepza

Krepza: tracking Lithuanian grocery prices

A self-hosted side project. A scraper, a search service and a web app, all on one database.

3 servicesweb, scraper and search, on one shared database

krepza.lt compares grocery prices across Lithuanian shops. You make a shopping list and see what each item, and the whole list, costs in each shop.

What it does

Besides the shopping list, it has a deals browser, price history charts, a nearest-shop finder, and Lithuanian and English versions.

How it’s built

Three services run in Docker Compose and share one SQLite database. A scraper, with one adapter per shop, refreshes prices on a schedule. A Python search service uses multilingual embeddings for product search, and an LLM to clean up brands and categories. A Next.js app serves the site and the API.

Conclusion

It runs on my home server, and tagging a version deploys it. Three services, one database and one deploy step keep it small enough to run on my own.

Tags apps python

alp ~ $ work bt-leak

One reset, two bugs, one of them in a closed library

Trackers sometimes reset during firmware downgrades and left no crash record. I fixed a watchdog bug, and a weekend test went from 21 resets to 0. QA still saw failures, though. The rest was hiding in a closed vendor library.

21  to 0resets in a weekend A/B test

Some trackers reset in the middle of a firmware downgrade. It happened on slow 2G links, under load, and not every time. Worse, no crash record survived, so there was nothing to start from.

It turned out to be two bugs. One was ours. The other was inside a Bluetooth library we only had as a binary.

Bug one: the watchdog

A new watchdog re-ping was too strict in three ways. It treated a late answer to a retransmitted ping as a protocol violation. It could post a ping into a full queue, which is fatal on this RTOS. And it reset a task on its first timeout, even when the task was only starved of CPU.

I changed it to accept a late answer as proof of life, skip the ping when the queue is full, give a starved task one more window, and log which task failed. Then I ran the old build and mine side by side over a weekend. The old one reset 21 times: 4 misread late answers and 17 MQTT watchdog timeouts. Mine reset 0 times.

Resets over a weekend, side by side

Bug two: 11 empty scans

That looked like the end of it. It wasn’t. QA still saw failures in about 20% of runs.

Every one of those reports had the same thing right before the reset: 11 Bluetooth inquiries in a row that found nothing.

[t+  0.0 s] bt: inquiry start
[t+ 10.0 s] bt: inquiry timeout, 0 results      (1)
[t+ 10.1 s] bt: inquiry start
          ...
[t+110.2 s] bt: inquiry timeout, 0 results      (11)
[t+110.4 s] sys: abnormal reset, crash record lost
Reconstructed excerpt that shows the pattern. It is not the original log.

That’s too regular to be luck. The odds are about 1 in 1010. So the Bluetooth code was involved, and that code was a closed vendor library.

Inside the closed library

I disassembled it. Inside is a pool of 32 command tokens. A Bluetooth command takes a token and gives it back when it completes. In crowded BLE areas, the vendor’s controller never finishes a classic inquiry, so every scan change queues a command whose token never comes back.

After a few minutes the pool is empty. The library then crashes and takes the crash record down with it. That’s why we never had one.

The fix

Scans now wait until the pool has room and no command is pending. If the radio hangs anyway, a watchdog restarts it and brings back classic scanning, sensor scanning and advertising. Crash breadcrumbs now live in retained RAM, so the next crash leaves something behind. The watchdog code has unit tests.

One reset, two causes. A watchdog misread late answers, and a Bluetooth token pool drained. First cause, weekend A/B test on the stand with both builds side by side: the base build reset 21 times, 4 of them a late watchdog answer counted as a protocol violation and 17 of them MQTT watchdog timeouts. My build with the watchdog fix reset 0 times. QA still saw resets in about 20% of runs with no crash record, so there was a second cause. Second cause: every QA report showed 11 Bluetooth inquiries in a row that found 0 devices, then the reset. The chance of that by luck is about 1 in 10 to the power of 10. In dense BLE areas the vendor controller never completes a classic inquiry, and each scan change queues a Bluetooth command whose token is never returned, from a 32-token pool inside the closed vendor library. The pool goes from 32 free at boot, to 16 free after 16 stranded tokens, to 0 free after 32, within minutes. When it is empty, the closed library crashes and takes the crash record with it, and the device restarts with nothing to read. Fix: a new scan only starts when the pool level is fine and no command is pending, otherwise the scan is held. When the radio hangs, a watchdog restarts it and restores classic scanning, sensor scanning and advertising. Result: a 15 hour overnight soak with 0 abnormal resets, and 174 of 174 vendor panics recovered. FIRST CAUSE FIXED · weekend A/B on the stand, side by side base build 21 resets 4 late answers misread 17 MQTT watchdog timeouts my build 0 resets QA still saw resets in about 20% of runs, with no crash record. There was a second cause. SECOND CAUSE · a 32-token pool inside the closed Bluetooth library The clue: every QA report had 11 empty inquiries right before the reset. Chance by luck: about 1 in 1010 0 0 0 0 0 0 0 0 0 0 0 reset devices found, per inquiry In dense BLE areas the classic inquiry never completes, so each scan change strands one token. boot · 32 free 16 leaked · 16 free 32 leaked · 0 free pool empty, no token for the next command the closed library crashes and takes the crash record with it abnormal reset, nothing to read FIX · gate every scan, restart the radio when it hangs new scan pool level ok, no command pending? yes scan no hold the scan radio hangs watchdog restarts radio restores classic scanning, sensor scanning and advertising 15 h overnight soak: 0 abnormal resets · 174 of 174 vendor panics recovered
One reset, two causes. A watchdog misread late answers, and a Bluetooth token pool drained.

Conclusion

Each bug got its own proof. For the watchdog, the weekend A/B test went from 21 resets to 0. For the token pool, a 15 h overnight soak had 0 abnormal resets, and 174 of 174 vendor panics were recovered. The vendor got an evidence package and a service ticket, since the real fix belongs in their controller.

A clean test only proves the bug it can trigger. The 20% that didn’t go away was the lead to the second one.

Tags ble watchdog reverse-engineering field-bug

alp ~ $ work one-dump

A weekly crash and one memory dump

A crash showed up about once a week on our stability stand, and never on a desk. One memory dump showed two tasks freeing the same MQTT buffers. I made the race happen on demand, then fixed it.

about 1/week  to 0abnormal restarts on the stability stand, one-week run

In 2026, devices on our stability stand crashed about once a week. They were sending records over MQTT with TLS to a cloud broker. The RTOS reported a failed buffer release from the record-send task and reset the device.

Once a week is awkward. You can’t reproduce it on a desk, and you can’t prove a fix by waiting a week and hoping.

Working from one dump

With no way to reproduce it, I had one memory dump to work from. I traced the failed release back to link teardown. Two tasks, the MQTT close path and the record-send close path, had both torn down the same link at the same moment.

The double free

Both paths released the link’s connection buffers. The first release returned them to the pool. The second asked the RTOS to free buffers that were already free. The RTOS treats that as fatal and resets. The two closes rarely line up, so it happened about once a week.

Making it happen on demand

I added a test command that forces both closes to run at the same time. Now I could trigger the race whenever I wanted.

Then the fix: link teardown is now guarded, so the buffers are released exactly once, whichever task gets there first. The second one sees they’re gone and skips.

Two tasks closed the same link and freed the same buffers twice Before: the MQTT close task and the record-send task both tear down the same link at the same moment. The MQTT close task releases the link’s connection buffers, and they go back to the pool. Then the record-send task releases the same buffers again. The record-send task asks the RTOS to free a buffer that is already free; the RTOS treats that as fatal and resets the device. The two closes rarely overlap, so this happened about once a week. After: link teardown is guarded. Whichever task closes first releases the link’s connection buffers once. The second task checks, sees they are already released, and skips, so the buffers go back to the pool and the device keeps running. A test command forces both closes to run together, so the race can be triggered on demand. Proof on the stability stand with MQTT over TLS: before, about one abnormal restart a week. With the fixed firmware, seven days in a row with zero abnormal restarts. BEFORE · two tasks close the same link at the same moment MQTT close link buffers record-send close link release buffers held back in pool, free close link release again device reset The second release asks the RTOS to free a buffer that is already free. That is fatal. AFTER · guarded teardown, buffers released once MQTT close link buffers record-send close link release once held back in pool, device runs on close link already freed Whichever task closes first frees the buffers. The other one skips. A test command forces both closes together, so the race runs on demand. PROOF · stability stand, MQTT over TLS, one week before about 1 abnormal restart a week fixed day 1 day 2 day 3 day 4 day 5 day 6 day 7 0 abnormal restarts
Two tasks closed the same link and freed the same buffers twice

Conclusion

The fixed firmware ran a full week on the stand with no abnormal restart. To be fair, one clean week is weak proof for a bug that showed up once a week.

What made this solvable was the test command. A rare race doesn’t need a longer soak. It needs a way to force it.

Tags mqtt tls rtos memory field-bug

alp ~ $ work modem-recovery

Hung modems that recover on their own

When a modem hung, trackers lost data, and a failed operator scan could leave them stuck for good. I built recovery in steps at 10, 20 and 30 minutes without data, tested it on five modem types, then fixed what broke along the way.

10  to 20  to 30 minwithout data before modem restart, pin reset, device reset

Trackers in the field lost data whenever their modem stopped working. Nobody is around to power-cycle a tracker in a vehicle, so the firmware has to notice and fix it by itself.

Restarting a modem is the easy part. The hard part is knowing when to do it, how hard to push, and making sure there’s no path where the tracker just gives up.

Escalating step by step

Recovery is driven by how long the tracker has sent no data. At 10 minutes it restarts the modem radio with CFUN. At 20 it resets the modem through its hardware pin. At 30 it resets the whole device. Cheapest step first. I tested it on five modem types.

One watchdog instead of two

I wrote the user story to merge this recovery into the old watchdog and remove the duplicate code: 137 lines added, 183 removed. The watchdog period now grows with the configuration, so a tracker that sends rarely isn’t mistaken for a hung one. Only one recovery can fire per 10 minutes. I documented it with flowcharts and a table for every counter.

Three recovery steps on a no-data clock, and a fast path for short drops Top, drawn to scale on an axis from 0 to 30 minutes without data sent: at 10 minutes the firmware restarts the modem radio with CFUN, at 20 minutes it resets the modem through its hardware pin on modems that support it, at 30 minutes it resets the whole device. At most one recovery fires per 10 minutes, and each 10 minute period grows by the larger of minimum records times record period and the send period. Tested on five modem types. Bottom, schematic: before, a short network drop went unnoticed and the tracker waited out a long timeout, a latency of 1 minute 10 seconds. After, a short drop puts the tracker in a suspicious state, and the next communication error starts a full recovery at once. The LTE-M recovery goes layer by layer: check registration with CEREG?, check the data context, reactivate the data context, restart the radio with CFUN. The sequences also run in deep sleep, where records had been delayed 6 to 7 minutes instead of 2. ESCALATION · time with no data sent · to scale 0 10 min 20 min 30 min 1 · restart modem radio, CFUN 2 · reset modem by hardware pin, where supported 3 · reset the whole device At most one recovery per 10 min. Tested on five modem types. Each 10 min period grows by max(min records × record period, send period). FAST PATH · after a short network drop schematic before short drop goes unnoticed long timeout recovery 1 min 10 s after short drop "suspicious" next comm. error full recovery at once LTE-M RECOVERY · layer by layer, in order 1 · registration CEREG? 2 · data context check 3 · data context reactivate 4 · radio CFUN restart Works in deep sleep too, where records had been delayed 6 to 7 min instead of 2.
Three recovery steps on a no-data clock, and a fast path for short drops

What the merge broke

The merge led to about 12 follow-up bugs, and I fixed them. The watchdog counted per server, even though all servers share one modem, so I made it one counter. A CFUN restart was clearing the no-GPRS counter. A record period longer than the watchdog period caused pointless modem restarts. And one modem type has no pin reset, so that step only runs where it’s supported.

Short drops and deep sleep

A short network drop went unnoticed, so the tracker waited out a long timeout and took 1 min 10 s to react. I added a “suspicious” state: after a short drop, the next communication error starts a full recovery right away.

For LTE-M I built a step-by-step sequence: check registration with CEREG?, check and reactivate the data context, then restart the radio. I made these work in deep sleep too, where records had been arriving 6–7 minutes late instead of 2.

A scan that could strand a tracker

The operator scan ignored errors, so an empty result left the tracker stuck for good. On top of that, a dropped data context banned the operator, started no new search, and reset the recovery counter.

Now the scan result is checked, and retries happen at 30, 60 and 120 s instead of every 5 minutes. After 3 failures the modem network restarts, a dropped context starts a search right away, and a ban no longer resets the counter.

First retry after a failed operator search

Conclusion

The client confirmed the LTE-M sequence works. QA could no longer reproduce the deep-sleep delay. The operator-search fix passed a test with a roaming SIM that had data suspended.

Most of the work came after the first version. Recovery code only runs when things go wrong, so its bugs show up late, one field case at a time.

Tags modem watchdog network rtos

alp ~ $ work tls-buffers

TLS ate the RTOS buffer pool

A tracker rebooted every 20 s during the TLS handshake, but not on every boot. Every boot that crashed was running a vehicle-bus scan. TLS and the scan were drawing from the same RTOS buffer pool, so I gave TLS its own.

42 blockstaken by one TLS handshake

A tracker connecting to a cloud MQTT broker kept rebooting every 20 s, in the middle of the TLS handshake. But not every boot crashed.

That was the useful part. If some boots survive, the difference between them is the clue.

What the crashing boots had in common

I lined up each crash against everything else the device was doing. Boots 1 to 7 all crashed, and all of them were running a vehicle-bus scan. The one that survived had the ignition off, so no scan ran. And at every crash, all 97 of the larger buffers in the RTOS message-buffer pool were in use.

One pool, two users

The TLS library took its small memory blocks from the same RTOS pool that tasks use to message each other. One handshake took 42 of them. The vehicle-bus scan was using the pool too. Together they emptied it, and the RTOS treats that as fatal.

TLS and the vehicle-bus scan drew from the same RTOS message-buffer pool Evidence: boots 1 to 7 were running a vehicle-bus scan and each one rebooted during the TLS handshake, about 20 seconds apart. The one boot that survived had the ignition off, so no scan ran. Before: task messages, the vehicle-bus scan and the TLS handshake all took blocks from the RTOS message-buffer pool, and one handshake alone took 42 of its small blocks. At each crash, all 97 of the pool’s larger buffers were in use. With no free buffer the RTOS treats the request as fatal and the device reboots. After: TLS takes its blocks from its own private fixed-block area, so the RTOS message-buffer pool only serves task messages and the scan. Before connecting, the device counts the free buffers in the pool. If there are enough it runs the handshake. If there are too few it reports a resource error and retries later instead of crashing. EVIDENCE · each boot against what the device was doing boot 1 scan on reboot boot 2 scan on reboot boot 3 scan on reboot boot 4 scan on reboot boot 5 scan on reboot boot 6 scan on reboot boot 7 scan on reboot one boot ignition off survived Each crash came in the TLS handshake, about 20 s after the one before. BEFORE · messages, scan and TLS share the RTOS message-buffer pool task messages vehicle-bus scan TLS handshake larger buffers at each crash: 97 of 97 no free buffer → RTOS fatal → reboot One TLS handshake took 42 of the pool’s small blocks. AFTER · TLS has its own memory, and a check before connecting task messages vehicle-bus scan message-buffer pool no TLS blocks free buffers ok? yes TLS handshake private TLS block area no resource error, retry later The scan and the handshake no longer draw from the same buffers. The reboot loop stopped.
TLS and the vehicle-bus scan drew from the same RTOS message-buffer pool

The fix

TLS now has its own private fixed-block memory, so it can’t drain the RTOS pool. Before connecting, the device also counts the free buffers. If there aren’t enough, it reports a resource error and tries again later instead of crashing. QA also got a way to create low-memory conditions on the bench.

Conclusion

The reboot loop stopped, and the customer confirmed that MQTT and vehicle-bus scanning now work together.

The crash happened in TLS, but TLS wasn’t really the problem. Sharing the pool was. When two things share a pool, the one that asks last is the one that crashes.

Tags tls rtos memory mqtt

alp ~ $ work at-contract

One SMS, 100 replies, and a reboot loop

One SMS command got over 100 replies. One modem event put trackers in a reboot loop. Both times the modem said something the firmware didn’t expect. I capped the resends, taught the parser the new format, and five blocked tickets were cleared.

100+SMS replies to one command, then an endless reboot loop

The firmware talks to the cellular modem through AT commands. It sends a command, and the modem answers or reports events on its own. Every handler in the firmware assumes something about what the modem will say.

Here are two times, a year apart, when the modem said something else.

100 replies to one SMS

In 2022 a user sent one SMS command to a tracker and got more than 100 replies.

Two things together caused it. The firmware resends an SMS reply if sending fails. And this modem returned an error for SMS sends even when the message had gone out. So each reply reached the user, the firmware thought it had failed, and sent it again.

The firmware can’t tell a fake error from a real one, so there’s no clean fix on our side. What I added is a safety net: once 10 replies are queued, the firmware restarts the modem instead of resending forever.

A reboot loop from three words

In 2023, trackers with one of our external modems went into an endless restart loop, with the default configuration. Five other tickets were blocked behind this one.

The modem reports packet data events with +CGEV. The handler for it sits in about 3,000 lines of URC handling in C. This modem also sent three-word events like +CGEV: ME PDN ACT. The handler had no case for that, parsed it wrong, and the device fell into a restart loop.

I added the three-word cases, ME PDN and NW PDN, so they’re parsed in their own format. The fix was merged to the release branch the same week.

The reboot loop: a three-word modem event had no case in the handler and put devices in a restart loop Schematic flow, not to scale. Before: the device boots with the default configuration. The modem sends the packet data event +CGEV: ME PDN ACT, which has three words after the colon. It reaches the +CGEV handler, part of about 3,000 lines of URC handling in C. The handler has no case for three-word events, so it mis-parses the event and the device restarts abnormally, then boots and restarts again in an endless loop. Five other tickets were blocked behind this crash. After: the same event reaches the handler, which now has the three-word cases ME PDN and NW PDN. The event is parsed as its own format and the device keeps running. The fix was merged to the release branch in the same week and all five dependent tickets were unblocked. BEFORE · the handler had no case for three-word events schematic device boots default config +CGEV: ME PDN ACT three words after the colon +CGEV handler ~3,000 lines of URC handling in C no case for three words unhandled format → restart abnormal restart endless restarts dependent tickets 1 2 3 4 5 5 blocked behind this crash AFTER · three-word cases added, merged the same week device boots default config +CGEV: ME PDN ACT same event +CGEV handler same handler, two cases added new cases: ME PDN, NW PDN parsed as its own format keeps running dependent tickets 1 2 3 4 5 all 5 unblocked the same week
The reboot loop: a three-word modem event had no case in the handler and put devices in a restart loop

Back to the same event, later

I came back to +CGEV more than once. In 2024 I used +CGEV: ME DETACH in traces to explain why a link closed right after a server command, and showed the flow was the same on every modem. In 2025 I changed how another modem family handles a network detach: the firmware now asks the modem for the context’s current IP, and keeps the connection open if the modem has already recovered.

Conclusion

The restarts stopped, and all five blocked tickets were cleared. The SMS cap stops the loop, but it doesn’t fix the modem’s false error.

Both bugs come from the same habit: trusting whatever the modem says. A modem handler has to cope with answers that are wrong, and with answers it has never seen.

Tags modem field-bug

alp ~ $ work gnss-honesty

GNSS off for 5.5 hours, and a cold start that lied

One lost “on” request kept GNSS off for 5.5 hours while the firmware said it was on. Separately, a test claimed 1–3 s cold starts because it read a stale NMEA line. I fixed both. Real cold starts are 30–65 s.

5.5 hGNSS module off after a lost request

A tracker drove out of an underground car park and didn’t get a fix for 5.5 hours. The logs showed the GNSS module had never been powered on, even though the firmware said it was.

Separately, the firmware’s fix-time test was reporting cold starts of 1–3 s. That’s far too good to be true.

Two different bugs, same problem: the firmware was reporting something about the GNSS that wasn’t real.

Reproducing the race

I suspected a race in the restart sequence. I added a test hook that floods the GNSS request path, and that reproduced it on the bench: the firmware says GNSS is on, the module stays off.

How a request got lost

A restart sends “off”, then “on” 2 s later. Requests were de-duplicated against the module’s current state. When “on” arrived, “off” hadn’t been applied yet, so the module still looked on, and “on” was dropped as a duplicate. Then “off” completed, and nothing turned it back on.

The fix

I added a pending-request counter and a FIFO, so requests can’t be lost or reordered while an earlier one is still being applied. As a backup, a check runs every second and resends the request if the wanted and actual states differ for 30 s.

Switching GNSS on or off by command went from 45 s to 3 s, with no reboot. In QA, two trackers got their fix back quickly in driving and long-term tests.

Time to switch GNSS on or off by command

The test that passed too fast

With tick logging, the test “passed” after about 215 ticks. The raw NMEA showed why. A valid $GNGGA line from before the reset arrives after the reset command but before the module’s start message, and the firmware took it as a fresh fix.

I added a 300-tick guard window, about 1.3 s, that ignores GNSS status right after a reset. A fix-holdout feature was also keeping the old status alive, so I cleared it in the two places where the test forces a fix loss. I also tried clearing the assistance data. That made no difference.

A lost "on" request, and a stale NMEA line read as a fix Before: a GNSS restart sends "off" at 0 s and "on" at 2 s. Requests are de-duplicated against the module’s actual state. The off has not been applied yet, so the module still looks on and the "on" request is dropped. Then the off completes, and nothing turns it back on. The firmware reports GNSS on while the module stays off, with no fix for 5.5 hours. After: the "on" request waits in a FIFO with a pending-request counter and runs after the off is applied, so the module powers on. A check runs every 1 s and sends the request again if the wanted and actual states differ for 30 s. Fix-time test: after the reset command, a stale but valid $GNGGA line from before the reset arrives before the module's start message. The old test took it as a new fix at about 215 ticks. A 300-tick guard window, about 1.3 s, now ignores GNSS status after a reset and covers that line. To scale on an axis from 0 to 70 s, the cold start reported by the old test was 1 to 3 s; the fixed test reports 30 to 65 s. BEFORE · restart sends off, then on 2 s later (not to scale) request off · t=0 s on · t=2 s module looks on: dropped module off not applied yet, looks on stays off · no fix for 5.5 h firmware reports GNSS on The firmware state and the real module state disagree, and stay that way. AFTER · the request waits its turn, a 1 s check backs it up request off · t=0 s on · t=2 s FIFO + pending count module off not applied yet, looks on powered on · fix regained check every 1 s: if wanted ≠ actual for 30 s, send again FIX-TIME TEST · what arrived after the reset command 1 · reset command 2 · stale $GNGGA, still valid 3 · module start message Line 2 is from before the reset. The old test took it as a new fix. TICKS AFTER RESET · to scale guard window · 300 ticks ≈ 1.3 s 0 300 ≈215: old test "passed" REPORTED COLD START · to scale old test 1–3 s fixed test 30–65 s 0 30 60 s
A lost "on" request, and a stale NMEA line read as a fix

Conclusion

GNSS now switches in 3 s instead of 45, and requests no longer get dropped. The test reports real numbers: cold start 30–65 s, warm about 3–5 s with assistance data, hot 3–4 s.

A result that looks too good deserves the same suspicion as one that looks bad. The cold-start number was off by more than ten times, and nothing about it looked broken.

Tags gnss rtos testing field-bug

alp ~ $ work zero-coords

700 trackers on Null Island

After a firmware update, 700 trackers sent positions at 0, 0 while claiming a GPS fix. The fix flag switched on before the NMEA data was stable. I made it wait, then fixed two older bugs that the wait exposed.

700 unitssent a GPS fix with zero coordinates

In 2024, after a firmware update, 700 trackers in the field started sending records with zero coordinates. The records said they had a GPS fix.

0, 0 is a spot in the Gulf of Guinea, known as Null Island. A record without a fix is just a gap. A record with a “fix” at 0, 0 is worse, because the server puts it on the map.

The flag moved too early

I reproduced it. The GPS fix signal switched to “fix” before NMEA stabilization had finished. Any record created in that window had a fix but no real position.

So I made the fix signal wait until NMEA is stable. In factory test mode the wait is skipped, so production tests stay fast.

A flag that never reset

While wiring that in, I found that the unstable-fix flag was never reset on sleep or on GPS loss, so it carried over into the next fix. I reworked it. The unstable-fix flag now only starts NMEA stabilization, and a separate “NMEA stable” flag feeds the fix filter, the same way the speed and altitude filters do.

What the wait exposed

The new delay uncovered an ordering bug. “Update good coordinates” ran before “update GPS fix state”, so coordinates changed while the status still said no fix, and went stale after ultra-deep sleep. I swapped the order, so coordinates only update on a stable fix, and the fix status now resets on sleep.

I also added GNSS status 4, “filter active”, so testers can see why coordinates are being held.

The fix flag waits for stable NMEA, and coordinates follow the fix state Schematic timeline, not to scale. Before: once the module gets a fix, NMEA goes through a stabilization window. The fix IO flipped to fix at the start of that window, so a record created inside it said fix and carried coordinates 0, 0. After: the fix IO flips only when NMEA is stable, so a record inside the window says no fix. Factory test mode skips the wait. Second part, the order of each GNSS update. Before: update good coordinates ran before update GPS fix state, so coordinates changed while the status still said no fix. After: the unstable-fix flag only starts NMEA stabilization. A separate NMEA stable flag feeds the fix filter together with the speed and altitude filters. The fix filter sets the GPS fix state first, and good coordinates update only when the fix is acquired and stable. Sleep resets the current fix status, and GNSS status 4, filter active, shows testers why coordinates are held. Test drives on 3 device families produced no zero-coordinate records. BEFORE · the fix IO flips while NMEA is still stabilizing schematic NMEA before fix stabilizing stable fix IO no fix fix (too early) fix record: fix = yes, coordinates 0, 0 AFTER · the fix IO waits for NMEA stable NMEA before fix stabilizing stable fix IO no fix fix record: fix = no Factory test mode skips the wait, so production test time stays short. ORDER IN EACH GNSS UPDATE before 1. good coordinates 2. GPS fix state coordinates change while the status says no fix after unstable-fix stabilize NMEA NMEA stable speed filter altitude filter fix filter 1. GPS fix state only if acquired and stable 2. good coordinates Sleep resets the current fix status. GNSS status 4, filter active, shows testers why coordinates are held. Test drives on 3 device families: no zero-coordinate records.
The fix flag waits for stable NMEA, and coordinates follow the fix state

Conclusion

Test drives on 3 device families produced no zero-coordinate records. All three fixes touch the same state, so I put them in one test firmware, and QA verified them together.

Adding a delay changes the order things happen in, and that brings hidden ordering bugs to the surface. The first fix was small. Most of the work was in what it uncovered.

Tags gnss field-bug

alp ~ $ work offline-loss

1 packet out of 11 survived the outage

After an outage, only 1 of 11 buffered serial packets reached the server, because every packet was saved as sequence 0. I bisected it to a merge from an unrelated feature, fixed it, and wrote the record database’s first unit tests.

1/11  to 11/11buffered serial packets delivered after an offline period

In RS232/RS485 buffered mode, the tracker stores data from a serial sensor and sends it when the server is reachable again. But after an outage, only the last packet was left.

Finding the change

Buffered mode had worked before, so something had broken it. The record database stores records by sequence number in a nested folder tree. I bisected on the device across releases and merges, and landed on a merge from an unrelated feature: a rewrite of the FIFO record database.

The flag nobody read

The multi-save call still took a sequence flag, but the rewritten code never read it. So every packet was written as sequence 0. The database then decided the file was corrupted and erased it, leaving only the newest packet.

Eleven packets buffered offline, one delivered The serial sensor sends 11 packets while the tracker is offline. Before the fix, the multi-save call took a sequence flag that the rewritten FIFO record database never read, so every packet was written to sequence 0. The database treated the file as corrupted and erased it, so only the newest packet, number 11, was left. After reconnecting, 1 of 11 packets was delivered. After the fix, each buffered packet gets its own storage place in the record store, and all 11 of 11 packets are delivered after the offline period. Each square is one packet, so the counts are drawn to scale. BEFORE · the sequence flag was never read packets sent while offline 1 2 3 4 5 6 7 8 9 10 11 record store multi-save, flag ignored all 11 → sequence 0 file seen as corrupted, erased; newest kept delivered 1 of 11 11 1 to 10 lost AFTER · each packet gets its own storage place packets sent while offline 1 2 3 4 5 6 7 8 9 10 11 record store 11 places 1 2 3 4 5 6 7 8 9 10 11 delivered 11 of 11 1 2 3 4 5 6 7 8 9 10 11 Same 11 packets. I bisected the loss on the device to a merge from an unrelated feature. One square is one packet.
Eleven packets buffered offline, one delivered

Two more problems on the same path

Buffered mode always used the TCP path, because only the non-buffered path checked the server type. So a device set up for MQTT never sent its buffered data at all. And in the JSON codec for the cloud, serial data over 18 B was dropped and 0x0A/0x0D bytes were changed. There was also an old edge case in delete-all.

The fixes: each buffered packet gets its own storage slot, which also fixed delete-all. Buffered mode now uses the same server-type check as the rest, so it works over TCP and MQTT, ASCII and binary. The JSON codec sends the full serial data as a hex string.

Conclusion

After an offline period, 11 of 11 packets now arrive, instead of 1 of 11.

Serial packets delivered after an offline period

Nothing caught the original break because the record database had no unit tests. I wrote the first ones, with an in-memory fake file system, a 269-line fake record-manager harness and a 5-step test plan.

The symptom was in one feature and the cause was in another. Bisecting is what connected them.

Tags records testing mqtt field-bug

alp ~ $ work not-the-device

Lost records, lost commands, no device bug

Records missing from a download, and commands that stopped reaching trackers. Both looked like firmware bugs. One was an old reader tool, the other a carrier NAT that forgot devices after about 15 seconds of quiet.

201 of 201records found by my parser; the old tool read 101

When data goes missing between a tracker and a server, everyone looks at the device first. It’s the part we build.

These two tickets were both about lost data, and in both the device turned out to be fine. The work was proving where the data really stopped.

Missing records

The first ticket estimated that the “Read records” download in the configuration tool was missing about 30% of the records on the device. If true, that’s a firmware data-loss bug.

The desktop tool for opening the downloaded .bin file had been discontinued. So I wrote my own parser in Python. It decodes the device’s record protocol straight from the dump and counts every record.

On the file I checked, the old tool showed 101 of 201 records. My parser found all 201. The old tool just didn’t read them.

Records decoded from the same .bin download

I published the parser in our internal tools docs, so anyone can check a download themselves. QA retested the download and it passed.

Missing commands

In the second ticket, trackers sent data to the server over UDP, and the commands the server sent back stopped arriving.

I reproduced it first. The modem was the obvious suspect, so I opened a case with the modem vendor. Then I suspected the forwarding on our office network. Neither explained it. The ticket stayed open from 2025 until I found the cause in February 2026.

A NAT with a short memory

On this operator’s APN, the device had no public address. It sat behind the operator’s carrier-grade NAT, which keeps a port mapping open only while traffic flows both ways. On this network, the mapping closed after about 15 seconds of quiet. Any command after that went to a port the NAT had forgotten, and was dropped before it reached the modem.

The fix was a setting: the existing network ping now runs every 10 s, so the NAT never sees 15 seconds of quiet. I also proposed a default ping interval for UDP links in the firmware, and updated the test automation script.

The lost commands: the carrier NAT forgets the device after about 15 s of quiet; a ping every 10 s keeps it open Path: the tracker has no public address and sits behind the operator's carrier-grade NAT, which maps a public port to the device. Data goes up through the NAT to the server, and server commands come back down through the same mapping. The mapping lives only while traffic flows both ways. Timeline drawn to scale on an axis from 0 to 30 seconds. Before: after the last data at 0 s the link goes quiet, the NAT mapping stays open until about 15 s, then closes, and a server command sent after that is dropped at the NAT. After: with the network ping timeout set to 10 s, the device sends the last data at 0 s and then a ping after every 10 s of quiet, at 10, 20 and 30 s. Each send opens a new window of about 15 s, the next ping arrives inside it with about 5 s to spare, the mapping stays open the whole time, and the server command is delivered. PATH · one UDP mapping carries both directions tracker no public address operator carrier-grade NAT public port → device server data, ping command The mapping lives only while traffic flows both ways. Here it closed after ~15 s of quiet. BEFORE · data, then quiet last data command dropped NAT mapping open closed, port forgotten ~15 s without two-way traffic AFTER · network ping timeout set to 10 s last data ping after 10 s quiet command delivered NAT mapping open the whole time each send: ~15 s window the next ping lands ~5 s before it closes 0 10 15 20 30 s Every ping lands inside the window the last one opened, so the NAT never forgets the port.
The lost commands: the carrier NAT forgets the device after about 15 s of quiet; a ping every 10 s keeps it open

Conclusion

Records: all 201 were in a file the old tool read as 101. Commands: with a ping every 10 s, they reach the trackers again. The 201 is from the one file I checked, and 15 s is this operator’s timeout, not a universal number.

Neither fix touched firmware code. The NAT case took long because the first two suspects were reasonable ones.

Tags records network python field-bug

alp ~ $ work battery

The battery gauge that took 3 hours to say 10%

Three battery complaints turned out to be three different bugs. One gauge took 3 hours to show its first 10%, and now takes 1.5. Another got stuck at 99%; it now reaches 100%, proven with a simulated −40 mV ADC offset.

3 h  to 1.5 hfor the gauge to show the first 10%

Three battery complaints came in from the field. On one model, the gauge took 3 hours to show its first 10%. Some devices stopped charging below 4.15 V. Others sat at 99% forever after a full charge.

It looked like one battery problem. It was three separate bugs.

The wrong table

The slow model was using the percentage table for a 170 mAh cell on a 1,000 mAh battery. I built new tables for 170, 1,000 and 1,800 mAh cells. I also wrote and documented an algorithm that moves toward the measured value and can’t get stuck.

Time for the gauge to show the first 10%

Believing the charger too early

The early stop came from the charger chip reporting DONE too early, and the firmware believing it. Now the firmware retries up to 3 early DONE signals before it accepts one. The noisy charger status lines also got a 3-sample debounce with hysteresis.

Stuck at 99%

The charger stops at 4,125–4,140 mV, but the table needs at least 4,148 mV for 100%. So a full battery always came out 8–23 mV short. On top of that, an older guard blocked the 10-minute recalculation, so the gauge never moved off 99%. Now, when the firmware accepts the battery as full, it sets 100%.

Charger DONE retries, and why a full battery read 99% Part 1, the charger state machine. Before, the first DONE from the charger IC turned charging off for good, below 4.15 V. Now, while charging, a DONE with fewer than 3 retries toggles the charger and adds one retry, then charging resumes. Retries reset to 0 whenever the charger reports CHARGING or STANDBY. A DONE after 3 retries is accepted as full, and accepting full now sets 100%. Part 2, to scale on a millivolt axis from 4080 to 4160: the charger IC ends the charge between 4,125 and 4,140 mV, but the percentage table needs at least 4,148 mV for 100%, so the reading is 8 to 23 mV short of 100%. The test image emulates a −40 mV ADC offset, as in field readings. Part 3: before the fix, the table said 99% and an old guard blocked the 10-minute recalculation, so the gauge stayed at 99% forever. After the fix, accepting full sets 100%, and the fixed build held 100% with the −40 mV offset, which proved the fix. THE CHARGER REPORTS DONE TOO EARLY Before, the first DONE turned charging off for good, below 4.15 V. Now: CHARGING charger: DONE after 3 retries accept full set 100% new fewer than 3 retries toggle charger, retries+1 Retries reset to 0 whenever the charger reports CHARGING or STANDBY. WHY A FULL BATTERY READ 99% · to scale 100% needs ≥ 4,148 mV charge ends 4,125–4,140 mV 8–23 mV short of 100% test image emulated −40 mV ADC offset 4080 4100 4120 4140 4160 mV WHAT THE GAUGE SHOWED AFTER A FULL CHARGE before table says 99% 10-min recalc blocked by guard 99% forever after accept full set 100% held 100% with the −40 mV offset
Charger DONE retries, and why a full battery read 99%

Testing with field conditions

To test the 99% fix, I built a test image with a simulated −40 mV ADC offset, like the readings from the field. The fixed build held 100%. Each fix also has a unit test.

Conclusion

The gauge now shows 10% after 1.5 hours instead of 3, holds a steady 100% without flickering back to 99, and batteries charge to 4.15 V again.

Splitting the complaints apart early meant each bug got its own fix and its own test.

Tags power testing

alp ~ $ work mqtt-latency

Why did locking a car take 10 seconds?

Locking a car remotely had crept past 10 seconds at p90. Commands, records and the protocol engine were all stuck in one MQTT task. After I split it in three, record send p90 dropped from 4.06 s to 0.12 s.

4.06 s  to 0.12 srecord send time, p90

A fleet customer locks and unlocks cars over MQTT. The server sends a command, the tracker does it and replies. Over time, the p90 for a lock crept past 10 seconds. That’s a long time to stand next to a car, waiting for the door.

Measuring first

I wrote a script that pulls the round-trip time of every publish out of the device logs, and added latency probes at each stage. Then I changed one thing at a time, each behind its own flag, and measured again.

One task doing everything

There was no single bug. It was a traffic jam. Commands, record publishing and the protocol engine all ran in one task. That task waited up to 60 s for a socket send, and with no limit at all for an MQTT ACK. One slow write and everything behind it stalled, including a lock command stuck behind a queue of records. On top of that, every record was built twice.

Splitting it up

I split MQTT into three tasks: commands, publishing and the protocol engine. Now a task waiting on the network doesn’t hold up the others.

They still share one TLS session, so writes have to be serialized somewhere. I first tried the shared network layer. It deadlocked. It works in the layer that owns the TLS session.

Both waits now have a cap. The transport answers ok, busy or fatal, so a busy modem no longer kills the TLS session. The duplicate record build is gone. I also fixed a record-storage mode that an earlier update had broken.

One MQTT task with two open waits, split into three tasks Before: lock commands, record publishes and the protocol engine share one MQTT task. Each record is built twice, then the task waits up to 60 seconds for the socket send and then for the MQTT ACK with no limit, so a lock command queued behind records waits for every job ahead of it. After: commands, publishing and the protocol engine run in three separate tasks. They meet in the layer that owns the TLS session, which serializes writes and caps both waits. The modem transport answers ok, busy or fatal, so a busy modem no longer closes the TLS session. To-scale bars: the time one reply held the connection layer went from 495 to 1,144 ms before, to 32 to 240 ms after, on an axis from 0 to 1,200 ms. BEFORE · one task, two open waits queue lock cmd record record one MQTT task for everything record built twice socket send waits ≤ 60 s MQTT ACK no limit The lock command waits for every job ahead of it, and any job can stall on either wait. AFTER · three tasks, one session owner, capped waits lock cmd command task record publish task, 1 build protocol engine task TLS session owner serializes writes send wait capped ACK wait capped ok/busy/fatal modem A busy modem now returns busy, so it no longer tears down the TLS session. HOW LONG ONE REPLY HELD THE CONNECTION LAYER · to scale before 495–1,144 ms after 32–240 ms 0 600 1,200 ms
One MQTT task with two open waits, split into three tasks

Conclusion

Record send p90 went from 4.06 s to 0.12 s, and the median from 1.03 s to 0.05 s. The time one reply held the connection layer went from 495–1,144 ms to 32–240 ms. A calibration command went from 1,664 ms to 4 ms.

Record send time, p90

One caveat: lock replies use the same publish path, so record send time is what I measured. I don’t have the end-to-end lock time after the fix. The delay still visible in the customer’s logs came from sockets dropping while roaming.

Nothing here was slow on its own. The problem was waiting without limits, and it only showed up once each stage was measured.

Tags mqtt rtos network

alp ~ $ work one-constant

The 100-second timeout that became 7.8 hours

After a release, devices on our stability stand kept crashing. Instead of reading crash dumps one by one, I diffed the code between the good and the bad release. One constant had been multiplied twice: a 100 s timeout had become 7.8 hours.

7.8 h  to 100 smodem send timeout, bug → fixed

After a release, devices on our long-run stability stand started crashing with abnormal restarts. Different modems, different models, no obvious pattern.

When unrelated devices start failing right after a release, the release is the place to look.

Diff, not dumps

I could have gone through the crash dumps one by one. Instead I diffed the record-save, record-send and watchdog code between the last good release and the bad one. Everything matched except one constant.

100 seconds became 7.8 hours

The timeout for the modem’s data-send command was multiplied twice. 100 s became 7.8 h.

Modem send timeout

With 100 s, a send that gets no answer fails, the task moves on and answers the watchdog. With 7.8 h, the task just waits. The record path stays blocked, watchdog pings go unanswered, and the watchdog restarts the device. A wrong timeout looked like a crash.

Fixing it, and the path around it

The fix itself was one line. Since I was already in that code, I also made the path more forgiving.

The watchdog now warns at 50% and 75%, and retries pings, so a slow answer isn’t treated as a dead task. The record manager slows down at 70% load, with at most 5 s of delay. Logs are rate-limited, and I fixed memory leaks in the SMS and network code. Beacon work waits when load is above 80%.

A send that never fails fast, and the safeguards added after it Part 1 is a flow. The send task issues the modem's data-send command and gets no answer. With the designed 100 second timeout, the command fails, the task gets control back, answers the watchdog and the device stays up. With the 7.8 hour timeout in the release, the task keeps waiting, the record path is blocked, watchdog pings go unanswered, and the watchdog fires an abnormal restart. Part 2 shows the thresholds added with the fix on 0 to 100 percent scales: watchdog warnings at 50 and 75 percent of the watchdog limit with the restart at 100 percent; record manager flow control from 70 percent load with at most 5 seconds of delay; beacon work deferred above 80 percent load. Watchdog pings are retried up to 10 times so a slow answer no longer counts as a dead task. 1 WHY ONE CONSTANT RESTARTED THE DEVICE The send task issues the modem’s data-send command and gets no answer timeout 100 s, as designed timeout 7.8 h, in the release After 100 s the command fails and the task gets control back The task answers the watchdog and the record path keeps moving Device stays up The task keeps waiting, for up to 7.8 h The record path is blocked and watchdog pings go unanswered Watchdog fires: abnormal restart 2 SAFEGUARDS ADDED WITH THE FIX, THRESHOLDS TO SCALE Watchdog timer % of its limit warn warn restart 0 50 75 100 Record manager load, % flow control, max delay 5 s 0 70 100 Beacon work load, % deferred above 80% 0 80 100 Watchdog pings are also retried, up to 10 times, so a slow answer is not a dead task.
A send that never fails fast, and the safeguards added after it

Conclusion

The stand went quiet. The only restarts left were on one small model under heavy bursts, which is a known limit. The whole branch went into the base firmware, so every build ships with these safeguards.

Comparing three code paths was a much smaller search than a stand full of crash dumps.

Tags modem watchdog rtos

alp ~ $ work ram-budget

Three features against one RAM budget

400 geofences, 10,000 driver IDs and two data connections at once, on a tracker already short on RAM. I estimated each before writing code, and each hit a different limit: free RAM, boot time, or room for one TLS session.

24 kBfree RAM with 400 geofences

Tracker firmware runs on a fixed amount of RAM, and most of it is already taken. Three requests each needed more of it. A client wanted 400 geofences, each a 15-point polygon. Another project needed the driver ID list to hold 10,000 entries. A customer needed two data connections on different APNs, open at the same time.

I wrote an estimate for each before writing any code: 154 h of firmware work plus 40 h for the configuration tool, 60 h, and 116 h. Each one hit a different wall: free RAM, boot time, and room for only one TLS session.

Geofences: measure, then cut

Every polygon point lives in RAM. With 400 zones, I measured 42 kB of free RAM at 10 points per polygon, and 24 kB at 15. That was an early measurement, and it was dangerously low.

Free RAM with 400 zones, preliminary measurement

Coordinates come with six decimal places, about 10 cm, and were stored as doubles. I pushed for 32-bit floats instead: half the bytes per coordinate, and a little less precision. I also pushed back on per-zone names as scope creep. It still didn’t fit, so I removed reserved features the build never used, and shortened a rarely used list to free 2.4 kB.

What the cuts broke

Trimming had removed some base parameters by mistake, so the immobilizer re-locked after unlock. I restored them. The #ifndef blocks I added to save RAM overlapped and changed an IO default, so an alarm event was always on. I patched that. And the configuration tool still treated coordinates as doubles, so every config showed unsaved changes. I updated it to float.

The geofence part: where the RAM for 400 polygons came from, and what the cuts broke Top, bytes drawn to scale: one polygon point is a latitude and a longitude. Stored as doubles it takes 8 bytes per coordinate, 16 bytes per point. Stored as 32-bit floats it takes 4 bytes per coordinate, 8 bytes per point. Coordinates arrive with six decimal places, about 10 cm; floats lose a little of that precision, which a geofence edge can afford. Every point of 400 zones with 15 points each is held in RAM. Middle, schematic: in that build, a rarely used list was shortened, freeing 2.4 kB. Removed reserved functionality the build never used also made room. Bottom: three things the cuts broke, each with its fix. The trimming removed some base parameters by mistake, so the immobilizer re-locked after unlock; restoring the parameters fixed it. #ifndef blocks added to save RAM overlapped and changed an IO default, so an alarm event was always present; it was patched in a new revision. The configuration tool still treated coordinates as doubles while the firmware used floats, so every config showed unsaved changes; the configuration tool was updated to float. QA confirmed 400 zones configurable and polygons working, and the build fits in RAM. ONE POLYGON POINT · latitude + longitude · bytes to scale lat lon double 16 bytes, 8 per coordinate float 8 bytes, 4 per coordinate a little less precise, fine for a geofence edge Coordinates arrive with six decimals, about 10 cm. Every point of 400 zones × 15 points is held in RAM. A RARELY USED LIST, SHORTENED · in that build · schematic before full list after kept removed · 2.4 kB freed Reserved functionality the build never used was removed too. WHAT THE RAM CUTS BROKE, AND THE FIX cause symptom fix trimming removed some base parameters immobilizer re-locked after unlock restored the parameters #ifndef blocks overlapped and changed an IO default alarm event always present patched in a new revision configuration tool kept doubles, firmware used floats every config showed unsaved changes configuration tool updated to float QA: “400 zones configurable and polygons working”, and the build fits in RAM.
The geofence part: where the RAM for 400 polygons came from, and what the cuts broke

10,000 driver IDs: boot was the problem

A driver touches an ID key to the 1-Wire reader, and the firmware checks it against the list for the immobilizer. With 10,000 entries the feature worked, but boot took about 100 s, against a 30 s limit. I made the list load faster at boot, and cut RAM and BSS use so the list fit.

Two connections, one TLS session

The firmware assumed there was only ever one data connection. The estimate is where the real limit showed up: there was RAM for one TLS session, not two. So I reworked the core network API to handle two PDP contexts. Both connections stay up, and secure traffic goes to one server at a time, so one TLS session is enough.

In testing, the backup connection was torn down and rebuilt on every send. The network rejected that, and the modem got stuck in a restart loop. I fixed the backup handling in several places and removed the settings-refresh loop.

Conclusion

Geofences: QA confirmed “400 zones configurable and polygons working”, and the build fits in RAM. Driver IDs: boot went from about 100 s to under 30 s, on two tracker models. Two connections: the customer accepted the design, which touches about 30 files across the network API and two modem drivers.

On a device this full, every feature costs something else. Here it was precision, unused features and a second TLS session. And cutting RAM breaks things too, so cuts need testing like any new feature.

Tags memory network tls security

alp ~ $ work heart-rate

Heart rate that fell apart when you walked

A health smartwatch lost the heart rate as soon as you started walking. I built a rig to compare it against a chest strap, then rebuilt the algorithm. Error while walking dropped from 8.2 to 5.5 bpm, and RAM from 6.9 to 2.7 kB.

8.2  to 5.5 bpmheart-rate error while moving

A health smartwatch with ECG, heart rate and SpO2 lost the heart rate as soon as you started walking. Walking is the hard case for a wrist sensor: every step moves it and adds its own rhythm to the signal.

It’s easy to tweak a heart-rate algorithm until it looks good on one walk. To know if it’s really better, you need a reference. So I built two things: a way to measure it, and a better algorithm.

The first bug was a unit

The motion detector compared a threshold of 0.10 against acceleration in the wrong unit. Every pulse was flagged as motion and thrown away.

A way to measure it

I built the whole loop. A BLE service on the watch streams per-beat PPG and heart rate, plus raw PPG at 100 Hz and accelerometer data at 50 Hz. A Kotlin Android app records the watch and a reference chest strap side by side, with activity markers. Python scripts score each session and replay the filters, so I can try a change on recorded data before it goes on the watch.

An algorithm that expects motion

The new estimator runs a DFT on a 25 Hz grid. The accelerometer gives the step rate, and notches at 0.5×, 1× and 2× that rate remove the step and arm-swing peaks. A beat-interval prior and a bounded tracker stop the estimate from jumping around, and clip, jump and artifact-hold gates keep it steady.

The old motion gate, the new estimator, and the loop that proved it Old path: the motion gate compared a threshold of 0.10 against acceleration in the wrong unit, so every pulse was flagged as motion and heart rate was lost while walking. New estimator, one estimate every 2 seconds: the accelerometer at 50 Hz gives the step rate. Raw PPG at 100 Hz goes through a DFT on a 25 Hz grid, then notches at 0.5, 1 and 2 times the step rate remove step and arm-swing peaks. The strongest remaining peak goes to a bounded tracker with a beat-interval prior that cannot jump from 90 to 140 bpm in 2 seconds, then to clip, jump and artifact-hold gates, then out as heart rate. How it was proven: a BLE service on the watch streams per-beat PPG and heart rate, raw PPG at 100 Hz and accelerometer at 50 Hz. A Kotlin Android app records the watch and a reference chest strap side by side with activity markers. Python scores every session, replays the gates and writes golden vectors. The C++ port with an optimized FFT library agrees with the Python twin to a median of 0.03 bpm. It ran in shadow mode next to the old path, then a selector with beat-gap fill switched it live, with other measurement paths unchanged. The loop then starts again with new recordings. OLD PATH · why walking lost heart rate acceleration in the wrong unit motion gate threshold 0.10 every pulse flagged as motion heart rate lost while walking NEW ESTIMATOR · one estimate every 2 s accel 50 Hz step rate raw PPG 100 Hz DFT on a 25 Hz grid step-rate notches at 0.5×, 1×, 2× strongest peak that is left bounded tracker beat-interval prior no 90 → 140 bpm in 2 s gates clip, jump artifact hold heart rate moving error 5.5 bpm The notches cut step rate and arm swing from the spectrum. The strongest peak left is the heart. HOW I PROVED IT · record, score, port, shadow, switch reference chest strap watch · BLE service per-beat PPG and HR raw PPG 100 Hz accel 50 Hz Android app · Kotlin watch + strap side by side activity markers Python scores every session replays the gates writes golden vectors C++ port · FFT library agrees with the Python twin to a median of 0.03 bpm shadow mode runs next to the old path old path still live switched live selector, beat-gap fill other paths unchanged next recordings
The old motion gate, the new estimator, and the loop that proved it

Fitting it on the watch

The MCU was already at 98% RAM. I ported the estimator to C++ with an optimized FFT library and checked it against golden vectors from Python. Before switching it on, I ran it in shadow mode next to the old one. Along the way I also fixed a stack overflow that only happened while the watch was being worn.

Conclusion

Mean absolute error while walking, against the chest strap, went from 8.2 to 5.5 bpm.

Heart-rate error while moving

The estimator’s RAM went from 6.9 kB to 2.7 kB.

Estimator RAM

The algorithm gets the attention, but the chest strap, the recorder app and the scoring scripts are what make 5.5 bpm a measurement and not a guess.

Tags dsp ble wearable

alp ~ $ work beacon-flood

4,000 beacons vs. one phone link

In crowded BLE areas, trackers lost their phone link, and some got stuck in a reset loop. I measured everything in a flood of about 4,000 beacons before touching any code. Stuck cycles went from 17 of 62 to 0 of 350.

17/62  to 0/350BLE cycles where the phone link got stuck

Trackers had two problems in places crowded with BLE devices. After a phone session ended, the tracker stayed invisible to the phone. And in very dense areas, it reset every 33 seconds.

Neither shows up at a desk. So I brought the crowd to the bench.

Measuring before changing

I set up a flood of about 4,000 beacons. Before changing any code, I measured stuck cycles, resets and how many phone sessions passed.

Getting the phone link back

For the invisible tracker, I added three layers that kick in one after another. A probe checks that the phone link is really alive. With no phone connected, an advertising watchdog restarts advertising if it gets stuck. And if the Bluetooth stack doesn’t confirm that advertising started, the radio restarts.

Stopping the reset loop

For the reset loop, scanning now runs in slices, so the scanner can’t hog the system. The records that beacons generate are also paced.

Three checks bring the phone link back; sliced scanning and record pacing end the reset loop Two columns, one per failure. Left, after a phone session the tracker stayed invisible to the phone. Three layers act in turn. A link probe checks the phone link every 10 seconds and closes the link after 2 failed probes. With no phone connected, an advertising watchdog restarts advertising every 15 seconds. If the Bluetooth stack does not confirm the advertising start within 5 seconds, a radio watchdog restarts Bluetooth. Result: the tracker is visible to the phone again, and stuck cycles went from 17 of 62 to 0 of 350 over 10 hours. Right, in very dense BLE areas the tracker reset every 33 seconds. A timeline drawn to scale over 36 seconds shows the scan before the fix as one unbroken non-stop scan, and after the fix as 10 second scan slices with 2 second gaps, so the scanner cannot hold the radio without a break. Scanning also pauses after phone-link events. Record pacing spreads out the records that beacons generate. Result: the reset loop is gone, from one reset every 33 seconds to 0 resets in 80 minutes. Tested in a flood of about 4,000 beacons. For the vendor’s side of the problem, the vendor got an air log of 1,009 sniffed connection events. The flow on the left is schematic; the scan timeline is to scale. TWO FAILURES · each with its own fixes after a phone session tracker stayed invisible to the phone link probe · every 10 s 2 failed probes → close the link no phone now advertising watchdog · every 15 s no phone → restart advertising if the stack stalls radio watchdog · 5 s timeout start not confirmed → restart Bluetooth tracker visible to the phone again stuck cycles 17/62 → 0/350, 10 h in very dense BLE areas one reset every 33 s before · non-stop scan, no gaps one scan, never a break after · 10 s slices, 2 s gaps 10 s 10 s 10 s 2 s off 0 s 12 24 36 s Scan also pauses after phone-link events. record pacing under beacon floods reset loop gone 1 per 33 s → 0 in 80 min Tested in a flood of about 4,000 beacons. For the vendor’s side of the problem, the vendor got an air log of 1,009 sniffed connection events.
Three checks bring the phone link back; sliced scanning and record pacing end the reset loop

Conclusion

Stuck cycles went from 17 of 62 to 0 of 350, over 10 hours. The reset loop went from one reset every 33 s to none in 80 minutes. Phone sessions passing went from 51% to 80%.

Phone session pass rate

80% still means one session in five failed, and my data doesn’t say why. For the vendor’s part of the problem, I sent them an air log of 1,009 sniffed connection events.

The baseline is what let me say “better” with numbers.

Tags ble watchdog field-bug

alp ~ $ work cold-chain

30,000 records over BLE in 5–6 minutes

Two BLE projects, one limit: how much data fits in a packet. In a phone app, 30,000 records now download in 5–6 minutes, every chunk CRC-checked. In tracker firmware, smaller chunks fixed the iPhone link, and I built the MTU exchange the vendor wouldn’t add.

30,000 recordsover BLE in 5–6 min, CRC-checked

BLE sends data in small packets. Both sides agree on a maximum size, the MTU, and anything bigger doesn’t get through.

I ran into this limit from two sides: a phone app that downloads logs from a cold-chain temperature recorder, and tracker firmware that iPhones couldn’t talk to.

The app: every record has to arrive

I co-developed the React Native app for a family of BLE sensors, on iOS and Android: about 140 commits, about 85 tickets and 14 releases to Google Play and TestFlight.

For the recorder’s cold-chain certification, every record has to arrive, and you have to be able to show it did. I added a CRC check on every chunk, so each row in the export says whether its chunk arrived intact. I also fixed a page-header shift, where the header sometimes arrived one slot late and the app read a record chunk as the header. Then I raised the MTU to 247 and the chunk size from 15 to 60 records.

How the app reads cold-chain records, one CRC-checked chunk at a time The read loop: the sensor stores records in pages. Each page starts with a header and holds at most 63 chunks, a firmware limit. Over a BLE link with an MTU of 247 the app reads one chunk of 60 records, computes its own CRC16 over the records, compares it with the chunk's CRC, writes Valid CRC yes or no on every CSV row, and sends next chunk. It repeats until the page ends, with leftover-record logic for the tail. The page-header shift: the next chunk command did not always refresh the characteristic, so the header arrived one slot later than the app expected and a record chunk sat in the header slot. Before the fix there was no header check, the app parsed that chunk as the header and the download stopped after 1 page. After the fix a header sanity check fails, the app re-reads the data as a record chunk, the real header comes next and all pages are read. QA download times after raising the MTU to 247 and the chunk size from 15 to 60 records, drawn to scale on an axis from 0 to 6 minutes: 6,830 records in about 1 minute, 15,510 records in about 2 minutes 26 seconds, and 30,000 records in 5 to 6 minutes. ONE PAGE ON THE SENSOR · read over BLE, MTU 247 header chunk 1 chunk 2 … chunk 63 header ≤ 63 chunks per page (firmware limit) read chunk 60 records compute CRC16 over records compare with chunk's CRC every CSV row Valid CRC Y/N send next chunk Repeat until the page ends, with leftover-record logic for the tail. THE PAGE-HEADER SHIFT · “next chunk” did not always refresh app expects chunk chunk header chunk arrived chunk chunk chunk header, +1 slot before no header check chunk parsed as the header download stopped after 1 page after header sanity check fails re-read as a record chunk real header next, all pages read QA DOWNLOAD TIMES · MTU 247, chunk 15 → 60 records · to scale 6,830 records ~1 min 15,510 records ~2 min 26 s 30,000 records 5–6 min 0 1 2 3 4 5 6 min
How the app reads cold-chain records, one CRC-checked chunk at a time

To test full-size downloads, I added a fake-record generator to the app. 30,000 records now download in 5–6 minutes.

The firmware: iPhones and 512-byte chunks

An iPhone can only reach the tracker over BLE. The tracker isn’t MFi certified, so iOS can’t use its Bluetooth Classic serial link. And our firmware sent BLE data in fixed 512 B chunks, which iOS can’t take.

In 2024 I wrote up how MTU negotiation really worked on the platform. The chip vendor’s Bluetooth layer negotiated the MTU, and our code sent fixed 512 B chunks on top of it. I set the chunk payload to 182 B, tested on iOS and Android, and released it as an experimental iOS-compatible firmware. That fixed the iPhone link.

When the vendor said no

The vendor SDK had no MTU negotiation in its GATT server layer. In 2025 I opened a support case, and they declined to add it. So I built the MTU exchange in the GATT client layer myself, for links where the tracker is the client. I tested it in the cold-chain firmware, where the tracker reads temperature recorders as a client, and updated the BLE MTU docs.

Conclusion

In the app, 30,000 records download in 5–6 minutes, and every chunk is CRC-checked. In the firmware, QA verified 182 B chunks on two tracker models.

On the firmware side, the write-up mattered more than the code. Once it was clear who negotiated the MTU and what our code did with it, the iPhone fix was small.

Tags ble apps release

alp ~ $ work trace-decoder

Writing my own trace decoder

Debugging our main cellular platform depended on the chip vendor’s closed trace tool. I wrote my own in C++20 and checked it against the vendor’s output at every step. On 108,755 traces, it produces the same bytes.

108,755traces decoded, byte-identical

Debugging our main cellular platform depended on the chip vendor’s trace tool. It was closed, so when it had problems, nobody outside the vendor could fix them.

A replacement is only worth using if people trust it as much as the original. So the real job wasn’t just building it. It was proving that it reads traces exactly the way the vendor’s tool does.

What it does

I wrote it in C++20, with a Dear ImGui GUI and a command-line version. It handles capture, decoding, merging, pcap export, GSMTAP export for Wireshark, symbols and a Ctrl+K command palette. It idles at 1.9% CPU with a 2-million-row log open.

Checked against the vendor’s tool

Every stage is tested against the vendor tool’s own output, with golden tests covering 3.7 million records. Decoding is compared byte for byte. I also rewrote one of the vendor’s message decoders and checked it the same way: 37,340 of 37,340 messages match. Capture has nothing to compare against, so it got a soak test instead.

Where my trace tool is checked against the vendor's own output Two paths through my tool, each drawn with my stages on top and the vendor reference they are compared to underneath. Trace path: the device’s trace capture feeds my capture stage, then trace decode, then a message decoder I reimplemented, then merge and export to pcap and GSMTAP for Wireshark. Capture has no vendor reference. It had 0 PC-side loss in a 30-minute soak. Trace decode is compared with the vendor trace tool's decode output: 108,755 traces are byte-identical, and 8.7 MB decodes in 0.8 s. The message decoder is checked against the vendor tool’s output: 37,340 of 37,340 random messages match. Crash path: a memory dump taken after a crash goes to a stage that walks the RTOS task blocks and queues, then a stage that unwinds stacks with DWARF CFI, giving symbolized call stacks. These are compared with the vendor's crash report: 57 of 57 call stacks match. Golden tests cover 3.7 million records. TRACE PATH · my stages on top, the vendor reference below device trace capture capture PC side decode traces message decoder reimplemented merge, export pcap · GSMTAP 0 lost 30-min soak 108,755 traces byte-identical 8.7 MB in 0.8 s vendor tool decode output 37,340 / 37,340 messages match random inputs vendor tool output CRASH PATH · the same check, against the vendor's crash report memory dump after a crash walk task blocks and queues unwind stacks DWARF CFI call stacks symbolized 57 / 57 call stacks match vendor crash report Golden tests cover 3.7 million records.
Where my trace tool is checked against the vendor's own output

Crash analysis

The crash analyzer walks the RTOS task blocks and queues, and unwinds the stacks using DWARF CFI. Its call stacks are compared with the vendor’s crash report.

Conclusion

Decoding is byte-identical on 108,755 traces, and 8.7 MB decodes in 0.8 s. 57 of 57 crash call stacks match the vendor’s report. A 30-minute capture soak lost nothing on the PC side.

Matching the vendor’s tool proves that mine agrees with it, not that both are right. For a replacement, agreement is the point.

Tags tooling reverse-engineering testing

alp ~ $ work security

3,505 files compiled, 597 shipped

Our firmware needed a software bill of materials for the EU Cyber Resilience Act, but standard tools couldn’t read the image. Working from the linked ELF, I found that only 597 of 3,505 compiled files ship. I also reworked unlock, updates and TLS.

597/3,505compiled files that really ship

Tracker firmware needed stronger device security, a modern TLS stack and a software bill of materials (SBOM) for the EU Cyber Resilience Act. This page covers the design only. The details stay private.

Which files actually ship?

Standard SBOM tools can’t read this kind of RTOS firmware image, so I worked from the linked ELF instead. Most of what the compiler sees never ends up in the image, and listing all of it would make the SBOM longer and less accurate.

Files in the build

I compared four ways of counting the same build. The compiler saw 3,505 files. The linker’s “unused” list was misleading. File symbols in the ELF over-count by about 2x. Counting only files that own a sized symbol in a loaded section gives 597, and that’s what the SBOM lists. The CI pipeline handles the scanner’s rate limit and uploads to Dependency-Track with CPE identifiers.

Counting only the files that put code in the image Standard SBOM tools cannot read this RTOS firmware image, so the component list is built from the linked ELF of the release. Four tests for which files of the same build ship were compared. Files seen by the compiler: 3,505, but most of them put no code into the image. The linker's "unused" list: misleading. File symbols in the ELF: they over-report by about 2x. A file counts only if it owns a sized symbol in a loaded section: 597 files, and this is the count the SBOM uses. In CI, every release build runs five steps: the linked ELF of the release, the sized-symbol filter that keeps only files with code in the image, an inventory with a validator, a SCANOSS code scan, and an upload to Dependency-Track with CPE identifiers. If SCANOSS hits its rate limit, a manual retry step runs and the release driver is notified. Dependency-Track keeps one collection and version per release. WHICH FILES SHIP · four tests on the same build seen by the compiler 3,505 · most add no code to the image linker “unused” list misleading file symbols in the ELF over-report by about 2x owns a sized symbol in a loaded section 597 · the count the SBOM uses Standard SBOM tools cannot read this RTOS image, so the evidence comes from the linked ELF. IN CI · every release build linked ELF of the release sized-symbol file filter inventory + validator SCANOSS code scan Dependency-Track CPE identifiers On a rate limit, a manual retry step runs and the release driver is notified. One collection and version per release.
Counting only the files that put code in the image

Unlock and passwords

Unlocking is challenge-response, and the device verifies a signature. Passwords are stored only as salted PBKDF2 hashes and compared in constant time. Attempt counters survive a power cut. If the lock’s record gets damaged, the lock stays closed.

Updates and TLS

Firmware updates got downgrade and tamper protection, designed together with five new unit-test suites. I shipped it as release owner.

I moved TLS to the library’s current long-term-support line, with TLS 1.3. MD5, SHA-1, DES, renegotiation and DTLS are off. Only AEAD and ECDHE cipher suites are left.

Conclusion

Fail-first unit tests, with fakes for crypto and storage, caught 12 of 12 deliberate mutations. I also ran bench loops with reboots and power cuts.

For security code, a test suite that has never failed doesn’t prove much.

Tags security tls release

alp ~ $ work watch-harness

71 tests that run on a real watch

71 tests run on a real smartwatch over BLE, and any new crash fails the test, even if the test’s own checks pass. Together with a memory gate in CI, the setup has found 16 bugs.

16 bugs · 71 testsbugs found, and device tests that run on the watch

A test on a real device can pass while the device crashes in the background. On a watch with little RAM, a change can also pass every test and still eat memory that a later feature needs.

So I built the harness to judge the whole watch, not just the feature under test, and to put a memory cost on every change.

The setup

A pytest harness drives a production smartwatch over BLE and a debug probe. It’s 75 files and 8.1k lines.

A crash guard around every test

Before each test, a guard reads the coredumps, the uptime and the BLE link. After the test, it checks for a new coredump, an unexpected disconnect or an uptime reset. If it finds one, the test fails, even if all its own checks passed.

The watch harness: every test wrapped by a crash guard, every change through a memory gate The rig: a pytest harness of 75 files and 8.1k lines, with 71 device tests, drives a production smartwatch, a dual-core BLE MCU running a watch RTOS, over BLE (SMP) and over a debug probe. Every test is wrapped by the crash guard. Before the test, the guard reads the coredumps, the uptime and the BLE link. Then the test body drives the watch and runs its own asserts. After the test, the guard asks three questions: is there a new coredump, was there an unexpected disconnect, did the uptime reset. Outcome table: own asserts pass and guard clean gives pass. Own asserts pass and guard trips gives fail, a case only the guard catches. Own asserts fail gives fail. Every change also goes through a memory gate in CI with three checks: a RAM gate with a byte budget, a static stack bound computed from the call graph, and a stack and heap peak recorder that writes to flash. The RAM gate caught a stack-size reduction that crashed over-the-air updates. Result: 71 device tests pass with 0 new crashes on the current build, and the harness found 16 bugs, each fixed in its own commit. THE RIG · bench and CI pytest harness 71 device tests 75 files · 8.1k lines BLE · SMP debug probe production smartwatch dual-core BLE MCU watch RTOS EVERY TEST · wrapped by the crash guard guard, before reads coredumps, uptime, BLE link test body drives the watch runs its own asserts guard, after new coredump? unexpected disconnect? uptime reset? own asserts crash guard result pass clean PASS pass trips FAIL only the guard catches this fail either FAIL EVERY CHANGE · memory gate in CI change RAM gate byte budget static stack bound from the call graph peak recorder stack and heap, to flash the RAM gate caught a stack-size reduction that crashed over-the-air updates RESULT 71 device tests pass · 0 new crashes · 16 bugs found
The watch harness: every test wrapped by a crash guard, every change through a memory gate

A memory gate in CI

Every change goes through a RAM budget in bytes, a stack limit worked out from the call graph, and a stack and heap peak recorder that writes to flash. The RAM gate caught a stack-size cut that crashed over-the-air updates.

What it found

The harness turned up 16 bugs, each fixed in its own commit. As a side effect, font loading went from 180 ms to 29 ms.

Font load time, a side result

I also built a binary event journal for the watch. It showed that every ECG abort had been reported with the wrong reason, because of an enum cast. I fixed it with an explicit mapping and a static_assert.

Conclusion

All 71 device tests pass with no new crashes on the current build, and the harness has found 16 bugs.

The crash guard is the part that pays off most. It turns every test into a crash test too.

Tags testing ble wearable tooling

alp ~ $ work inventory-app

The inventory app that outgrew 50k reads

My test-hardware inventory app hit Firebase’s 50k read limit. I priced switching to an off-the-shelf tool (64 h) against moving to my own API (40 h), and moved it. It has been used company-wide since June 2026.

487 ms  to 92 mslist sort, 2,640 devices

R&D and QA have a lot of test hardware, and every device is identified by its IMEI. I built and run the inventory for it: Expo / React Native for Android, iOS and web, an Express API, MongoDB, a C++ scanner service, Docker and GitLab CI. So far that’s at least 33 tickets and about 356 commits.

It started as a small tool and grew into a company-wide service.

Outgrowing Firebase

The first version went live on GitLab Pages the same day it was asked for. Then it moved to Firebase. When it hit Firebase’s 50k read limit, it needed a new home.

Before rebuilding, I priced the alternative. Moving to an off-the-shelf open-source asset manager came to 64 hours. Moving my app to its own backend came to 40. So I moved it to MongoDB behind my own Express API.

Scanning etched codes

The app reads QR and DataMatrix codes with zxing-cpp in three places: a Kotlin module on Android, WASM in the browser, and the scanner service on the server. Laser-etched codes are hard to read, so the decoders try several scales and inverted images.

I also added a server-side photo decoder (OpenCV, Tesseract OCR, zxing). It reads every code on a PCB from one photo, and does it better than the live scan.

One C++ decoder in three runtimes, feeding one API that serves people and CI Top: the same zxing-cpp decoder, with multi-scale and inverted passes for hard-to-read laser-etched codes, runs in three places. On Android it is a Kotlin module fed by a live phone-camera scan. In the browser it is WASM fed by a live browser-camera scan. On the server it is a C++ scanner service, with an OpenCV and Tesseract OCR photo decoder that reads every code on a PCB from one photo. All three produce the IMEI, the unique key of every device, which goes to the Express API. Bottom: the Express API rejects duplicate IMEIs, fills in device details automatically, and has an OpenAPI spec with a drift test. People use the Expo app on Android, iOS and web and log in through company single sign-on. The firmware CI pipeline uses a separate machine API with its own credentials to fetch its device list, responses carry ETags for 304s, and webhooks are HMAC-signed. The API stores data in MongoDB with a daily backup and a change tree with revert. MongoDB replaced Firebase after the app hit Firebase's 50k read limit. ONE C++ DECODER · compiled into three runtimes Laser-etched codes are hard to read, so every runtime runs multi-scale and inverted passes. live scan, phone camera live scan, browser camera one photo of a PCB Android Kotlin module zxing-cpp multi-scale + inverted Browser WASM zxing-cpp multi-scale + inverted Server C++ scanner service zxing-cpp multi-scale + inverted + OpenCV, Tesseract OCR every code on a PCB, one photo IMEI, the unique key people Expo app Android, iOS, web SSO company Express API duplicate IMEI rejected details filled in automatically OpenAPI spec + drift test firmware CI fetches its device list machine API ETag, 304 when unchanged HMAC-signed webhooks MongoDB daily backup, change tree Replaced Firebase after its 50k read limit.
One C++ decoder in three runtimes, feeding one API that serves people and CI

Everyday use

Search supports AND/OR, exact (=) and partial (~) matching, suggestions and shareable filter links. Excel import and export puts each location on its own sheet. Device details fill in automatically, duplicate IMEIs are rejected, and devices can be reserved and released. There’s a daily backup, and admin mode has a git-like change history with revert.

An API for CI

The firmware CI pipeline needed the device list, but the app only had interactive single sign-on. So I built a separate machine API for CI, with its own credentials. CI firmware builds now use it to fetch their device list.

Webhooks are signed with HMAC, responses carry ETags for 304s, and a test catches the OpenAPI docs falling behind the routes.

Conclusion

Sorting 2,640 devices went from 487 ms to 92 ms. The app has 352 Jest tests and 1,371 screenshot comparisons. In June 2026 it launched company-wide, with company single sign-on.

List sort, 2,640 devices

Pricing the off-the-shelf option was worth it, even though I didn’t take it. 40 hours against 64 made it a decision, not a preference.

Tags apps tooling testing

alp ~ $ tools

Tools

ToolWhat it doesStack
Bench consoleA web console for the device bench, with plugins for a GPS server, test runner, log parser, crash analysis, release creator, doc search, update-failure reproducer and TLS port router. 1,547 tests.Python · FastAPI · Preact · SQLite
Trace decoderMy own replacement for a closed vendor trace and crash tool. Read it.C++20 · CMake · Dear ImGui
Bench case layerBench cases written as JSON, covering serial, power, PSU, BLE, flashing and MQTT. A setup watchdog took one test suite from 0 of 5 passing to 5 of 5.Python
Device inventory appUsed by the whole company since June 2026. DataMatrix and QR scanning on Android, web and server, search, Excel import/export, and a machine API used by CI firmware builds. Sorting 2,640 devices went from 487 ms to 92 ms.Expo · React Native · Express · MongoDB
Device serverA server for tracker protocols, with file transfer, update sessions, commands and a serial monitor.Python · PySide6 · SQLite
HR comparison appRecords the watch and a reference chest strap side by side, with activity markers.Kotlin
ParsersRecords, AT logs, update logs, signal quality, LTE feature groups, no-fix reasons, NMEA trip maps and MQTT latency.Python
alp ~ $ contact

Contact

Say hi.

email
[email protected]
linkedin
linkedin.com/in/alp-orak
github
github.com/alporak
based in
Vilnius, Lithuania