Alp Sezer Orak
Senior Embedded Software Developer, Vilnius
Firmware for cellular GNSS trackers, BLE sensors and a health smartwatch.
www.alporak.com · Vilnius, Lithuania · [email protected] · linkedin.com/in/alp-orak · github.com/alporak
C C++ Python RTOS LTE GNSS BLE MQTT
Career
- Platform maintainer and release owner for cellular GNSS tracker firmware: about 140 release tickets driven (firmware and app), base updates across about 15 customer branches. alporak.com/work/the-job
how: When it was not the firmware
This is the work around the code at Teltonika since 2022. In that time I’ve made 903 commits, implemented about 567 tickets, investigated about 313 more, and driven about 140 release tickets.
Tickets by my role, 2022–2026 On the platform maintainer rotation, I answer the questions that QA and support bring to the firmware team. Plenty of those end with “the firmware is fine”. That answer only counts if there’s data behind it.
Releases
I own releases on the tracker platform: five base lines, about 15 customer branches, experimental builds and mobile app builds. Each release comes with notes telling QA what to test. I carry base updates and cherry-picks across the customer branches, and I cancel and redo a release when QA results call for it. One base release carried 53 tickets. I also added a CI check that blocks a customer release if its firmware entry is missing.
Support
I’ve answered about 150 support cases with a root cause found in device logs.
In one case an ACK went missing. I rebuilt the test traffic against my own TCP server. The device sent two messages, and the server received them as one 1,388 B block. The server’s parser was the problem, and I suggested splitting messages by their length field.
In another, a reported 1.5–2 minute clock error wouldn’t reproduce on two modem types and many SIMs. The evidence pointed to the operator’s network time.
Networks and vendors
A carrier NAT that forgets devices after 15 quiet seconds became half of another write-up. One escalation covered thousands of devices on one network, and forcing 4G with manual band selection solved it.
When a problem might be inside the modem, I build the case. I captured the modem’s internal logs on the bench and sent them to the vendor. One case took months of back and forth, and the fix reached the customer in a base update. I also turn vendor advice into firmware work, like a work item for faster switching between 2G and 4G.
Estimates
I’ve written at least 25 research documents with effort estimates, from 30 h to 332 h. The dual APN estimate came to 116 h. It showed that only one TLS session fits in RAM, and the customer accepted a design built around that.
Conclusion
About 150 support cases with a root cause, about 140 release tickets, at least 25 estimates, and about 567 tickets implemented.
“Not our bug” is a finding too, and it needs the same proof as a fix: a rebuilt test, a trace, a modem log.
- Redesigned modem recovery and the RTOS task watchdog as one staged recovery, tested on five modem types. alporak.com/work/modem-recovery
how: Hung modems that recover on their own
Trackers in the field lost data whenever their modem stopped working. Nobody is around to power-cycle a tracker in a vehicle, so the firmware has to notice and fix it by itself.
Restarting a modem is the easy part. The hard part is knowing when to do it, how hard to push, and making sure there’s no path where the tracker just gives up.
Escalating step by step
Recovery is driven by how long the tracker has sent no data. At 10 minutes it restarts the modem radio with
CFUN. At 20 it resets the modem through its hardware pin. At 30 it resets the whole device. Cheapest step first. I tested it on five modem types.One watchdog instead of two
I wrote the user story to merge this recovery into the old watchdog and remove the duplicate code: 137 lines added, 183 removed. The watchdog period now grows with the configuration, so a tracker that sends rarely isn’t mistaken for a hung one. Only one recovery can fire per 10 minutes. I documented it with flowcharts and a table for every counter.
What the merge broke
The merge led to about 12 follow-up bugs, and I fixed them. The watchdog counted per server, even though all servers share one modem, so I made it one counter. A
CFUNrestart was clearing the no-GPRS counter. A record period longer than the watchdog period caused pointless modem restarts. And one modem type has no pin reset, so that step only runs where it’s supported.Short drops and deep sleep
A short network drop went unnoticed, so the tracker waited out a long timeout and took 1 min 10 s to react. I added a “suspicious” state: after a short drop, the next communication error starts a full recovery right away.
For LTE-M I built a step-by-step sequence: check registration with
CEREG?, check and reactivate the data context, then restart the radio. I made these work in deep sleep too, where records had been arriving 6–7 minutes late instead of 2.A scan that could strand a tracker
The operator scan ignored errors, so an empty result left the tracker stuck for good. On top of that, a dropped data context banned the operator, started no new search, and reset the recovery counter.
Now the scan result is checked, and retries happen at 30, 60 and 120 s instead of every 5 minutes. After 3 failures the modem network restarts, a dropped context starts a search right away, and a ban no longer resets the counter.
First retry after a failed operator search Conclusion
The client confirmed the LTE-M sequence works. QA could no longer reproduce the deep-sleep delay. The operator-search fix passed a test with a roaming SIM that had data suspended.
Most of the work came after the first version. Recovery code only runs when things go wrong, so its bugs show up late, one field case at a time.
- Cut MQTT record latency p90 from 4.06 s to 0.12 s for a large fleet. alporak.com/work/mqtt-latency
how: Why did locking a car take 10 seconds?
A fleet customer locks and unlocks cars over MQTT. The server sends a command, the tracker does it and replies. Over time, the p90 for a lock crept past 10 seconds. That’s a long time to stand next to a car, waiting for the door.
Measuring first
I wrote a script that pulls the round-trip time of every publish out of the device logs, and added latency probes at each stage. Then I changed one thing at a time, each behind its own flag, and measured again.
One task doing everything
There was no single bug. It was a traffic jam. Commands, record publishing and the protocol engine all ran in one task. That task waited up to 60 s for a socket send, and with no limit at all for an MQTT ACK. One slow write and everything behind it stalled, including a lock command stuck behind a queue of records. On top of that, every record was built twice.
Splitting it up
I split MQTT into three tasks: commands, publishing and the protocol engine. Now a task waiting on the network doesn’t hold up the others.
They still share one TLS session, so writes have to be serialized somewhere. I first tried the shared network layer. It deadlocked. It works in the layer that owns the TLS session.
Both waits now have a cap. The transport answers
ok,busyorfatal, so a busy modem no longer kills the TLS session. The duplicate record build is gone. I also fixed a record-storage mode that an earlier update had broken.Conclusion
Record send p90 went from 4.06 s to 0.12 s, and the median from 1.03 s to 0.05 s. The time one reply held the connection layer went from 495–1,144 ms to 32–240 ms. A calibration command went from 1,664 ms to 4 ms.
Record send time, p90 One caveat: lock replies use the same publish path, so record send time is what I measured. I don’t have the end-to-end lock time after the fix. The delay still visible in the customer’s logs came from sockets dropping while roaming.
Nothing here was slow on its own. The problem was waiting without limits, and it only showed up once each stage was measured.
- Designed signed device unlock, salted password hashes and downgrade protection. Moved TLS to TLS 1.3, and built the EU CRA SBOM pipeline. alporak.com/work/security
how: 3,505 files compiled, 597 shipped
Tracker firmware needed stronger device security, a modern TLS stack and a software bill of materials (SBOM) for the EU Cyber Resilience Act. This page covers the design only. The details stay private.
Which files actually ship?
Standard SBOM tools can’t read this kind of RTOS firmware image, so I worked from the linked ELF instead. Most of what the compiler sees never ends up in the image, and listing all of it would make the SBOM longer and less accurate.
Files in the build I compared four ways of counting the same build. The compiler saw 3,505 files. The linker’s “unused” list was misleading. File symbols in the ELF over-count by about 2x. Counting only files that own a sized symbol in a loaded section gives 597, and that’s what the SBOM lists. The CI pipeline handles the scanner’s rate limit and uploads to Dependency-Track with CPE identifiers.
Unlock and passwords
Unlocking is challenge-response, and the device verifies a signature. Passwords are stored only as salted PBKDF2 hashes and compared in constant time. Attempt counters survive a power cut. If the lock’s record gets damaged, the lock stays closed.
Updates and TLS
Firmware updates got downgrade and tamper protection, designed together with five new unit-test suites. I shipped it as release owner.
I moved TLS to the library’s current long-term-support line, with TLS 1.3. MD5, SHA-1, DES, renegotiation and DTLS are off. Only AEAD and ECDHE cipher suites are left.
Conclusion
Fail-first unit tests, with fakes for crypto and storage, caught 12 of 12 deliberate mutations. I also ran bench loops with reboots and power cuts.
For security code, a test suite that has never failed doesn’t prove much.
- Built dual-APN networking and LTE-M recovery sequences together with modem vendors. alporak.com/work/ram-budget
how: Three features against one RAM budget
Tracker firmware runs on a fixed amount of RAM, and most of it is already taken. Three requests each needed more of it. A client wanted 400 geofences, each a 15-point polygon. Another project needed the driver ID list to hold 10,000 entries. A customer needed two data connections on different APNs, open at the same time.
I wrote an estimate for each before writing any code: 154 h of firmware work plus 40 h for the configuration tool, 60 h, and 116 h. Each one hit a different wall: free RAM, boot time, and room for only one TLS session.
Geofences: measure, then cut
Every polygon point lives in RAM. With 400 zones, I measured 42 kB of free RAM at 10 points per polygon, and 24 kB at 15. That was an early measurement, and it was dangerously low.
Free RAM with 400 zones, preliminary measurement Coordinates come with six decimal places, about 10 cm, and were stored as doubles. I pushed for 32-bit floats instead: half the bytes per coordinate, and a little less precision. I also pushed back on per-zone names as scope creep. It still didn’t fit, so I removed reserved features the build never used, and shortened a rarely used list to free 2.4 kB.
What the cuts broke
Trimming had removed some base parameters by mistake, so the immobilizer re-locked after unlock. I restored them. The
#ifndefblocks I added to save RAM overlapped and changed an IO default, so an alarm event was always on. I patched that. And the configuration tool still treated coordinates as doubles, so every config showed unsaved changes. I updated it to float.10,000 driver IDs: boot was the problem
A driver touches an ID key to the 1-Wire reader, and the firmware checks it against the list for the immobilizer. With 10,000 entries the feature worked, but boot took about 100 s, against a 30 s limit. I made the list load faster at boot, and cut RAM and BSS use so the list fit.
Two connections, one TLS session
The firmware assumed there was only ever one data connection. The estimate is where the real limit showed up: there was RAM for one TLS session, not two. So I reworked the core network API to handle two PDP contexts. Both connections stay up, and secure traffic goes to one server at a time, so one TLS session is enough.
In testing, the backup connection was torn down and rebuilt on every send. The network rejected that, and the modem got stuck in a restart loop. I fixed the backup handling in several places and removed the settings-refresh loop.
Conclusion
Geofences: QA confirmed “400 zones configurable and polygons working”, and the build fits in RAM. Driver IDs: boot went from about 100 s to under 30 s, on two tracker models. Two connections: the customer accepted the design, which touches about 30 files across the network API and two modem drivers.
On a device this full, every feature costs something else. Here it was precision, unused features and a second TLS session. And cutting RAM breaks things too, so cuts need testing like any new feature.
- Traced 700 trackers sending zero coordinates to a fix flag that moved before NMEA was stable. alporak.com/work/zero-coords
how: 700 trackers on Null Island
In 2024, after a firmware update, 700 trackers in the field started sending records with zero coordinates. The records said they had a GPS fix.
0, 0 is a spot in the Gulf of Guinea, known as Null Island. A record without a fix is just a gap. A record with a “fix” at 0, 0 is worse, because the server puts it on the map.
The flag moved too early
I reproduced it. The GPS fix signal switched to “fix” before NMEA stabilization had finished. Any record created in that window had a fix but no real position.
So I made the fix signal wait until NMEA is stable. In factory test mode the wait is skipped, so production tests stay fast.
A flag that never reset
While wiring that in, I found that the unstable-fix flag was never reset on sleep or on GPS loss, so it carried over into the next fix. I reworked it. The unstable-fix flag now only starts NMEA stabilization, and a separate “NMEA stable” flag feeds the fix filter, the same way the speed and altitude filters do.
What the wait exposed
The new delay uncovered an ordering bug. “Update good coordinates” ran before “update GPS fix state”, so coordinates changed while the status still said no fix, and went stale after ultra-deep sleep. I swapped the order, so coordinates only update on a stable fix, and the fix status now resets on sleep.
I also added GNSS status 4, “filter active”, so testers can see why coordinates are being held.
Conclusion
Test drives on 3 device families produced no zero-coordinate records. All three fixes touch the same state, so I put them in one test firmware, and QA verified them together.
Adding a delay changes the order things happen in, and that brings hidden ordering bugs to the surface. The first fix was small. Most of the work was in what it uncovered.
- Found a GNSS race that left the receiver off for 5.5 h. GNSS commands went from 45 s to 3 s. alporak.com/work/gnss-honesty
how: GNSS off for 5.5 hours, and a cold start that lied
A tracker drove out of an underground car park and didn’t get a fix for 5.5 hours. The logs showed the GNSS module had never been powered on, even though the firmware said it was.
Separately, the firmware’s fix-time test was reporting cold starts of 1–3 s. That’s far too good to be true.
Two different bugs, same problem: the firmware was reporting something about the GNSS that wasn’t real.
Reproducing the race
I suspected a race in the restart sequence. I added a test hook that floods the GNSS request path, and that reproduced it on the bench: the firmware says GNSS is on, the module stays off.
How a request got lost
A restart sends “off”, then “on” 2 s later. Requests were de-duplicated against the module’s current state. When “on” arrived, “off” hadn’t been applied yet, so the module still looked on, and “on” was dropped as a duplicate. Then “off” completed, and nothing turned it back on.
The fix
I added a pending-request counter and a FIFO, so requests can’t be lost or reordered while an earlier one is still being applied. As a backup, a check runs every second and resends the request if the wanted and actual states differ for 30 s.
Switching GNSS on or off by command went from 45 s to 3 s, with no reboot. In QA, two trackers got their fix back quickly in driving and long-term tests.
Time to switch GNSS on or off by command The test that passed too fast
With tick logging, the test “passed” after about 215 ticks. The raw NMEA showed why. A valid
$GNGGAline from before the reset arrives after the reset command but before the module’s start message, and the firmware took it as a fresh fix.I added a 300-tick guard window, about 1.3 s, that ignores GNSS status right after a reset. A fix-holdout feature was also keeping the old status alive, so I cleared it in the two places where the test forces a fix loss. I also tried clearing the assistance data. That made no difference.
Conclusion
GNSS now switches in 3 s instead of 45, and requests no longer get dropped. The test reports real numbers: cold start 30–65 s, warm about 3–5 s with assistance data, hot 3–4 s.
A result that looks too good deserves the same suspicion as one that looks bad. The cold-start number was off by more than ten times, and nothing about it looked broken.
- Bisected offline record loss on the device to a merge commit, and fixed an older delete edge case. Delivery went from 1 of 11 packets to 11 of 11. alporak.com/work/offline-loss
how: 1 packet out of 11 survived the outage
In RS232/RS485 buffered mode, the tracker stores data from a serial sensor and sends it when the server is reachable again. But after an outage, only the last packet was left.
Finding the change
Buffered mode had worked before, so something had broken it. The record database stores records by sequence number in a nested folder tree. I bisected on the device across releases and merges, and landed on a merge from an unrelated feature: a rewrite of the FIFO record database.
The flag nobody read
The multi-save call still took a sequence flag, but the rewritten code never read it. So every packet was written as sequence 0. The database then decided the file was corrupted and erased it, leaving only the newest packet.
Two more problems on the same path
Buffered mode always used the TCP path, because only the non-buffered path checked the server type. So a device set up for MQTT never sent its buffered data at all. And in the JSON codec for the cloud, serial data over 18 B was dropped and 0x0A/0x0D bytes were changed. There was also an old edge case in delete-all.
The fixes: each buffered packet gets its own storage slot, which also fixed delete-all. Buffered mode now uses the same server-type check as the rest, so it works over TCP and MQTT, ASCII and binary. The JSON codec sends the full serial data as a hex string.
Conclusion
After an offline period, 11 of 11 packets now arrive, instead of 1 of 11.
Serial packets delivered after an offline period Nothing caught the original break because the record database had no unit tests. I wrote the first ones, with an in-memory fake file system, a 269-line fake record-manager harness and a 5-step test plan.
The symptom was in one feature and the cause was in another. Bisecting is what connected them.
- Fitted 400 polygon geofences and a 10,000-entry driver ID list into trackers short of RAM. With the full list, boot went from about 100 s to under 30 s.
- Co-developed a React Native BLE sensor app with 14 store releases. alporak.com/work/cold-chain
how: 30,000 records over BLE in 5–6 minutes
BLE sends data in small packets. Both sides agree on a maximum size, the MTU, and anything bigger doesn’t get through.
I ran into this limit from two sides: a phone app that downloads logs from a cold-chain temperature recorder, and tracker firmware that iPhones couldn’t talk to.
The app: every record has to arrive
I co-developed the React Native app for a family of BLE sensors, on iOS and Android: about 140 commits, about 85 tickets and 14 releases to Google Play and TestFlight.
For the recorder’s cold-chain certification, every record has to arrive, and you have to be able to show it did. I added a CRC check on every chunk, so each row in the export says whether its chunk arrived intact. I also fixed a page-header shift, where the header sometimes arrived one slot late and the app read a record chunk as the header. Then I raised the MTU to 247 and the chunk size from 15 to 60 records.
To test full-size downloads, I added a fake-record generator to the app. 30,000 records now download in 5–6 minutes.
The firmware: iPhones and 512-byte chunks
An iPhone can only reach the tracker over BLE. The tracker isn’t MFi certified, so iOS can’t use its Bluetooth Classic serial link. And our firmware sent BLE data in fixed 512 B chunks, which iOS can’t take.
In 2024 I wrote up how MTU negotiation really worked on the platform. The chip vendor’s Bluetooth layer negotiated the MTU, and our code sent fixed 512 B chunks on top of it. I set the chunk payload to 182 B, tested on iOS and Android, and released it as an experimental iOS-compatible firmware. That fixed the iPhone link.
When the vendor said no
The vendor SDK had no MTU negotiation in its GATT server layer. In 2025 I opened a support case, and they declined to add it. So I built the MTU exchange in the GATT client layer myself, for links where the tracker is the client. I tested it in the cold-chain firmware, where the tracker reads temperature recorders as a client, and updated the BLE MTU docs.
Conclusion
In the app, 30,000 records download in 5–6 minutes, and every chunk is CRC-checked. In the firmware, QA verified 182 B chunks on two tracker models.
On the firmware side, the write-up mattered more than the code. Once it was clear who negotiated the MTU and what our code did with it, the iPhone fix was small.
- Built and own the device inventory app for R&D and QA test hardware, used company-wide since June 2026. alporak.com/work/inventory-app
how: The inventory app that outgrew 50k reads
R&D and QA have a lot of test hardware, and every device is identified by its IMEI. I built and run the inventory for it: Expo / React Native for Android, iOS and web, an Express API, MongoDB, a C++ scanner service, Docker and GitLab CI. So far that’s at least 33 tickets and about 356 commits.
It started as a small tool and grew into a company-wide service.
Outgrowing Firebase
The first version went live on GitLab Pages the same day it was asked for. Then it moved to Firebase. When it hit Firebase’s 50k read limit, it needed a new home.
Before rebuilding, I priced the alternative. Moving to an off-the-shelf open-source asset manager came to 64 hours. Moving my app to its own backend came to 40. So I moved it to MongoDB behind my own Express API.
Scanning etched codes
The app reads QR and DataMatrix codes with zxing-cpp in three places: a Kotlin module on Android, WASM in the browser, and the scanner service on the server. Laser-etched codes are hard to read, so the decoders try several scales and inverted images.
I also added a server-side photo decoder (OpenCV, Tesseract OCR, zxing). It reads every code on a PCB from one photo, and does it better than the live scan.
Everyday use
Search supports AND/OR, exact (
=) and partial (~) matching, suggestions and shareable filter links. Excel import and export puts each location on its own sheet. Device details fill in automatically, duplicate IMEIs are rejected, and devices can be reserved and released. There’s a daily backup, and admin mode has a git-like change history with revert.An API for CI
The firmware CI pipeline needed the device list, but the app only had interactive single sign-on. So I built a separate machine API for CI, with its own credentials. CI firmware builds now use it to fetch their device list.
Webhooks are signed with HMAC, responses carry ETags for 304s, and a test catches the OpenAPI docs falling behind the routes.
Conclusion
Sorting 2,640 devices went from 487 ms to 92 ms. The app has 352 Jest tests and 1,371 screenshot comparisons. In June 2026 it launched company-wide, with company single sign-on.
List sort, 2,640 devices Pricing the off-the-shelf option was worth it, even though I didn’t take it. 40 hours against 64 made it a decision, not a preference.
- Smartwatch (C++): motion-tolerant heart rate, an event-log platform and an on-device test harness.
- Firmware for a heavy-vehicle tracker line, and tracker firmware that reads EN 12830 cold-chain recorders for their certification.
- About 567 tickets implemented, about 313 investigated and at least 25 research estimates. Vendor cases with modem and chip makers. Regularly stood in as team lead.
- Ultra-low-power temperature monitor on ESP8266 that runs a year on one battery, using tuned deep sleep and I2C sensors.
- NB-IoT vibration analysis on LPC1517 with on-device FFT, for predictive maintenance.
- Modular low-level drivers and sensor integration for LPC1769, and a move of the legacy communication stack to Mbed OS with multithreaded Ethernet and a cloud link.
- Took part in CE certification, and designed the enclosure and a breakout board for its safety and EMI requirements.
- Firmware updates over the air and over USB. alporak.com/work/dvm
how: Three years of building whole devices
From 2019 to 2022 I was an electronics engineer at DvM IoT Solutions. We built IoT devices, and I worked on every layer: drivers, firmware, updates and the enclosure.
When you build the whole device, everything is connected. Battery life depends on what the firmware does between readings. The enclosure decides whether the product passes CE.
A freezer monitor that lasts a year
I designed an ultra-low-power temperature monitor for freezers, on a Wi-Fi MCU. It reads its sensors over I2C and spends the rest of the time in deep sleep. I tuned that sleep until one battery lasted a year.
A vibration monitor that does its own FFT
For predictive maintenance, I built a vibration and temperature monitor on a Cortex-M MCU with NB-IoT. It runs an FFT on its own vibration data, checks the result against thresholds, and sends an alert over NB-IoT when one is crossed.
Drivers and an RTOS
On another Cortex-M MCU, I wrote modular low-level drivers and integrated analog and digital sensors for industrial automation. I moved the old communication stack onto an embedded RTOS, which made the Ethernet communication multithreaded and connected it to the cloud.
Updates and CE
The devices can take new firmware two ways: over the air, or over USB with the device in USB client mode.
I took part in CE certification, and designed the enclosure and a breakout board to meet the safety and EMI requirements.
Conclusion
A freezer monitor that runs a year on one battery, a vibration monitor that analyzes its own data, an old stack moved onto an RTOS, and an enclosure designed for CE.
Those years taught me to start from the hardware. Power, buses and radio set the limits the code has to live within.
- Thesis, in a team of four: a ground station that detects drones in no-fly zones with YOLOv3, and a drone that catches them. It won 1st place at the university project fair. alporak.com/work/thesis
how: A drone that hunts drones
Cheap drones are a risk near sensitive places like government buildings and crowds. For our B.Sc. graduation project, as a team of four, we set out to take control of a drone that flies into a restricted area.
It had to be fast and autonomous, and leave the drone undamaged. That broke down into four steps: detect it, confirm it, share its position, and catch it.
Detecting and confirming
The ground station picks up the radio signals the drone sends. A camera running YOLOv3, an object detector that looks at a whole image in one pass, then confirms that it’s a drone.
Sharing and catching
A PyQt program shares the results between the ground computers over UDP broadcast. The ground control station sends the drone’s GPS position to our own predator drone, which we designed and built. A guidance algorithm steers it to track and catch the target. I wrote the system in C and Python.
Conclusion
The project won 1st place at the university project fair. The full report is on ResearchGate.
I finished my B.Sc. in Electrical and Electronics Engineering at Eskişehir Technical University in 2019, after spending 2017–18 on Erasmus+ at VSB Technical University of Ostrava. The same year, I started as an electronics engineer at DvM.
- Erasmus+ exchange at VSB Technical University of Ostrava, 2017–18.
Daily
- C
- C++17/20
- Python
- RTOS: Nucleus, Zephyr, Keil RTX
- Cellular modems, AT, PDP, FOTA
- GNSS, NMEA
- BLE
- MQTT, TLS
- J-Link, GitLab CI
Regular
- CAN, OBD-II, tachograph
- RS232/485, Modbus, 1-Wire
- I2C, SPI, UART, ADC
- mbedTLS, TLS 1.3
- CppUTest, Ceedling, pytest
- Ghidra, Wireshark, sniffers
- Docker
Used on projects
- TypeScript, React Native
- Kotlin, C#/.NET
- FastAPI, Node.js
- Mbed OS, FreeRTOS
- Altium, KiCad
- Fusion 360
more: all 24 write-ups, tools I built
Selected work
- 21 to 0 One reset, two bugs, one of them in a closed library alporak.com/work/bt-leak
- 5.5 h GNSS off for 5.5 hours, and a cold start that lied alporak.com/work/gnss-honesty
- 4.06 s to 0.12 s Why did locking a car take 10 seconds? alporak.com/work/mqtt-latency
- 7.8 h to 100 s The 100-second timeout that became 7.8 hours alporak.com/work/one-constant
- 8.2 to 5.5 bpm Heart rate that fell apart when you walked alporak.com/work/heart-rate
- 108,755 Writing my own trace decoder alporak.com/work/trace-decoder
All 24 write-ups: alporak.com/#work
Work
Twenty-four write-ups from real hardware and real releases.Each one is written up in full, with the method and the proof, at www.alporak.com.
Work log
Crashes & resets
Wrong or lost field data
Speed & memory
Bluetooth & wearables
Tools & security
When it was not the firmware
A lot of firmware work happens around the code. I’ve traced about 150 support cases to a root cause, often showing with data that the firmware wasn’t at fault, and driven about 140 release tickets.
This is the work around the code at Teltonika since 2022. In that time I’ve made 903 commits, implemented about 567 tickets, investigated about 313 more, and driven about 140 release tickets.
On the platform maintainer rotation, I answer the questions that QA and support bring to the firmware team. Plenty of those end with “the firmware is fine”. That answer only counts if there’s data behind it.
Releases
I own releases on the tracker platform: five base lines, about 15 customer branches, experimental builds and mobile app builds. Each release comes with notes telling QA what to test. I carry base updates and cherry-picks across the customer branches, and I cancel and redo a release when QA results call for it. One base release carried 53 tickets. I also added a CI check that blocks a customer release if its firmware entry is missing.
Support
I’ve answered about 150 support cases with a root cause found in device logs.
In one case an ACK went missing. I rebuilt the test traffic against my own TCP server. The device sent two messages, and the server received them as one 1,388 B block. The server’s parser was the problem, and I suggested splitting messages by their length field.
In another, a reported 1.5–2 minute clock error wouldn’t reproduce on two modem types and many SIMs. The evidence pointed to the operator’s network time.
Networks and vendors
A carrier NAT that forgets devices after 15 quiet seconds became half of another write-up. One escalation covered thousands of devices on one network, and forcing 4G with manual band selection solved it.
When a problem might be inside the modem, I build the case. I captured the modem’s internal logs on the bench and sent them to the vendor. One case took months of back and forth, and the fix reached the customer in a base update. I also turn vendor advice into firmware work, like a work item for faster switching between 2G and 4G.
Estimates
I’ve written at least 25 research documents with effort estimates, from 30 h to 332 h. The dual APN estimate came to 116 h. It showed that only one TLS session fits in RAM, and the customer accepted a design built around that.
Conclusion
About 150 support cases with a root cause, about 140 release tickets, at least 25 estimates, and about 567 tickets implemented.
“Not our bug” is a finding too, and it needs the same proof as a fix: a rebuilt test, a trace, a modem log.
Three years of building whole devices
For three years I built complete IoT devices: drivers, firmware, updates and the enclosure for CE. One Wi-Fi freezer monitor runs for a year on its battery.
From 2019 to 2022 I was an electronics engineer at DvM IoT Solutions. We built IoT devices, and I worked on every layer: drivers, firmware, updates and the enclosure.
When you build the whole device, everything is connected. Battery life depends on what the firmware does between readings. The enclosure decides whether the product passes CE.
A freezer monitor that lasts a year
I designed an ultra-low-power temperature monitor for freezers, on a Wi-Fi MCU. It reads its sensors over I2C and spends the rest of the time in deep sleep. I tuned that sleep until one battery lasted a year.
A vibration monitor that does its own FFT
For predictive maintenance, I built a vibration and temperature monitor on a Cortex-M MCU with NB-IoT. It runs an FFT on its own vibration data, checks the result against thresholds, and sends an alert over NB-IoT when one is crossed.
Drivers and an RTOS
On another Cortex-M MCU, I wrote modular low-level drivers and integrated analog and digital sensors for industrial automation. I moved the old communication stack onto an embedded RTOS, which made the Ethernet communication multithreaded and connected it to the cloud.
Updates and CE
The devices can take new firmware two ways: over the air, or over USB with the device in USB client mode.
I took part in CE certification, and designed the enclosure and a breakout board to meet the safety and EMI requirements.
Conclusion
A freezer monitor that runs a year on one battery, a vibration monitor that analyzes its own data, an old stack moved onto an RTOS, and an enclosure designed for CE.
Those years taught me to start from the hardware. Power, buses and radio set the limits the code has to live within.
A drone that hunts drones
Team of four. A ground station spots a drone in a no-fly zone, YOLOv3 confirms it, and we guide our own drone to catch it. 1st place at the university project fair.
Cheap drones are a risk near sensitive places like government buildings and crowds. For our B.Sc. graduation project, as a team of four, we set out to take control of a drone that flies into a restricted area.
It had to be fast and autonomous, and leave the drone undamaged. That broke down into four steps: detect it, confirm it, share its position, and catch it.
Detecting and confirming
The ground station picks up the radio signals the drone sends. A camera running YOLOv3, an object detector that looks at a whole image in one pass, then confirms that it’s a drone.
Sharing and catching
A PyQt program shares the results between the ground computers over UDP broadcast. The ground control station sends the drone’s GPS position to our own predator drone, which we designed and built. A guidance algorithm steers it to track and catch the target. I wrote the system in C and Python.
Conclusion
The project won 1st place at the university project fair. The full report is on ResearchGate.
I finished my B.Sc. in Electrical and Electronics Engineering at Eskişehir Technical University in 2019, after spending 2017–18 on Erasmus+ at VSB Technical University of Ostrava. The same year, I started as an electronics engineer at DvM.
Krepza: tracking Lithuanian grocery prices
A self-hosted side project. A scraper, a search service and a web app, all on one database.
krepza.lt compares grocery prices across Lithuanian shops. You make a shopping list and see what each item, and the whole list, costs in each shop.
What it does
Besides the shopping list, it has a deals browser, price history charts, a nearest-shop finder, and Lithuanian and English versions.
How it’s built
Three services run in Docker Compose and share one SQLite database. A scraper, with one adapter per shop, refreshes prices on a schedule. A Python search service uses multilingual embeddings for product search, and an LLM to clean up brands and categories. A Next.js app serves the site and the API.
Conclusion
It runs on my home server, and tagging a version deploys it. Three services, one database and one deploy step keep it small enough to run on my own.
One reset, two bugs, one of them in a closed library
Trackers sometimes reset during firmware downgrades and left no crash record. I fixed a watchdog bug, and a weekend test went from 21 resets to 0. QA still saw failures, though. The rest was hiding in a closed vendor library.
Some trackers reset in the middle of a firmware downgrade. It happened on slow 2G links, under load, and not every time. Worse, no crash record survived, so there was nothing to start from.
It turned out to be two bugs. One was ours. The other was inside a Bluetooth library we only had as a binary.
Bug one: the watchdog
A new watchdog re-ping was too strict in three ways. It treated a late answer to a retransmitted ping as a protocol violation. It could post a ping into a full queue, which is fatal on this RTOS. And it reset a task on its first timeout, even when the task was only starved of CPU.
I changed it to accept a late answer as proof of life, skip the ping when the queue is full, give a starved task one more window, and log which task failed. Then I ran the old build and mine side by side over a weekend. The old one reset 21 times: 4 misread late answers and 17 MQTT watchdog timeouts. Mine reset 0 times.
Bug two: 11 empty scans
That looked like the end of it. It wasn’t. QA still saw failures in about 20% of runs.
Every one of those reports had the same thing right before the reset: 11 Bluetooth inquiries in a row that found nothing.
[t+ 0.0 s] bt: inquiry start
[t+ 10.0 s] bt: inquiry timeout, 0 results (1)
[t+ 10.1 s] bt: inquiry start
...
[t+110.2 s] bt: inquiry timeout, 0 results (11)
[t+110.4 s] sys: abnormal reset, crash record lost
That’s too regular to be luck. The odds are about 1 in 1010. So the Bluetooth code was involved, and that code was a closed vendor library.
Inside the closed library
I disassembled it. Inside is a pool of 32 command tokens. A Bluetooth command takes a token and gives it back when it completes. In crowded BLE areas, the vendor’s controller never finishes a classic inquiry, so every scan change queues a command whose token never comes back.
After a few minutes the pool is empty. The library then crashes and takes the crash record down with it. That’s why we never had one.
The fix
Scans now wait until the pool has room and no command is pending. If the radio hangs anyway, a watchdog restarts it and brings back classic scanning, sensor scanning and advertising. Crash breadcrumbs now live in retained RAM, so the next crash leaves something behind. The watchdog code has unit tests.
Conclusion
Each bug got its own proof. For the watchdog, the weekend A/B test went from 21 resets to 0. For the token pool, a 15 h overnight soak had 0 abnormal resets, and 174 of 174 vendor panics were recovered. The vendor got an evidence package and a service ticket, since the real fix belongs in their controller.
A clean test only proves the bug it can trigger. The 20% that didn’t go away was the lead to the second one.
A weekly crash and one memory dump
A crash showed up about once a week on our stability stand, and never on a desk. One memory dump showed two tasks freeing the same MQTT buffers. I made the race happen on demand, then fixed it.
In 2026, devices on our stability stand crashed about once a week. They were sending records over MQTT with TLS to a cloud broker. The RTOS reported a failed buffer release from the record-send task and reset the device.
Once a week is awkward. You can’t reproduce it on a desk, and you can’t prove a fix by waiting a week and hoping.
Working from one dump
With no way to reproduce it, I had one memory dump to work from. I traced the failed release back to link teardown. Two tasks, the MQTT close path and the record-send close path, had both torn down the same link at the same moment.
The double free
Both paths released the link’s connection buffers. The first release returned them to the pool. The second asked the RTOS to free buffers that were already free. The RTOS treats that as fatal and resets. The two closes rarely line up, so it happened about once a week.
Making it happen on demand
I added a test command that forces both closes to run at the same time. Now I could trigger the race whenever I wanted.
Then the fix: link teardown is now guarded, so the buffers are released exactly once, whichever task gets there first. The second one sees they’re gone and skips.
Conclusion
The fixed firmware ran a full week on the stand with no abnormal restart. To be fair, one clean week is weak proof for a bug that showed up once a week.
What made this solvable was the test command. A rare race doesn’t need a longer soak. It needs a way to force it.
Hung modems that recover on their own
When a modem hung, trackers lost data, and a failed operator scan could leave them stuck for good. I built recovery in steps at 10, 20 and 30 minutes without data, tested it on five modem types, then fixed what broke along the way.
Trackers in the field lost data whenever their modem stopped working. Nobody is around to power-cycle a tracker in a vehicle, so the firmware has to notice and fix it by itself.
Restarting a modem is the easy part. The hard part is knowing when to do it, how hard to push, and making sure there’s no path where the tracker just gives up.
Escalating step by step
Recovery is driven by how long the tracker has sent no data. At 10 minutes it restarts the modem radio with CFUN. At 20 it resets the modem through its hardware pin. At 30 it resets the whole device. Cheapest step first. I tested it on five modem types.
One watchdog instead of two
I wrote the user story to merge this recovery into the old watchdog and remove the duplicate code: 137 lines added, 183 removed. The watchdog period now grows with the configuration, so a tracker that sends rarely isn’t mistaken for a hung one. Only one recovery can fire per 10 minutes. I documented it with flowcharts and a table for every counter.
What the merge broke
The merge led to about 12 follow-up bugs, and I fixed them. The watchdog counted per server, even though all servers share one modem, so I made it one counter. A CFUN restart was clearing the no-GPRS counter. A record period longer than the watchdog period caused pointless modem restarts. And one modem type has no pin reset, so that step only runs where it’s supported.
Short drops and deep sleep
A short network drop went unnoticed, so the tracker waited out a long timeout and took 1 min 10 s to react. I added a “suspicious” state: after a short drop, the next communication error starts a full recovery right away.
For LTE-M I built a step-by-step sequence: check registration with CEREG?, check and reactivate the data context, then restart the radio. I made these work in deep sleep too, where records had been arriving 6–7 minutes late instead of 2.
A scan that could strand a tracker
The operator scan ignored errors, so an empty result left the tracker stuck for good. On top of that, a dropped data context banned the operator, started no new search, and reset the recovery counter.
Now the scan result is checked, and retries happen at 30, 60 and 120 s instead of every 5 minutes. After 3 failures the modem network restarts, a dropped context starts a search right away, and a ban no longer resets the counter.
Conclusion
The client confirmed the LTE-M sequence works. QA could no longer reproduce the deep-sleep delay. The operator-search fix passed a test with a roaming SIM that had data suspended.
Most of the work came after the first version. Recovery code only runs when things go wrong, so its bugs show up late, one field case at a time.
TLS ate the RTOS buffer pool
A tracker rebooted every 20 s during the TLS handshake, but not on every boot. Every boot that crashed was running a vehicle-bus scan. TLS and the scan were drawing from the same RTOS buffer pool, so I gave TLS its own.
A tracker connecting to a cloud MQTT broker kept rebooting every 20 s, in the middle of the TLS handshake. But not every boot crashed.
That was the useful part. If some boots survive, the difference between them is the clue.
What the crashing boots had in common
I lined up each crash against everything else the device was doing. Boots 1 to 7 all crashed, and all of them were running a vehicle-bus scan. The one that survived had the ignition off, so no scan ran. And at every crash, all 97 of the larger buffers in the RTOS message-buffer pool were in use.
One pool, two users
The TLS library took its small memory blocks from the same RTOS pool that tasks use to message each other. One handshake took 42 of them. The vehicle-bus scan was using the pool too. Together they emptied it, and the RTOS treats that as fatal.
The fix
TLS now has its own private fixed-block memory, so it can’t drain the RTOS pool. Before connecting, the device also counts the free buffers. If there aren’t enough, it reports a resource error and tries again later instead of crashing. QA also got a way to create low-memory conditions on the bench.
Conclusion
The reboot loop stopped, and the customer confirmed that MQTT and vehicle-bus scanning now work together.
The crash happened in TLS, but TLS wasn’t really the problem. Sharing the pool was. When two things share a pool, the one that asks last is the one that crashes.
One SMS, 100 replies, and a reboot loop
One SMS command got over 100 replies. One modem event put trackers in a reboot loop. Both times the modem said something the firmware didn’t expect. I capped the resends, taught the parser the new format, and five blocked tickets were cleared.
The firmware talks to the cellular modem through AT commands. It sends a command, and the modem answers or reports events on its own. Every handler in the firmware assumes something about what the modem will say.
Here are two times, a year apart, when the modem said something else.
100 replies to one SMS
In 2022 a user sent one SMS command to a tracker and got more than 100 replies.
Two things together caused it. The firmware resends an SMS reply if sending fails. And this modem returned an error for SMS sends even when the message had gone out. So each reply reached the user, the firmware thought it had failed, and sent it again.
The firmware can’t tell a fake error from a real one, so there’s no clean fix on our side. What I added is a safety net: once 10 replies are queued, the firmware restarts the modem instead of resending forever.
A reboot loop from three words
In 2023, trackers with one of our external modems went into an endless restart loop, with the default configuration. Five other tickets were blocked behind this one.
The modem reports packet data events with +CGEV. The handler for it sits in about 3,000 lines of URC handling in C. This modem also sent three-word events like +CGEV: ME PDN ACT. The handler had no case for that, parsed it wrong, and the device fell into a restart loop.
I added the three-word cases, ME PDN and NW PDN, so they’re parsed in their own format. The fix was merged to the release branch the same week.
Back to the same event, later
I came back to +CGEV more than once. In 2024 I used +CGEV: ME DETACH in traces to explain why a link closed right after a server command, and showed the flow was the same on every modem. In 2025 I changed how another modem family handles a network detach: the firmware now asks the modem for the context’s current IP, and keeps the connection open if the modem has already recovered.
Conclusion
The restarts stopped, and all five blocked tickets were cleared. The SMS cap stops the loop, but it doesn’t fix the modem’s false error.
Both bugs come from the same habit: trusting whatever the modem says. A modem handler has to cope with answers that are wrong, and with answers it has never seen.
GNSS off for 5.5 hours, and a cold start that lied
One lost “on” request kept GNSS off for 5.5 hours while the firmware said it was on. Separately, a test claimed 1–3 s cold starts because it read a stale NMEA line. I fixed both. Real cold starts are 30–65 s.
A tracker drove out of an underground car park and didn’t get a fix for 5.5 hours. The logs showed the GNSS module had never been powered on, even though the firmware said it was.
Separately, the firmware’s fix-time test was reporting cold starts of 1–3 s. That’s far too good to be true.
Two different bugs, same problem: the firmware was reporting something about the GNSS that wasn’t real.
Reproducing the race
I suspected a race in the restart sequence. I added a test hook that floods the GNSS request path, and that reproduced it on the bench: the firmware says GNSS is on, the module stays off.
How a request got lost
A restart sends “off”, then “on” 2 s later. Requests were de-duplicated against the module’s current state. When “on” arrived, “off” hadn’t been applied yet, so the module still looked on, and “on” was dropped as a duplicate. Then “off” completed, and nothing turned it back on.
The fix
I added a pending-request counter and a FIFO, so requests can’t be lost or reordered while an earlier one is still being applied. As a backup, a check runs every second and resends the request if the wanted and actual states differ for 30 s.
Switching GNSS on or off by command went from 45 s to 3 s, with no reboot. In QA, two trackers got their fix back quickly in driving and long-term tests.
The test that passed too fast
With tick logging, the test “passed” after about 215 ticks. The raw NMEA showed why. A valid $GNGGA line from before the reset arrives after the reset command but before the module’s start message, and the firmware took it as a fresh fix.
I added a 300-tick guard window, about 1.3 s, that ignores GNSS status right after a reset. A fix-holdout feature was also keeping the old status alive, so I cleared it in the two places where the test forces a fix loss. I also tried clearing the assistance data. That made no difference.
Conclusion
GNSS now switches in 3 s instead of 45, and requests no longer get dropped. The test reports real numbers: cold start 30–65 s, warm about 3–5 s with assistance data, hot 3–4 s.
A result that looks too good deserves the same suspicion as one that looks bad. The cold-start number was off by more than ten times, and nothing about it looked broken.
700 trackers on Null Island
After a firmware update, 700 trackers sent positions at 0, 0 while claiming a GPS fix. The fix flag switched on before the NMEA data was stable. I made it wait, then fixed two older bugs that the wait exposed.
In 2024, after a firmware update, 700 trackers in the field started sending records with zero coordinates. The records said they had a GPS fix.
0, 0 is a spot in the Gulf of Guinea, known as Null Island. A record without a fix is just a gap. A record with a “fix” at 0, 0 is worse, because the server puts it on the map.
The flag moved too early
I reproduced it. The GPS fix signal switched to “fix” before NMEA stabilization had finished. Any record created in that window had a fix but no real position.
So I made the fix signal wait until NMEA is stable. In factory test mode the wait is skipped, so production tests stay fast.
A flag that never reset
While wiring that in, I found that the unstable-fix flag was never reset on sleep or on GPS loss, so it carried over into the next fix. I reworked it. The unstable-fix flag now only starts NMEA stabilization, and a separate “NMEA stable” flag feeds the fix filter, the same way the speed and altitude filters do.
What the wait exposed
The new delay uncovered an ordering bug. “Update good coordinates” ran before “update GPS fix state”, so coordinates changed while the status still said no fix, and went stale after ultra-deep sleep. I swapped the order, so coordinates only update on a stable fix, and the fix status now resets on sleep.
I also added GNSS status 4, “filter active”, so testers can see why coordinates are being held.
Conclusion
Test drives on 3 device families produced no zero-coordinate records. All three fixes touch the same state, so I put them in one test firmware, and QA verified them together.
Adding a delay changes the order things happen in, and that brings hidden ordering bugs to the surface. The first fix was small. Most of the work was in what it uncovered.
1 packet out of 11 survived the outage
After an outage, only 1 of 11 buffered serial packets reached the server, because every packet was saved as sequence 0. I bisected it to a merge from an unrelated feature, fixed it, and wrote the record database’s first unit tests.
In RS232/RS485 buffered mode, the tracker stores data from a serial sensor and sends it when the server is reachable again. But after an outage, only the last packet was left.
Finding the change
Buffered mode had worked before, so something had broken it. The record database stores records by sequence number in a nested folder tree. I bisected on the device across releases and merges, and landed on a merge from an unrelated feature: a rewrite of the FIFO record database.
The flag nobody read
The multi-save call still took a sequence flag, but the rewritten code never read it. So every packet was written as sequence 0. The database then decided the file was corrupted and erased it, leaving only the newest packet.
Two more problems on the same path
Buffered mode always used the TCP path, because only the non-buffered path checked the server type. So a device set up for MQTT never sent its buffered data at all. And in the JSON codec for the cloud, serial data over 18 B was dropped and 0x0A/0x0D bytes were changed. There was also an old edge case in delete-all.
The fixes: each buffered packet gets its own storage slot, which also fixed delete-all. Buffered mode now uses the same server-type check as the rest, so it works over TCP and MQTT, ASCII and binary. The JSON codec sends the full serial data as a hex string.
Conclusion
After an offline period, 11 of 11 packets now arrive, instead of 1 of 11.
Nothing caught the original break because the record database had no unit tests. I wrote the first ones, with an in-memory fake file system, a 269-line fake record-manager harness and a 5-step test plan.
The symptom was in one feature and the cause was in another. Bisecting is what connected them.
Lost records, lost commands, no device bug
Records missing from a download, and commands that stopped reaching trackers. Both looked like firmware bugs. One was an old reader tool, the other a carrier NAT that forgot devices after about 15 seconds of quiet.
When data goes missing between a tracker and a server, everyone looks at the device first. It’s the part we build.
These two tickets were both about lost data, and in both the device turned out to be fine. The work was proving where the data really stopped.
Missing records
The first ticket estimated that the “Read records” download in the configuration tool was missing about 30% of the records on the device. If true, that’s a firmware data-loss bug.
The desktop tool for opening the downloaded .bin file had been discontinued. So I wrote my own parser in Python. It decodes the device’s record protocol straight from the dump and counts every record.
On the file I checked, the old tool showed 101 of 201 records. My parser found all 201. The old tool just didn’t read them.
I published the parser in our internal tools docs, so anyone can check a download themselves. QA retested the download and it passed.
Missing commands
In the second ticket, trackers sent data to the server over UDP, and the commands the server sent back stopped arriving.
I reproduced it first. The modem was the obvious suspect, so I opened a case with the modem vendor. Then I suspected the forwarding on our office network. Neither explained it. The ticket stayed open from 2025 until I found the cause in February 2026.
A NAT with a short memory
On this operator’s APN, the device had no public address. It sat behind the operator’s carrier-grade NAT, which keeps a port mapping open only while traffic flows both ways. On this network, the mapping closed after about 15 seconds of quiet. Any command after that went to a port the NAT had forgotten, and was dropped before it reached the modem.
The fix was a setting: the existing network ping now runs every 10 s, so the NAT never sees 15 seconds of quiet. I also proposed a default ping interval for UDP links in the firmware, and updated the test automation script.
Conclusion
Records: all 201 were in a file the old tool read as 101. Commands: with a ping every 10 s, they reach the trackers again. The 201 is from the one file I checked, and 15 s is this operator’s timeout, not a universal number.
Neither fix touched firmware code. The NAT case took long because the first two suspects were reasonable ones.
The battery gauge that took 3 hours to say 10%
Three battery complaints turned out to be three different bugs. One gauge took 3 hours to show its first 10%, and now takes 1.5. Another got stuck at 99%; it now reaches 100%, proven with a simulated −40 mV ADC offset.
Three battery complaints came in from the field. On one model, the gauge took 3 hours to show its first 10%. Some devices stopped charging below 4.15 V. Others sat at 99% forever after a full charge.
It looked like one battery problem. It was three separate bugs.
The wrong table
The slow model was using the percentage table for a 170 mAh cell on a 1,000 mAh battery. I built new tables for 170, 1,000 and 1,800 mAh cells. I also wrote and documented an algorithm that moves toward the measured value and can’t get stuck.
Believing the charger too early
The early stop came from the charger chip reporting DONE too early, and the firmware believing it. Now the firmware retries up to 3 early DONE signals before it accepts one. The noisy charger status lines also got a 3-sample debounce with hysteresis.
Stuck at 99%
The charger stops at 4,125–4,140 mV, but the table needs at least 4,148 mV for 100%. So a full battery always came out 8–23 mV short. On top of that, an older guard blocked the 10-minute recalculation, so the gauge never moved off 99%. Now, when the firmware accepts the battery as full, it sets 100%.
Testing with field conditions
To test the 99% fix, I built a test image with a simulated −40 mV ADC offset, like the readings from the field. The fixed build held 100%. Each fix also has a unit test.
Conclusion
The gauge now shows 10% after 1.5 hours instead of 3, holds a steady 100% without flickering back to 99, and batteries charge to 4.15 V again.
Splitting the complaints apart early meant each bug got its own fix and its own test.
Why did locking a car take 10 seconds?
Locking a car remotely had crept past 10 seconds at p90. Commands, records and the protocol engine were all stuck in one MQTT task. After I split it in three, record send p90 dropped from 4.06 s to 0.12 s.
A fleet customer locks and unlocks cars over MQTT. The server sends a command, the tracker does it and replies. Over time, the p90 for a lock crept past 10 seconds. That’s a long time to stand next to a car, waiting for the door.
Measuring first
I wrote a script that pulls the round-trip time of every publish out of the device logs, and added latency probes at each stage. Then I changed one thing at a time, each behind its own flag, and measured again.
One task doing everything
There was no single bug. It was a traffic jam. Commands, record publishing and the protocol engine all ran in one task. That task waited up to 60 s for a socket send, and with no limit at all for an MQTT ACK. One slow write and everything behind it stalled, including a lock command stuck behind a queue of records. On top of that, every record was built twice.
Splitting it up
I split MQTT into three tasks: commands, publishing and the protocol engine. Now a task waiting on the network doesn’t hold up the others.
They still share one TLS session, so writes have to be serialized somewhere. I first tried the shared network layer. It deadlocked. It works in the layer that owns the TLS session.
Both waits now have a cap. The transport answers ok, busy or fatal, so a busy modem no longer kills the TLS session. The duplicate record build is gone. I also fixed a record-storage mode that an earlier update had broken.
Conclusion
Record send p90 went from 4.06 s to 0.12 s, and the median from 1.03 s to 0.05 s. The time one reply held the connection layer went from 495–1,144 ms to 32–240 ms. A calibration command went from 1,664 ms to 4 ms.
One caveat: lock replies use the same publish path, so record send time is what I measured. I don’t have the end-to-end lock time after the fix. The delay still visible in the customer’s logs came from sockets dropping while roaming.
Nothing here was slow on its own. The problem was waiting without limits, and it only showed up once each stage was measured.
The 100-second timeout that became 7.8 hours
After a release, devices on our stability stand kept crashing. Instead of reading crash dumps one by one, I diffed the code between the good and the bad release. One constant had been multiplied twice: a 100 s timeout had become 7.8 hours.
After a release, devices on our long-run stability stand started crashing with abnormal restarts. Different modems, different models, no obvious pattern.
When unrelated devices start failing right after a release, the release is the place to look.
Diff, not dumps
I could have gone through the crash dumps one by one. Instead I diffed the record-save, record-send and watchdog code between the last good release and the bad one. Everything matched except one constant.
100 seconds became 7.8 hours
The timeout for the modem’s data-send command was multiplied twice. 100 s became 7.8 h.
With 100 s, a send that gets no answer fails, the task moves on and answers the watchdog. With 7.8 h, the task just waits. The record path stays blocked, watchdog pings go unanswered, and the watchdog restarts the device. A wrong timeout looked like a crash.
Fixing it, and the path around it
The fix itself was one line. Since I was already in that code, I also made the path more forgiving.
The watchdog now warns at 50% and 75%, and retries pings, so a slow answer isn’t treated as a dead task. The record manager slows down at 70% load, with at most 5 s of delay. Logs are rate-limited, and I fixed memory leaks in the SMS and network code. Beacon work waits when load is above 80%.
Conclusion
The stand went quiet. The only restarts left were on one small model under heavy bursts, which is a known limit. The whole branch went into the base firmware, so every build ships with these safeguards.
Comparing three code paths was a much smaller search than a stand full of crash dumps.
Three features against one RAM budget
400 geofences, 10,000 driver IDs and two data connections at once, on a tracker already short on RAM. I estimated each before writing code, and each hit a different limit: free RAM, boot time, or room for one TLS session.
Tracker firmware runs on a fixed amount of RAM, and most of it is already taken. Three requests each needed more of it. A client wanted 400 geofences, each a 15-point polygon. Another project needed the driver ID list to hold 10,000 entries. A customer needed two data connections on different APNs, open at the same time.
I wrote an estimate for each before writing any code: 154 h of firmware work plus 40 h for the configuration tool, 60 h, and 116 h. Each one hit a different wall: free RAM, boot time, and room for only one TLS session.
Geofences: measure, then cut
Every polygon point lives in RAM. With 400 zones, I measured 42 kB of free RAM at 10 points per polygon, and 24 kB at 15. That was an early measurement, and it was dangerously low.
Coordinates come with six decimal places, about 10 cm, and were stored as doubles. I pushed for 32-bit floats instead: half the bytes per coordinate, and a little less precision. I also pushed back on per-zone names as scope creep. It still didn’t fit, so I removed reserved features the build never used, and shortened a rarely used list to free 2.4 kB.
What the cuts broke
Trimming had removed some base parameters by mistake, so the immobilizer re-locked after unlock. I restored them. The #ifndef blocks I added to save RAM overlapped and changed an IO default, so an alarm event was always on. I patched that. And the configuration tool still treated coordinates as doubles, so every config showed unsaved changes. I updated it to float.
10,000 driver IDs: boot was the problem
A driver touches an ID key to the 1-Wire reader, and the firmware checks it against the list for the immobilizer. With 10,000 entries the feature worked, but boot took about 100 s, against a 30 s limit. I made the list load faster at boot, and cut RAM and BSS use so the list fit.
Two connections, one TLS session
The firmware assumed there was only ever one data connection. The estimate is where the real limit showed up: there was RAM for one TLS session, not two. So I reworked the core network API to handle two PDP contexts. Both connections stay up, and secure traffic goes to one server at a time, so one TLS session is enough.
In testing, the backup connection was torn down and rebuilt on every send. The network rejected that, and the modem got stuck in a restart loop. I fixed the backup handling in several places and removed the settings-refresh loop.
Conclusion
Geofences: QA confirmed “400 zones configurable and polygons working”, and the build fits in RAM. Driver IDs: boot went from about 100 s to under 30 s, on two tracker models. Two connections: the customer accepted the design, which touches about 30 files across the network API and two modem drivers.
On a device this full, every feature costs something else. Here it was precision, unused features and a second TLS session. And cutting RAM breaks things too, so cuts need testing like any new feature.
Heart rate that fell apart when you walked
A health smartwatch lost the heart rate as soon as you started walking. I built a rig to compare it against a chest strap, then rebuilt the algorithm. Error while walking dropped from 8.2 to 5.5 bpm, and RAM from 6.9 to 2.7 kB.
A health smartwatch with ECG, heart rate and SpO2 lost the heart rate as soon as you started walking. Walking is the hard case for a wrist sensor: every step moves it and adds its own rhythm to the signal.
It’s easy to tweak a heart-rate algorithm until it looks good on one walk. To know if it’s really better, you need a reference. So I built two things: a way to measure it, and a better algorithm.
The first bug was a unit
The motion detector compared a threshold of 0.10 against acceleration in the wrong unit. Every pulse was flagged as motion and thrown away.
A way to measure it
I built the whole loop. A BLE service on the watch streams per-beat PPG and heart rate, plus raw PPG at 100 Hz and accelerometer data at 50 Hz. A Kotlin Android app records the watch and a reference chest strap side by side, with activity markers. Python scripts score each session and replay the filters, so I can try a change on recorded data before it goes on the watch.
An algorithm that expects motion
The new estimator runs a DFT on a 25 Hz grid. The accelerometer gives the step rate, and notches at 0.5×, 1× and 2× that rate remove the step and arm-swing peaks. A beat-interval prior and a bounded tracker stop the estimate from jumping around, and clip, jump and artifact-hold gates keep it steady.
Fitting it on the watch
The MCU was already at 98% RAM. I ported the estimator to C++ with an optimized FFT library and checked it against golden vectors from Python. Before switching it on, I ran it in shadow mode next to the old one. Along the way I also fixed a stack overflow that only happened while the watch was being worn.
Conclusion
Mean absolute error while walking, against the chest strap, went from 8.2 to 5.5 bpm.
The estimator’s RAM went from 6.9 kB to 2.7 kB.
The algorithm gets the attention, but the chest strap, the recorder app and the scoring scripts are what make 5.5 bpm a measurement and not a guess.
4,000 beacons vs. one phone link
In crowded BLE areas, trackers lost their phone link, and some got stuck in a reset loop. I measured everything in a flood of about 4,000 beacons before touching any code. Stuck cycles went from 17 of 62 to 0 of 350.
Trackers had two problems in places crowded with BLE devices. After a phone session ended, the tracker stayed invisible to the phone. And in very dense areas, it reset every 33 seconds.
Neither shows up at a desk. So I brought the crowd to the bench.
Measuring before changing
I set up a flood of about 4,000 beacons. Before changing any code, I measured stuck cycles, resets and how many phone sessions passed.
Getting the phone link back
For the invisible tracker, I added three layers that kick in one after another. A probe checks that the phone link is really alive. With no phone connected, an advertising watchdog restarts advertising if it gets stuck. And if the Bluetooth stack doesn’t confirm that advertising started, the radio restarts.
Stopping the reset loop
For the reset loop, scanning now runs in slices, so the scanner can’t hog the system. The records that beacons generate are also paced.
Conclusion
Stuck cycles went from 17 of 62 to 0 of 350, over 10 hours. The reset loop went from one reset every 33 s to none in 80 minutes. Phone sessions passing went from 51% to 80%.
80% still means one session in five failed, and my data doesn’t say why. For the vendor’s part of the problem, I sent them an air log of 1,009 sniffed connection events.
The baseline is what let me say “better” with numbers.
30,000 records over BLE in 5–6 minutes
Two BLE projects, one limit: how much data fits in a packet. In a phone app, 30,000 records now download in 5–6 minutes, every chunk CRC-checked. In tracker firmware, smaller chunks fixed the iPhone link, and I built the MTU exchange the vendor wouldn’t add.
BLE sends data in small packets. Both sides agree on a maximum size, the MTU, and anything bigger doesn’t get through.
I ran into this limit from two sides: a phone app that downloads logs from a cold-chain temperature recorder, and tracker firmware that iPhones couldn’t talk to.
The app: every record has to arrive
I co-developed the React Native app for a family of BLE sensors, on iOS and Android: about 140 commits, about 85 tickets and 14 releases to Google Play and TestFlight.
For the recorder’s cold-chain certification, every record has to arrive, and you have to be able to show it did. I added a CRC check on every chunk, so each row in the export says whether its chunk arrived intact. I also fixed a page-header shift, where the header sometimes arrived one slot late and the app read a record chunk as the header. Then I raised the MTU to 247 and the chunk size from 15 to 60 records.
To test full-size downloads, I added a fake-record generator to the app. 30,000 records now download in 5–6 minutes.
The firmware: iPhones and 512-byte chunks
An iPhone can only reach the tracker over BLE. The tracker isn’t MFi certified, so iOS can’t use its Bluetooth Classic serial link. And our firmware sent BLE data in fixed 512 B chunks, which iOS can’t take.
In 2024 I wrote up how MTU negotiation really worked on the platform. The chip vendor’s Bluetooth layer negotiated the MTU, and our code sent fixed 512 B chunks on top of it. I set the chunk payload to 182 B, tested on iOS and Android, and released it as an experimental iOS-compatible firmware. That fixed the iPhone link.
When the vendor said no
The vendor SDK had no MTU negotiation in its GATT server layer. In 2025 I opened a support case, and they declined to add it. So I built the MTU exchange in the GATT client layer myself, for links where the tracker is the client. I tested it in the cold-chain firmware, where the tracker reads temperature recorders as a client, and updated the BLE MTU docs.
Conclusion
In the app, 30,000 records download in 5–6 minutes, and every chunk is CRC-checked. In the firmware, QA verified 182 B chunks on two tracker models.
On the firmware side, the write-up mattered more than the code. Once it was clear who negotiated the MTU and what our code did with it, the iPhone fix was small.
Writing my own trace decoder
Debugging our main cellular platform depended on the chip vendor’s closed trace tool. I wrote my own in C++20 and checked it against the vendor’s output at every step. On 108,755 traces, it produces the same bytes.
Debugging our main cellular platform depended on the chip vendor’s trace tool. It was closed, so when it had problems, nobody outside the vendor could fix them.
A replacement is only worth using if people trust it as much as the original. So the real job wasn’t just building it. It was proving that it reads traces exactly the way the vendor’s tool does.
What it does
I wrote it in C++20, with a Dear ImGui GUI and a command-line version. It handles capture, decoding, merging, pcap export, GSMTAP export for Wireshark, symbols and a Ctrl+K command palette. It idles at 1.9% CPU with a 2-million-row log open.
Checked against the vendor’s tool
Every stage is tested against the vendor tool’s own output, with golden tests covering 3.7 million records. Decoding is compared byte for byte. I also rewrote one of the vendor’s message decoders and checked it the same way: 37,340 of 37,340 messages match. Capture has nothing to compare against, so it got a soak test instead.
Crash analysis
The crash analyzer walks the RTOS task blocks and queues, and unwinds the stacks using DWARF CFI. Its call stacks are compared with the vendor’s crash report.
Conclusion
Decoding is byte-identical on 108,755 traces, and 8.7 MB decodes in 0.8 s. 57 of 57 crash call stacks match the vendor’s report. A 30-minute capture soak lost nothing on the PC side.
Matching the vendor’s tool proves that mine agrees with it, not that both are right. For a replacement, agreement is the point.
3,505 files compiled, 597 shipped
Our firmware needed a software bill of materials for the EU Cyber Resilience Act, but standard tools couldn’t read the image. Working from the linked ELF, I found that only 597 of 3,505 compiled files ship. I also reworked unlock, updates and TLS.
Tracker firmware needed stronger device security, a modern TLS stack and a software bill of materials (SBOM) for the EU Cyber Resilience Act. This page covers the design only. The details stay private.
Which files actually ship?
Standard SBOM tools can’t read this kind of RTOS firmware image, so I worked from the linked ELF instead. Most of what the compiler sees never ends up in the image, and listing all of it would make the SBOM longer and less accurate.
I compared four ways of counting the same build. The compiler saw 3,505 files. The linker’s “unused” list was misleading. File symbols in the ELF over-count by about 2x. Counting only files that own a sized symbol in a loaded section gives 597, and that’s what the SBOM lists. The CI pipeline handles the scanner’s rate limit and uploads to Dependency-Track with CPE identifiers.
Unlock and passwords
Unlocking is challenge-response, and the device verifies a signature. Passwords are stored only as salted PBKDF2 hashes and compared in constant time. Attempt counters survive a power cut. If the lock’s record gets damaged, the lock stays closed.
Updates and TLS
Firmware updates got downgrade and tamper protection, designed together with five new unit-test suites. I shipped it as release owner.
I moved TLS to the library’s current long-term-support line, with TLS 1.3. MD5, SHA-1, DES, renegotiation and DTLS are off. Only AEAD and ECDHE cipher suites are left.
Conclusion
Fail-first unit tests, with fakes for crypto and storage, caught 12 of 12 deliberate mutations. I also ran bench loops with reboots and power cuts.
For security code, a test suite that has never failed doesn’t prove much.
71 tests that run on a real watch
71 tests run on a real smartwatch over BLE, and any new crash fails the test, even if the test’s own checks pass. Together with a memory gate in CI, the setup has found 16 bugs.
A test on a real device can pass while the device crashes in the background. On a watch with little RAM, a change can also pass every test and still eat memory that a later feature needs.
So I built the harness to judge the whole watch, not just the feature under test, and to put a memory cost on every change.
The setup
A pytest harness drives a production smartwatch over BLE and a debug probe. It’s 75 files and 8.1k lines.
A crash guard around every test
Before each test, a guard reads the coredumps, the uptime and the BLE link. After the test, it checks for a new coredump, an unexpected disconnect or an uptime reset. If it finds one, the test fails, even if all its own checks passed.
A memory gate in CI
Every change goes through a RAM budget in bytes, a stack limit worked out from the call graph, and a stack and heap peak recorder that writes to flash. The RAM gate caught a stack-size cut that crashed over-the-air updates.
What it found
The harness turned up 16 bugs, each fixed in its own commit. As a side effect, font loading went from 180 ms to 29 ms.
I also built a binary event journal for the watch. It showed that every ECG abort had been reported with the wrong reason, because of an enum cast. I fixed it with an explicit mapping and a static_assert.
Conclusion
All 71 device tests pass with no new crashes on the current build, and the harness has found 16 bugs.
The crash guard is the part that pays off most. It turns every test into a crash test too.
The inventory app that outgrew 50k reads
My test-hardware inventory app hit Firebase’s 50k read limit. I priced switching to an off-the-shelf tool (64 h) against moving to my own API (40 h), and moved it. It has been used company-wide since June 2026.
R&D and QA have a lot of test hardware, and every device is identified by its IMEI. I built and run the inventory for it: Expo / React Native for Android, iOS and web, an Express API, MongoDB, a C++ scanner service, Docker and GitLab CI. So far that’s at least 33 tickets and about 356 commits.
It started as a small tool and grew into a company-wide service.
Outgrowing Firebase
The first version went live on GitLab Pages the same day it was asked for. Then it moved to Firebase. When it hit Firebase’s 50k read limit, it needed a new home.
Before rebuilding, I priced the alternative. Moving to an off-the-shelf open-source asset manager came to 64 hours. Moving my app to its own backend came to 40. So I moved it to MongoDB behind my own Express API.
Scanning etched codes
The app reads QR and DataMatrix codes with zxing-cpp in three places: a Kotlin module on Android, WASM in the browser, and the scanner service on the server. Laser-etched codes are hard to read, so the decoders try several scales and inverted images.
I also added a server-side photo decoder (OpenCV, Tesseract OCR, zxing). It reads every code on a PCB from one photo, and does it better than the live scan.
Everyday use
Search supports AND/OR, exact (=) and partial (~) matching, suggestions and shareable filter links. Excel import and export puts each location on its own sheet. Device details fill in automatically, duplicate IMEIs are rejected, and devices can be reserved and released. There’s a daily backup, and admin mode has a git-like change history with revert.
An API for CI
The firmware CI pipeline needed the device list, but the app only had interactive single sign-on. So I built a separate machine API for CI, with its own credentials. CI firmware builds now use it to fetch their device list.
Webhooks are signed with HMAC, responses carry ETags for 304s, and a test catches the OpenAPI docs falling behind the routes.
Conclusion
Sorting 2,640 devices went from 487 ms to 92 ms. The app has 352 Jest tests and 1,371 screenshot comparisons. In June 2026 it launched company-wide, with company single sign-on.
Pricing the off-the-shelf option was worth it, even though I didn’t take it. 40 hours against 64 made it a decision, not a preference.
Tools
| Tool | What it does | Stack |
|---|---|---|
| Bench console | A web console for the device bench, with plugins for a GPS server, test runner, log parser, crash analysis, release creator, doc search, update-failure reproducer and TLS port router. 1,547 tests. | Python · FastAPI · Preact · SQLite |
| Trace decoder | My own replacement for a closed vendor trace and crash tool. Read it. | C++20 · CMake · Dear ImGui |
| Bench case layer | Bench cases written as JSON, covering serial, power, PSU, BLE, flashing and MQTT. A setup watchdog took one test suite from 0 of 5 passing to 5 of 5. | Python |
| Device inventory app | Used by the whole company since June 2026. DataMatrix and QR scanning on Android, web and server, search, Excel import/export, and a machine API used by CI firmware builds. Sorting 2,640 devices went from 487 ms to 92 ms. | Expo · React Native · Express · MongoDB |
| Device server | A server for tracker protocols, with file transfer, update sessions, commands and a serial monitor. | Python · PySide6 · SQLite |
| HR comparison app | Records the watch and a reference chest strap side by side, with activity markers. | Kotlin |
| Parsers | Records, AT logs, update logs, signal quality, LTE feature groups, no-fix reasons, NMEA trip maps and MQTT latency. | Python |
Contact
Say hi.
- [email protected]
- linkedin.com/in/alp-orak
- github
- github.com/alporak
- based in
- Vilnius, Lithuania