fix(boot): OTA-enable app layer + sensorgnome ordering + app-service retry-forever

CHANGELOG.md

@@ -16,6 +16,30 @@ regenerated from these entries.
 
 ### Fixed
 
+- **OTA no longer clobbers a Quectel's `autoconnect` (cellular survives every update).**
+  `deploy_dir` copied the repo `station-modem` profile wholesale, and its "preserve the
+  runtime-owned keys" step only stripped them from the *diff*, not the installed file — so
+  the repo's `autoconnect=false` overwrote the live value on any redeploy. It bit only
+  Quectel because NetworkManager omits `autoconnect=true` (the default) from the keyfile, so
+  "preserve the line if present" preserved nothing. `merge_preserved_keys` now reads an
+  **absent** preserved key as the daemon default and omits it (never falling back to the
+  source value); `install-network.sh` also re-runs `modem-datapath.sh` after the deploy to
+  re-assert the per-modem policy. Verified on real NM 1.30.6 (absent line → `autoconnect: yes`)
+  and on hardware. (PR #50.)
+- **Application layer is now OTA-self-enabling, correctly ordered, and retries forever.**
+  Boot-sequence hardening from the boot-sequence review: (1) `install-systemd.sh`'s
+  `MUST_BE_ENABLED` now covers the Node/SensorGnome units (`station-hardware-server`,
+  `station-radio-interface`, `station-web-interface`, `station-lcd-interface`,
+  `station-boot`, `bootcount`, `sensorgnome`) — previously deployed as files but enabled only
+  by Ansible/manufacturing, so a lost symlink or an Ansible-free image came up with **no app
+  layer**. (2) `sensorgnome.service` now orders `After=ctt-board-detect.service
+  bootcount.service` — the units that actually produce its two synchronous inputs
+  (`/etc/ctt/station-id`, `/etc/bootcount`); the old `After=station-boot` rested on a stale
+  premise (station-boot no longer writes `station-id`) and raced `bootcount`. Its malformed
+  `WantedBy=…station-boot.service` is dropped. (3) The four long-running `station-*` services
+  get `StartLimitIntervalSec=0` + `RestartSec=5`, so a transient boot-time crash-loop retries
+  forever instead of hitting systemd's default 5-starts-in-10s give-up and leaving a headless
+  station dark (matching the radio driver's policy).
 - **Quectel APN is now selected from the SIM's IMSI, not its ICCID country code.**
   `ctt-modem-provision`'s Quectel driver mapped ICCID digits `[2:4] == "46"` to the Telenor
   APN and everything else to `super`. Telenor also ships SIMs in an **`8901` (US-numbered)

system/scripts/hooks/post-merge.d/install-systemd.sh

@@ -43,6 +43,19 @@ MUST_BE_ENABLED=(
   ctt-modem-wake.service         # wake a shut-down Telit at boot (ON_OFF# pulse) so a hard reset self-recovers; runs Before modem-boot-state
   ctt-modem-provision.service    # idempotent ECM provision GUARD; Before MM, no-op on a provisioned modem, converts a fresh/swapped RNDIS one
   ctt-modem-ecm-up.service       # bring up the ECM data iface mdm0 (DHCP + fallback route); NM won't manage an MM modem net port
+
+  # Application layer (Node services + SensorGnome). Enable here so an OTA self-heals
+  # a lost symlink and an image built without the legacy Ansible enablement still comes
+  # up with a full app layer — previously these were deployed as files only and their
+  # enable state depended solely on Ansible / manufacturing.
+  station-hardware-server.service  # REST hardware API :3000 (the other station-* units order after it)
+  station-radio-interface.service  # radio acquisition + pipeline, WS :8001
+  station-web-interface.service    # dashboard :80
+  station-lcd-interface.service    # LCD menu + buttons
+  station-boot.service             # modem data-path policy + per-SIM APN (oneshot, After ModemManager)
+  bootcount.service                # writes /etc/bootcount (SensorGnome reads it at load) + SG hub-map symlink
+  sensorgnome.service              # SG master :3010 (tag detection)
+
   # ctt-radio-driver@.service and ctt-blu-driver@.service are TEMPLATES — udev
   # activates per-channel instances via ENV{SYSTEMD_WANTS}; they are deployed as
   # files but must NOT be enabled here.

system/systemd/sensorgnome.service

@@ -1,11 +1,13 @@
 [Unit]
 Description=Sensorgnome Startup
-# Wait for station-boot to finish before starting. station-boot creates
-# /etc/ctt/station-id (among other things), and sensorgnome's master/machine.js
-# reads it synchronously at module load — without the After= dependency,
-# sensorgnome races station-boot and crashes with ENOENT, then waits the
-# 60s RestartSec before retrying. With it, sensorgnome starts cleanly first time.
-After=network.target station-boot.service
+# master/machine.js reads TWO files synchronously at module load:
+#   /etc/ctt/station-id  (written by ctt-board-detect.service)
+#   /etc/bootcount       (written by bootcount.service, itself After=station-boot)
+# Order after BOTH producers so a fresh boot can't ENOENT-crash and then wait out
+# the 60s RestartSec. bootcount is After=station-boot, so the old modem-policy
+# ordering is subsumed. (Historic note: this comment used to say station-boot
+# creates station-id — ctt-board-detect owns that identity now.)
+After=network.target ctt-board-detect.service bootcount.service
 
 [Service]
 ExecStart=nohup /etc/ctt/scripts/start-sensorgnome.sh
@@ -15,4 +17,4 @@ RestartSec=60
 Restart=on-failure
 
 [Install]
-WantedBy=multi-user.target station-boot.service
+WantedBy=multi-user.target

system/systemd/station-hardware-server.service

@@ -5,12 +5,17 @@ Description=start station-hardware-server
 # After=station-hardware-server, so they inherit the ordering.
 After=network.target ctt-board-detect.service
 Wants=ctt-board-detect.service
+# Retry forever on a headless field station: without this, systemd's default
+# start-limit (5 starts / 10s) gives up and leaves a core service dark until a
+# manual reboot. Paired with RestartSec below so it backs off, not hammer-restarts.
+StartLimitIntervalSec=0
 
 [Service]
 Type=simple
 WorkingDirectory=/lib/ctt/sensor-station-software
 ExecStart=/usr/local/bin/node ./src/station-hardware-server/bin/www.js
 Restart=on-failure
+RestartSec=5
 TimeoutSec=60
 
 [Install]

system/systemd/station-lcd-interface.service

@@ -1,12 +1,15 @@
 [Unit]
 Description=start station-lcd-interface
 After=station-hardware-server.service
+# retry forever, not systemd's default 5-starts-then-give-up (see hardware-server)
+StartLimitIntervalSec=0
 
 [Service]
 Type=simple
 WorkingDirectory=/lib/ctt/sensor-station-software
 ExecStart=/usr/local/bin/node ./src/station-lcd-interface/index.js
 Restart=on-failure
+RestartSec=5
 TimeoutSec=60
 
 [Install]

system/systemd/station-radio-interface.service

@@ -1,12 +1,15 @@
 [Unit]
 Description=start station-radio-interface
 After=station-hardware-server.service
+# retry forever, not systemd's default 5-starts-then-give-up (see hardware-server)
+StartLimitIntervalSec=0
 
 [Service]
 Type=simple
 WorkingDirectory=/lib/ctt/sensor-station-software
 ExecStart=/usr/local/bin/node ./src/station-radio-interface/index.js
 Restart=on-failure
+RestartSec=5
 TimeoutSec=60
 
 [Install]

system/systemd/station-web-interface.service

@@ -1,12 +1,15 @@
 [Unit]
 Description=start station-web-interface
 After=station-hardware-server.service
+# retry forever, not systemd's default 5-starts-then-give-up (see hardware-server)
+StartLimitIntervalSec=0
 
 [Service]
 Type=simple
 WorkingDirectory=/lib/ctt/sensor-station-software
 ExecStart=/usr/local/bin/node ./src/station-interface/bin/www.js
 Restart=on-failure
+RestartSec=5
 TimeoutSec=60
 
 [Install]