IPSC3 includes an independent ipsc3-monitor service. It does not run in the DMR packet loop,
POC RTP loop, or a dashboard request, so monitoring cannot delay live audio.
The monitor samples:
/api/health HTTP endpointSELECT 1If MySQL is unavailable but ipsc3.service remains active, the IPSC heartbeat is shown as unknown
rather than creating a duplicate IPSC alarm. The MySQL failure remains a separate alarm.
The MMDVM listener check is intentionally separate from the daemon check. Its heartbeat is advanced
inside the hotspot UDP packet loop and persisted by the daemon's background heartbeat worker. This
detects a process that is still running while its hotspot listener thread has exited or is blocked.
Database cleanup and packet-processing errors are contained and rate-limited instead of terminating
the listener. A fatal UDP socket error exits the daemon so systemd can restart it cleanly.
Every enabled repeater marked operational is checked from the independent monitor process. Motorola
IPSC, Hytera, and provisioned MMDVM/Homebrew repeaters are included. A newly provisioned repeater is
shown as awaiting its first connection and does not generate an outage alert until it has registered
successfully at least once.
The normal failure and recovery confirmation counts debounce each repeater independently. When the
repeater Pushover option is enabled, Pushover receives one alert when a repeater is confirmed down
and one recovery when it is confirmed back up. When repeater email is enabled, SMTP2GO is configured,
and the repeater has a valid custodian email address, the custodian receives a plain-text alert and
recovery email. Failed custodian email attempts are retried no more often than every ten minutes. A
recovery email is sent only when an outage alert was previously sent to that recipient.
Repeater state is not advanced while MySQL or the IPSC3 daemon is unhealthy. MMDVM repeater state is
also paused while the MMDVM listener heartbeat is unhealthy. This prevents one central server fault
from producing a misleading outage message for every repeater site.
Failure and recovery confirmation counts are configurable. The defaults require two consecutive
failed checks to turn an alarm on and two consecutive successful checks to turn it off. Notifications
are emitted only for those transitions, not on every sample while an alarm remains active. Pushover
and email each have independent switches for repeater transitions and other service transitions.
Service email covers the dashboard, IPSC3 daemon, MMDVM listener, POC daemon, MySQL, load, and disk
checks and is delivered to the configured administrator alert address.
Alarm notifications use high priority. Recovery notifications use low priority. Emergency/repeating
priority is intentionally not used.
Administrators configure monitoring under Configuration > Monitoring. Available controls include:
Pushover requires both a 30-character application token and a 30-character user/group key. SMTP2GO
uses the documented POST https://api.smtp2go.com/v3/email/send JSON endpoint and requires an API
key plus a sender address authorised by that SMTP2GO account. Values are stored in a mode-0600
runtime configuration file. Existing secret values are never returned to the browser; the dashboard
only reports whether each value is configured.
The Monitoring tab displays current state for each check and a bounded alarm history with local time,
alarm-on/alarm-off transitions, details, and Pushover and email delivery status. A disabled delivery
route is recorded as disabled, while the underlying health transition remains in the history. Runtime files are stored under
/opt/ipsc3/runtime/monitor by default. The monitor's own operational output is available through:
journalctl -u ipsc3-monitor.service