Releases
Observability Module for Ignition 8.3
Every published version of the module, with what it has been verified against, plus direct downloads for the .modl asset, the user manual and the Grafana dashboard pack.
v0.9.0
v0.9.023 Oct 2026OpenTelemetry and Prometheus observability for Ignition 8.3: curated gateway metrics, logs, traces and opt-in audit export, pushed to any OTLP backend or scraped on :9464.
- Ignition:
- 8.3.0+
- Backend:
- Any OpenTelemetry HTTP/protobuf endpoint, or Prometheus scraping the gateway
- Other:
- Ignition Edge: contact Mustry Solutions before deploying
Changes
- Breaking: the gateway CPU metric (ignition.gateway.cpu.utilization, ignition_gateway_cpu_utilization_ratio in Prometheus) is now a 0 to 1 ratio, as documented. It used to be 100 times too high. If you alert on it or chart it in your own dashboards, divide your thresholds by 100 (> 80 becomes > 0.8). The bundled dashboards now show it correctly.
- Breaking: the project, historian, provider, engine and queue labels now keep the name as you configured it. They used to be lower-cased with punctuation turned into _, so two instances such as Line-1 and Line_1 could share one series and lose data. A query that filters on a value needs the new spelling, for example provider="System" or queue="Tag - default - DB Values". To cover the upgrade, match both: provider=~"system|System". The bundled dashboards and alert rules need no change.
- Breaking, for redundant pairs that push over OTLP: the backup now reports itself as <master name>/backup instead of using the master's name. Before, the data of both nodes mixed into one series, and the backup-active alert fired on a healthy pair. Scraped Prometheus data is not affected.
- Breaking, for custom metrics on the Prometheus endpoint: names and label keys with repeated punctuation convert differently (line..1 becomes line_1, and a label key that starts with a digit gets key_ in front). Two custom metrics that now convert to the same name make the whole /metrics scrape fail until one is renamed. Use letters and digits with single ., _, - or / between them. The curated metrics and OTLP push are not affected.
- Breaking, if you upgrade from 0.7.0: export health is now tracked per destination. Export/MetricsHealthy and Export/LogsHealthy are true only when every enabled destination is healthy, and a destination that cannot read its secret switches off on its own while the others keep running (alarm on the new FailedClosed tags). Gateways with one destination see no change.
- If you upgrade from 0.7.0 and your backend copies resource attributes onto series (the Grafana OTel-LGTM stack does), every series starts fresh once, because telemetry now also carries service.version, host.name and the module version. Graphs recover on their own within minutes. Plain Prometheus scraping is not affected.
- If you copied the bundled alert rules into your own setup, copy them again. IgnitionDeviceFaulted now also fires for a disconnected device, the most common failure. IgnitionClockDrift could never fire before and now counts drift events. The certificate rules now filter on scope.
- The /health endpoint reports pushEnabled instead of otlpEnabled. Update any script that reads it.
- Data minimisation, all off by default: pseudonymise user names (the same user always gets the same pseudonym, so counting still works), redact IP addresses, limit which MDC keys are exported, and drop stack traces. One setting, Data Profile: Minimal, turns all of them on at once, and a settings lock stops anyone loosening them from the Settings page. User manual §3.11 and §3.12.
- "What leaves this gateway": the Export Status page now lists what the module actually sent in the last hour, per signal and destination: hosts, metric names, attribute keys and series counts, never the values. Download it as JSON for a security or supplier review.
- New metrics, with example alert rules in the Grafana pack: latency percentiles for database queries, scripts, the historian and Perspective; load per database connection; JVM memory, garbage collection and class loading; redundancy role and state, and the status of each gateway network connection; licence mode, edition and installed modules; gateway health-check problems; scheduled backup age; and an alarm activation counter for alarm-flood alerts.
- Certificate expiry: the gateway's HTTPS certificate, its gateway network certificate and its trusted peer certificates, with alert rules for certificates that are about to expire.
- Connection events: with log export on, every database, device or OPC connection that changes state sends one log event naming the connection, so an alert can say which one faulted.
- Sending data through a corporate network: new settings for an HTTP proxy (without authentication) and for extra CA certificates, for proxies that inspect TLS. No more editing the gateway's Java trust store.
- Pushing straight to Datadog now works, and direct push to Grafana Cloud, Datadog and New Relic is now tested against live accounts.
- Exports ride out short outages: a refused export is retried for about 15 seconds. For longer outages, the user manual (§4.10) shows how to put an OpenTelemetry Collector with a disk-backed queue in front.
- Generic OTLP export is now gzip-compressed, so less data leaves the site.
- Fixed: health tags no longer read as failing for 15 to 25 seconds after every gateway restart or settings save, so alarms on them no longer go off each time.
- Fixed: a destination that failed to start, or log export that stopped because its pseudonymisation key was missing, looked like a part that was switched off, so no alarm fired. New health tags Export/OTLP/FellBack, Export/Azure/FellBack and Export/LogsFailedClosed show it.
- Fixed: Azure Monitor no longer receives the full total of every counter as one spike after a settings save or restart, which set off false alerts.
- Fixed: a disabled OPC connection counted as faulted, and restarting the gateway while a database was down raised the fault alert a second time.
- Fixed: a stored setting that the new save checks would refuse made the gateway fall back to defaults after a restart. That dropped the Prometheus bearer token and served /metrics without authentication. The stored settings now load, and only the affected destination stays off.
- Settings saves are now checked, with an error under each field that is wrong. Pressing Save with nothing edited applies a secret rotated in its provider. On a redundant backup, a change is refused with a clear message to save on the master.
- Test Connection now says what is wrong: a host name that does not resolve, an unreachable proxy, or an untrusted certificate, instead of a generic failure.
- Fixed: an https:// endpoint named without a dot, such as a Docker or Kubernetes service name, failed to export while Test Connection said it would work.
- Fixed: a failed Microsoft Entra token refresh threw away a token that was still valid, so Azure rejected data until the next successful refresh.
Known issues
- On Ignition versions before 8.3.6, some curated metrics are missing because the gateway does not provide them (16 on 8.3.0, down to 9 on 8.3.4). Up to 8.3.3 this includes faulted database connections, so that alert stays silent. Use 8.3.6 or later for full coverage.
- The certificate metric reads the gateway's HTTPS listener on localhost. A gateway whose HTTPS port is bound to one specific network interface reports no certificate series.
- The Settings and Export Status pages use undocumented parts of the gateway web interface, tested on 8.3.6. A later 8.3.x update could break those pages. Export itself would keep working.
- Traces are manual only, through system.observability.span(). Ignition offers no way to trace HTTP requests, database queries or scripts automatically.
- Spans sent straight to Datadog get no APM trace metrics.
- Azure Monitor's OTLP ingestion is still a Microsoft preview.
- Proxy authentication and client certificates (mutual TLS) are not supported yet.
- The curated metrics do not yet cover SQL Bridge, EAM, Sepasoft or the remaining Cirrus modules.