Skip to content

Centralize netdata monitoring on nuc-mini

jedmund requested to merge jedmund/jedmund/netdata-monitoring into main

Created by: jedmund

Summary

  • New monitoring role installs netdata on every host and configures parent/child streaming so all metrics aggregate to nuc-mini with 30-day retention.
  • Adds https://netdata.atelier.house (TinyAuth-protected dashboard) and https://netdata.atelier.house/api/* (no-auth bypass) so the LINE dashboard backend can pull metrics for widgets.
  • Discovers and fixes a long-standing ddclient bug: the compose mount path didn't match what linuxserver/ddclient actually reads, so DNS records have only been working because they were added in Cloudflare manually.

Topology

  • Parent (nuc-mini) receives streamed metrics from children, dbengine retention configured for ~30 days at minute resolution via tier 1.
  • Children (max, rack-mini) stream to nuc-mini and keep ~1 day of local retention to buffer through parent outages.
  • All three nodes appear in http://nuc-mini:19999/api/v2/nodes once running.

Notes

  • Streaming API key lives in vault_netdata_stream_api_key in the compute_servers vault. Generate with uuidgen if you don't have one set.
  • The Linux handler uses systemctl restart --no-block + wait_for because netdata's dbengine reconfig takes longer than systemd's D-Bus method timeout, which would otherwise fail every restart even when netdata succeeds.
  • For the macOS host (rack-mini), brew-installed netdata needs System Settings → Privacy & Security → Local Network permission turned on. Without it, the daemon fails LAN connections silently with errno 65 No route to host even though nc/curl work fine. Can't be automated.
  • ddclient's cloudflare protocol only updates existing A records. The netdata.atelier.house A record needs to be created once in Cloudflare before ddclient picks it up.

Test plan

  • make deploy-prerequisites — installs netdata on all three hosts, configures streaming
  • curl -s http://nuc-mini:19999/api/v2/nodes returns nuc-mini, atelier-max, rack-mini
  • make deploy-infra-gateway — applies new Traefik routes and ddclient mount fix
  • ddclient actively pushing updates to Cloudflare (verified in container logs)
  • After Cloudflare A record is created: https://netdata.atelier.house redirects to TinyAuth, https://netdata.atelier.house/api/v2/nodes returns the unauthenticated node list

Merge request reports