Most of the time, you hear about an outage from a customer: “the site is down.” Then it turns out the database server’s disk filled up yesterday, a replica was an hour behind, and the log line with the error sat on a third machine nobody looks at. The data was all there. Nobody was collecting it in one place.

The opposite happens too. There is monitoring, but it’s five dashboards with five passwords, and the alerts chat has been muted for months because it gets a hundred messages a day. That kind of monitoring only gives you the feeling that things are under control.

We set up a compact stack that answers three questions: what’s happening inside your servers, what your services are writing to their logs, and whether your users can actually reach you. One entry point, one login, and alerts only for things that need someone to act.

What we build

  • Prometheus collects metrics from everything you run. Every server, VM and LXC container gets node_exporter: disks, memory, load, network, systemd state.
  • pve-exporter looks at Proxmox from the hypervisor side: status and usage of all VMs and containers, including stopped ones. It talks to the API with a read-only token.
  • Databases: exporters for PostgreSQL and MariaDB, plus the built-in metrics of Patroni, etcd, HAProxy and ProxySQL. You see who the primary is right now, how far replicas lag, and how many Galera nodes are in sync.
  • Alertmanager groups alerts and routes them to Telegram, email or your team chat.
  • Loki, with Promtail or Alloy as the agent, collects logs from every machine. You search them in the same Grafana, next to the graphs.
  • Uptime Kuma checks sites, APIs and ports from the outside, watches TLS certificate expiry and gives you a public status page for your customers.
  • Grafana puts it all on dashboards built for your infrastructure.
  • Traefik sits in front as the reverse proxy and handles Let’s Encrypt certificates on its own.
  • Authentik gives you one login with 2FA for Grafana, Uptime Kuma and internal panels. Tools with no authentication of their own get covered by Traefik’s forwardAuth.

We deploy with docker compose or into dedicated LXC containers. Configs, alert rules and dashboards live in git, so the whole stack can be rebuilt from scratch.

Making alerts something people read

The biggest risk here isn’t technical. If an alert fires and nobody needs to do anything, within a week people stop opening them. So every rule gets one question: what does someone do when this fires? If there’s no answer, it’s not an alert. It’s a graph on a dashboard.

In practice that means a few things. Every alert has a for: clause, so a short spike doesn’t wake anyone at night. Disk usage is alerted on the trend, not a fixed “90%” threshold. Dead and expired services are removed from checks, otherwise they stay red forever and teach everyone that red means nothing. A rule looks roughly like this:

groups:
  - name: infra
    rules:
      - alert: DiskWillFillIn24h
        expr: predict_linear(node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}[6h], 24*3600) < 0
        for: 30m
        labels: {severity: warning}
        annotations:
          summary: "{{ $labels.host }}: {{ $labels.mountpoint }} will be full within a day"
      - alert: GaleraNodeLost
        expr: mysql_global_status_wsrep_cluster_size < 3
        for: 2m
        labels: {severity: critical}

Dashboards are built around specific questions, not “everything about everything.” For the daily look, one screen: the hypervisor, storage, which guests use the most, who’s running out of space, who has no backup. Database clusters need a different view: node role and Galera state are drawn as bands on a timeline, so a failover is obvious at a glance, the band simply moves to another node. Large dashboards are generated by a script, and after each change we check that no panel comes back empty.

A couple of things worth knowing up front. Inside LXC, node_exporter partly sees the host’s metrics because the kernel is shared, so real container usage comes from pve-exporter. Machines are labelled by name in Prometheus, not by IP, so nobody has to remember addresses in Grafana.

Trade-offs, and when this isn’t the right fit

Loki is not Elasticsearch. It only indexes labels (host, service, level) and scans the text at query time. That makes it light and cheap to store, but full-text search across months of logs will be slow. If you need log analytics or long retention for audits, ELK or OpenSearch is the better tool, on a different scale of hardware.

Monitoring has to live apart from what it monitors. If Prometheus runs on the same server as production, you won’t even get an alert when that server goes down. At minimum, a separate VM or LXC. Uptime Kuma is best placed in another location or with another provider, since it’s your view through your users’ eyes. We also set up a watchdog alert that should always be firing: if it goes quiet, the notification chain itself is broken.

Authentik becomes the front door. If it’s down, nobody gets into the panels. So its database is backed up before every upgrade, and after upgrading we check not just that the page loads but where the login redirect actually points. Emergency access to the servers stays separate and doesn’t depend on SSO.

For a single small VPS the full stack is overkill: Uptime Kuma and a few alerts will do. It starts paying off with several servers, or wherever you run replicated databases.

What you get

  • A working Prometheus, Alertmanager, Grafana, Loki and Uptime Kuma stack behind Traefik, over HTTPS.
  • One login with 2FA for every panel, with accounts for your team.
  • Metrics for all servers, VMs/LXC and databases, including replication, Patroni or Galera.
  • Dashboards made for your infrastructure, not generic imports.
  • Alerts to Telegram, email or chat, each with a note on what it means and what to do.
  • A status page for your customers, if you want one.
  • The entire stack in git: configs, rules, dashboards, plus docs on adding a new server or check.

If you already run a few servers and still learn about problems from your customers, get in touch. Tell us what runs where, and we’ll suggest how to watch it all from one place.