I’m curious as to which tools and technologies you all are using to keep track of all those services you are deploying, whether it be resource tracking, network traffic, logs, traces, or uptime.
As a bonus question, how have you organized your network or your services to reduce the overhead of implementing observability?
Bezsel, simple and easy, constantly having new features added.
I asked this of the #homelab fediverse a few weeks back, as I was starting to deploy a family of apps to my “homelab” which is really a raspi3b with a repurposed spinny-platty laptop drive connected to some cloudflare tunnels. I settled on VictoriaLogs and VictoriaMetrics for the actual observing bits, fluentd for a log driver (I’m using docker-compose), and postgres-exporter, node-exporter, and handrolled metrics endpoints for prometheus metrics exposure (victoriametrics can do the metrics scraping itself).
If I were to change anything at the moment, I’d replace fluentd first, as some of the filters I use require the ruby interpreter and that seems a little heavy for my setup. Second I’d probably move to Grafana for dashboards. Alerting is gaping hole in the setup right now; I’ve used alertmanager in the past and not been particularly happy with it, but I don’t have a specific alternative I would suggest at the moment. If I had a much larger deployment, especially spread across multiple nodes, I’d consider ProxMox, but I’m just not there yet.
There were other suggestions that I didn’t go too deep into, especially after I got something working sufficiently, including dockhand, gatus and beszel, but so far (and to my surprise) having what I described above all running in containers on this little pi is doing the trick.
I’d second this. Victoria metrics and Victoria logs is orders of magnitude better than Prometheus and Loki. However, I’d switch fluent for Vector for logs collection.
Ah good to know re:vector. When I have some time I’ll give that a look.
Uptime Kuma if I want something to alert me (still trying to decide between ntfy & gotify)
Home Assistant is also checking some stuff
Watchdog to reboot my Raspberry Pi Zeros when they fall off the wifi
Smokeping for a general, long term feel of the network, which might answer your bonus question?
Did you ever decide on Ntfy vs Gotify?
Not yet 🙂
Both services work well, but the (Android) app is also a consideration… Gotify appears to keep losing the connection, but as I have different alerts going via different routes I need to sit down and spend some quality time with these, which I don’t have at the moment.
If you’re wanting to go hard in the paint here, Grafana has a great OSS stack with Grafana, Loki, Mimir, Tempo, and Alloy.
