I’m curious as to which tools and technologies you all are using to keep track of all those services you are deploying, whether it be resource tracking, network traffic, logs, traces, or uptime.
As a bonus question, how have you organized your network or your services reduce the overhead of implementing observability?
I asked this of the #homelab fediverse a few weeks back, as I was starting to deploy a family of apps to my “homelab” which is really a raspi3b with a repurposed spinny-platty laptop drive connected to some cloudflare tunnels. I settled on VictoriaLogs and VictoriaMetrics for the actual observing bits, fluentd for a log driver (I’m using docker-compose), and postgres-exporter, node-exporter, and handrolled metrics endpoints for prometheus metrics exposure (victoriametrics can do the metrics scraping itself).
If I were to change anything at the moment, I’d replace fluentd first, as some of the filters I use require the ruby interpreter and that seems a little heavy for my setup. Second I’d probably move to Grafana for dashboards. Alerting is gaping hole in the setup right now; I’ve used alertmanager in the past and not been particularly happy with it, but I don’t have a specific alternative I would suggest at the moment. If I had a much larger deployment, especially spread across multiple nodes, I’d consider ProxMox, but I’m just not there yet.
There were other suggestions that I didn’t go too deep into, especially after I got something working sufficiently, including dockhand, gatus and beszel, but so far (and to my surprise) having what I described above all running in containers on this little pi is doing the trick.
Thanks for sharing. Some good info here.