I’m thinking about just doing something outside kubernetes that just copies the data from the directory that NFS provides to another storage.

This is what I’m doing for the most part. A TrueNAS server provides the NFS shares and periodically backs them up with restic.

Some apps don’t like NFS very much, especially those that require SQLite. If you’re running Jellyfin over NFS you probably know what I mean. For those apps I use Ceph instead, which is highly available and a lot faster but also more complicated. Those PVCs I backup from within kubernetes to S3 storage with velero.

🤌🤌🤌

I use Promtail + Loki + Grafana to monitor application logs. Promtail scrapes logs, Loki stores and indexes them and Grafana can query Loki with LogQL and also send alerts.

Apparently Promtail is superseded by Grafana Alloy, which I don’t have experience with.

Anyway, I set this up mostly for fun and to preserve logs of terminated pods in my kubernetes cluster. I don’t have any alerts in place, but I probably could.

https://grafana.com/docs/loki/latest/