The backup server has been offline for some time. I'm wondering if it's worth resetting it entirely and bringing up a new machine with a good strategy to figure out what needs to be backed up and having the backups set in Ansible (if it isn't already) At some point, we should establish the process of restoring something from backup and test it regularly. For example, restoring Jenkins from backup is going to need some documentation to get right. Perhaps it's even worth automating a prod -> staging push every week so we can test things better.
I did install another backup server a few month ago: https://github.com/gluster/gluster.org_ansible_configuration/commit/40202aa41a732a41d68c2c36df498eca089b7a9e We have doc awaiting some reviews too: https://github.com/gluster/infra-docs/pull/23
This is now back online.