Description of problem: If velero has an invalid BackupStorageLocation(BSL) during startup, it will crashloop until the invalid BSL is removed. Version-Release number of selected component (if applicable): MTC 1.3, velero 1.4.2 How reproducible: Always. Steps to Reproduce: 1. created an invalid backupstoragelocation ``` $ cat <<EOF | oc create -f - apiVersion: velero.io/v1 kind: BackupStorageLocation metadata: name: bad-bsl namespace: openshift-migration spec: config: insecureSkipTLSVerify: "false" region: us-east-2 s3ForcePathStyle: "false" objectStorage: bucket: bucket-that-does-not-exist prefix: velero provider: aws EOF ``` 2. kill the velero pod Expected results: Expect the Velero pod to start with out error with atleast 1 valid BSL Actual Results: velero crashloops - `kubectl logs deployment/velero -n velero` ``` $ kubectl logs deployment/velero -n openshift-migration time="2020-09-22T21:19:21Z" level=info msg="setting log-level to INFO" logSource="pkg/cmd/server/server.go:177" time="2020-09-22T21:19:21Z" level=info msg="Starting Velero server v1.4.2-konveyor (-)" logSource="pkg/cmd/server/server.go:179" time="2020-09-22T21:19:21Z" level=info msg="No feature flags enabled" logSource="pkg/cmd/server/server.go:183" time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-remap-version time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pv time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pv-from-pvc time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pvc-from-pod time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-pvc-node-selector time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-storage-class time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/cluster-role-bindings time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-preserve-fields time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/job time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/restic time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/role-bindings time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-gcp kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/gcp time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-gcp kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/gcp time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-microsoft-azure kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/azure time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-microsoft-azure kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/azure time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/01-common-backup-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/02-pv-backup-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/03-is-backup-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/08-deploymentconfig-backup-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/13-deployment-backup-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/18-serviceaccount-backup-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/01-common-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/01-migcommon-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/01-namespace-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/02-serviceaccount-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/03-is-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/03-pv-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/03-pvc-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/04-imagestreamtag-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/05-route-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/06-build-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/07-pod-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/08-deploymentconfig-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/09-replicationcontroller-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/10-job-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/11-daemonset-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/12-replicaset-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/13-deployment-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/14-statefulset-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/15-service-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/16-cronjob-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/17-buildconfig-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/19-secret-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/20-scc-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/21-role-bindings-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/22-cluster-role-bindings-restore-plugin time="2020-09-22T21:19:21Z" level=info msg="Checking existence of namespace" logSource="pkg/cmd/server/server.go:361" namespace=openshift-migration time="2020-09-22T21:19:21Z" level=info msg="Namespace exists" logSource="pkg/cmd/server/server.go:367" namespace=openshift-migration time="2020-09-22T21:19:21Z" level=info msg="Checking existence of Velero custom resource definitions" logSource="pkg/cmd/server/server.go:396" time="2020-09-22T21:19:21Z" level=info msg="All Velero custom resource definitions exist" logSource="pkg/cmd/server/server.go:430" time="2020-09-22T21:19:21Z" level=info msg="Checking that all backup storage locations are valid" logSource="pkg/cmd/server/server.go:437" An error occurred: some backup storage locations are invalid: backup store for location "bad-bsl" is invalid: rpc error: code = Unknown desc = NoSuchBucket: The specified bucket does not exist status code: 404, request id: <id>, host id: <host-id> ``` - kubectl get backupstoragelocation -n openshift-migration ``` $ kubectl get backupstoragelocation -n openshift-migration NAME AGE bad-bsl 10m velero-4-vchdc 20h ``` Additional info: This is fixed in velero 1.5 https://github.com/vmware-tanzu/velero/issues/1967
@avital this is an issue that will be present until ~MTC 1.4.0 when we move to a newer version of Velero. This specific problem has impacted several customer already, would like to add this information to our docs, maybe in a known issue.
LGTM
Changes merged.