Bug 1881707 - [Doc] Add a known issue of: velero crashloops with invalid BSL on startup
Summary: [Doc] Add a known issue of: velero crashloops with invalid BSL on startup
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Migration Toolkit for Containers
Classification: Red Hat
Component: Documentation
Version: 1.3.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 1.3.0
Assignee: Avital Pinnick
QA Contact: Xin jiang
Avital Pinnick
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-09-22 22:23 UTC by Alay Patel
Modified: 2020-09-24 10:01 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-09-24 10:01:29 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Alay Patel 2020-09-22 22:23:15 UTC
Description of problem:

If velero has an invalid BackupStorageLocation(BSL) during startup, it will crashloop until the invalid BSL is removed.



Version-Release number of selected component (if applicable):
MTC 1.3, velero 1.4.2


How reproducible:

Always.


Steps to Reproduce:

1. created an invalid backupstoragelocation
```
$ cat <<EOF | oc create -f -
apiVersion: velero.io/v1
kind: BackupStorageLocation
metadata:
  name: bad-bsl
  namespace: openshift-migration
spec:
  config:
    insecureSkipTLSVerify: "false"
    region: us-east-2
    s3ForcePathStyle: "false"
  objectStorage:
    bucket:  bucket-that-does-not-exist
    prefix: velero
  provider: aws
EOF
```
2. kill the velero pod


Expected results:

Expect the Velero pod to start with out error with atleast 1 valid BSL

Actual Results:
velero crashloops

- `kubectl logs deployment/velero -n velero`
```
$ kubectl logs deployment/velero -n openshift-migration
time="2020-09-22T21:19:21Z" level=info msg="setting log-level to INFO" logSource="pkg/cmd/server/server.go:177"
time="2020-09-22T21:19:21Z" level=info msg="Starting Velero server v1.4.2-konveyor (-)" logSource="pkg/cmd/server/server.go:179"
time="2020-09-22T21:19:21Z" level=info msg="No feature flags enabled" logSource="pkg/cmd/server/server.go:183"
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-remap-version
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pv
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pv-from-pvc
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/add-pvc-from-pod
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-pvc-node-selector
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/change-storage-class
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/cluster-role-bindings
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/crd-preserve-fields
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/job
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/pod
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/restic
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/role-bindings
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/velero kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/service-account
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-aws kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/aws
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-gcp kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/gcp
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-gcp kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/gcp
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-microsoft-azure kind=VolumeSnapshotter logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/azure
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugin-for-microsoft-azure kind=ObjectStore logSource="pkg/plugin/clientmgmt/registry.go:100" name=velero.io/azure
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/01-common-backup-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/02-pv-backup-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/03-is-backup-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/08-deploymentconfig-backup-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/13-deployment-backup-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=BackupItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/18-serviceaccount-backup-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/01-common-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/01-migcommon-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/01-namespace-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/02-serviceaccount-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/03-is-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/03-pv-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/03-pvc-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/04-imagestreamtag-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/05-route-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/06-build-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/07-pod-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/08-deploymentconfig-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/09-replicationcontroller-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/10-job-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/11-daemonset-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/12-replicaset-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/13-deployment-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/14-statefulset-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/15-service-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/16-cronjob-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/17-buildconfig-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/19-secret-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/20-scc-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/21-role-bindings-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="registering plugin" command=/plugins/velero-plugins kind=RestoreItemAction logSource="pkg/plugin/clientmgmt/registry.go:100" name=openshift.io/22-cluster-role-bindings-restore-plugin
time="2020-09-22T21:19:21Z" level=info msg="Checking existence of namespace" logSource="pkg/cmd/server/server.go:361" namespace=openshift-migration
time="2020-09-22T21:19:21Z" level=info msg="Namespace exists" logSource="pkg/cmd/server/server.go:367" namespace=openshift-migration
time="2020-09-22T21:19:21Z" level=info msg="Checking existence of Velero custom resource definitions" logSource="pkg/cmd/server/server.go:396"
time="2020-09-22T21:19:21Z" level=info msg="All Velero custom resource definitions exist" logSource="pkg/cmd/server/server.go:430"
time="2020-09-22T21:19:21Z" level=info msg="Checking that all backup storage locations are valid" logSource="pkg/cmd/server/server.go:437"
An error occurred: some backup storage locations are invalid: backup store for location "bad-bsl" is invalid: rpc error: code = Unknown desc = NoSuchBucket: The specified bucket does not exist
        status code: 404, request id: <id>, host id: <host-id>
```

- kubectl get backupstoragelocation -n openshift-migration

```
$ kubectl get backupstoragelocation -n openshift-migration
NAME             AGE
bad-bsl          10m
velero-4-vchdc   20h
```


Additional info:
This is fixed in velero 1.5 https://github.com/vmware-tanzu/velero/issues/1967

Comment 1 John Matthews 2020-09-22 22:36:35 UTC
@avital this is an issue that will be present until ~MTC 1.4.0 when we move to a newer version of Velero.

This specific problem has impacted several customer already, would like to add this information to our docs, maybe in a known issue.

Comment 3 Xin jiang 2020-09-23 09:00:37 UTC
LGTM

Comment 4 Avital Pinnick 2020-09-24 10:01:29 UTC
Changes merged.


Note You need to log in before you can comment on or make changes to this bug.