Bug 1970102

Summary: Update 'mig-log-reader' filters to reduce noise caused by Velero 1.6 log strings
Product: Migration Toolkit for Containers Reporter: Derek Whatley <dwhatley>
Component: ControllerAssignee: Derek Whatley <dwhatley>
Status: CLOSED ERRATA QA Contact: Xin jiang <xjiang>
Severity: unspecified Docs Contact: Avital Pinnick <apinnick>
Priority: unspecified    
Version: 1.5.0CC: ernelson, rjohnson, sregidor
Target Milestone: ---   
Target Release: 1.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-28 04:08:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Derek Whatley 2021-06-09 19:54:17 UTC
Description of problem:
See title.

Examples of new noisy log messages that dominate logs. This is not exahustive. Would be good to run through a migration and decide on a new list of log messages to filter.

openshift-migration velero-77cd56899c-qkdnc velero time="2021-06-09T19:52:51Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=aws-s3-mb5d8 controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121"
openshift-migration velero-77cd56899c-qkdnc velero time="2021-06-09T19:52:51Z" level=info msg="Validating backup storage location" backup-storage-location=aws-s3-dlzjc controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:114"
openshift-migration velero-77cd56899c-qkdnc velero time="2021-06-09T19:52:51Z" level=info msg="Backup storage location valid, marking as available" backup-storage-location=aws-s3-dlzjc controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:121"
openshift-migration velero-77cd56899c-qkdnc velero time="2021-06-09T19:52:51Z" level=warning msg="There is no existing backup storage location set as default. Please see `velero backup-location -h` for options." controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:173"
openshift-migration velero-77cd56899c-qkdnc velero time="2021-06-09T19:52:51Z" level=warning msg="There is no existing backup storage location set as default. Please see `velero backup-location -h` for options." controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:173"
openshift-migration velero-77cd56899c-qkdnc velero time="2021-06-09T19:52:51Z" level=warning msg="There is no existing backup storage location set as default. Please see `velero backup-location -h` for options." controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:173"
openshift-migration velero-77cd56899c-qkdnc velero time="2021-06-09T19:52:51Z" level=warning msg="There is no existing backup storage location set as default. Please see `velero backup-location -h` for options." controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:173"
openshift-migration velero-77cd56899c-qkdnc velero time="2021-06-09T19:52:51Z" level=warning msg="There is no existing backup storage location set as default. Please see `velero backup-location -h` for options." controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:173"
openshift-migration velero-77cd56899c-qkdnc velero time="2021-06-09T19:52:51Z" level=warning msg="There is no existing backup storage location set as default. Please see `velero backup-location -h` for options." controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:173"


Exclusions need to be updated in https://github.com/konveyor/mig-log-reader/blob/master/stern.sh

Comment 5 Sergio 2021-06-25 13:52:02 UTC
Verified using MTC 1.5.0

    openshift-migration-rhel7-operator@sha256:00e77706ca22bcb557d13c16822180fc877e6ea1639a72fda8eb9f5488b039a2
    - name: MIG_CONTROLLER_REPO
      value: openshift-migration-controller-rhel8@sha256
    - name: MIG_CONTROLLER_TAG
      value: 7f657df15e9514df4ef42da3431f558a19b8d3233a2ef1222cd8e27793c93816
    - name: MIG_UI_REPO
      value: openshift-migration-ui-rhel8@sha256
    - name: MIG_UI_TAG
      value: bda1959e1f44170905499b304ce28f9f95ee1841b0dc29f9a9e2790d0dd0bd71
    - name: MIG_LOG_READER_REPO
      value: openshift-migration-log-reader-rhel8@sha256
    - name: MIG_LOG_READER_TAG
      value: 5b89aaf00f8a660905c69ea41cc9ed2c7822f20f02a3b85ff3c4b008b8b5ced4


We can find the excluded texts in the stern.sh file

$ oc exec migration-log-reader-58dc94fbf8-47hhd cat /usr/bin/stern.sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
Defaulting container name to color.
Use 'oc describe pod/migration-log-reader-58dc94fbf8-47hhd -n openshift-migration' to see all of the containers in this pod.
#!/bin/bash
CONFIGPATH=/var/cache/sa2kubeconfig/kubeconfig-${KSUFFIX}

stern -l app.kubernetes.io/part-of=openshift-migration \
--color ${STERN_COLOR} \
--exclude-container discovery \
--exclude "watch is too old"  \
--exclude "Found new dockercfg secret" \
--exclude "specified default backup location" \
--exclude "needs to be at least 1 backup location" \
--exclude "No backup locations were ready to be verified" \
--exclude "Backup cannot be garbage-collected" \
--exclude "Error checking repository for stale locks" \
--exclude "Backup has expired" \
--exclude "error getting backup storage location" \
--exclude "There is no existing backup storage location set as default" \
--since 5s \
--all-namespaces \
--kubeconfig ${CONFIGPATH} \



We can find the log message in the velero pod, but we cannot find it in the migration-logs pod. So it is effectively excluded

$ oc logs velero-5bb4644844-wkm2x  | grep "location set as default" | wc -l
382
$ oc logs migration-log-reader-58dc94fbf8-47hhd -c plain |grep "location set as default" | wc -l
0

Moved to VERIFIED status.

Comment 11 errata-xmlrpc 2021-07-28 04:08:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Migration Toolkit for Containers (MTC) image release advisory 1.5.0), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:2929