Bug 1873647

Summary: Downstream must-gather image hangs and never produces output with numerous errors
Product: OpenShift Container Platform Reporter: Erik Nelson <ernelson>
Component: Migration ToolingAssignee: Derek Whatley <dwhatley>
Status: CLOSED ERRATA QA Contact: Xin jiang <xjiang>
Severity: low Docs Contact:
Priority: unspecified    
Version: 4.5CC: dwhatley, maufart, mberube, rjohnson, sregidor
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-09-30 18:42:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Erik Nelson 2020-08-28 21:06:58 UTC
Description of problem:
Running the downstream must-gather image on my OCP4 cluster hangs and produces a number of errors. The command:

oc adm must-gather --image=registry.redhat.io/rhcam-1-2/openshift-migration-must-gather-rhel8 --dest-dir=.

Sample of errors I'm able to get (I can't seem to get the whole log):

[must-gather-9jdm7] POD     openshift-infra
[must-gather-9jdm7] POD     openshift-logging
[must-gather-9jdm7] POD     openshift-metrics-server
[must-gather-9jdm7] POD     openshift-migration
[must-gather-9jdm7] POD     openshift-monitoring
[must-gather-9jdm7] POD     openshift-node
[must-gather-9jdm7] POD     openshift-sdn
[must-gather-9jdm7] POD     openshift-storage
[must-gather-9jdm7] POD     openshift-template-service-broker
[must-gather-9jdm7] POD     openshift-web-console
[must-gather-9jdm7] POD     pvc-migrate-bmark
[must-gather-9jdm7] POD 
[must-gather-9jdm7] POD Using project "".
[must-gather-9jdm7] POD Error from server (BadRequest): previous terminated container "cam" in pod "migration-controller-79c9b4fd7b-lqf6d" not found
[must-gather-9jdm7] POD Error from server (BadRequest): previous terminated container "ansible" in pod "migration-operator-5967c7d5fc-qrmmx" not found
[must-gather-9jdm7] POD Error from server (BadRequest): previous terminated container "mig-ui" in pod "migration-ui-79fcc7fdbc-mkpfq" not found
[must-gather-9jdm7] POD Error from server (BadRequest): previous terminated container "setup-certificate-secret" in pod "restic-4cmfw" not found
[must-gather-9jdm7] POD Error from server (BadRequest): previous terminated container "setup-certificate-secret" in pod "restic-lxpcj" not found
[must-gather-9jdm7] POD Error from server (BadRequest): previous terminated container "setup-certificate-secret" in pod "restic-sjrx2" not found
[must-gather-9jdm7] POD Error from server (BadRequest): previous terminated container "velero-plugin" in pod "velero-854749986b-2jshw" not found
[must-gather-9jdm7] POD An error occurred: timed out waiting for download URL
[must-gather-9jdm7] POD command terminated with exit code 1
[must-gather-9jdm7] POD An error occurred: timed out waiting for download URL
[must-gather-9jdm7] POD command terminated with exit code 1
[must-gather-9jdm7] POD An error occurred: timed out waiting for download URL
[must-gather-9jdm7] POD command terminated with exit code 1

Comment 1 Marek Aufart 2020-08-31 13:38:58 UTC
Hi, based on details from the customer ticket and experiments on a dev cluster, I would:

1. Recommend CU to extend timeout for must-gather feature (600s by default) e.g. to 3600s (add parameter --timeout=3600 to the must-gather command)

2. Look deeper on listing cluster loop [1] which seems to duplicate data dump operations in case of multiple non-host clusters discovered. And provide optimization PR as a fix of this BZ.


[1] https://github.com/konveyor/must-gather/blob/master/collection-scripts/gather#L4-L26 which

Comment 3 Derek Whatley 2020-09-17 21:46:26 UTC
After looking deeper into this issue, we saw some opportunities for performance and logging enhancements to konveyor must-gather. 

These improvements have been merged to master and release-1.3.0 branches. These changes should included in the next round of 1.3.0 builds
https://github.com/konveyor/must-gather/pull/11

Comment 7 Sergio 2020-09-22 15:52:41 UTC
Verified using MTC 1.3

openshift-migration-rhel7-operator@sha256:66efea27fa3d6498ef8c722ef9dec45ceba2a9db695b8092e0e65b5070c94d87


After testing several platforms and versions (testruns for 1.3), we have not seen this problem in our testruns.

We move this issue to VERIFIED status.

Comment 11 errata-xmlrpc 2020-09-30 18:42:39 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Migration Toolkit for Containers (MTC) Tool image release advisory 1.3.0), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4148