Description of problem: Running the downstream must-gather image on my OCP4 cluster hangs and produces a number of errors. The command: oc adm must-gather --image=registry.redhat.io/rhcam-1-2/openshift-migration-must-gather-rhel8 --dest-dir=. Sample of errors I'm able to get (I can't seem to get the whole log): [must-gather-9jdm7] POD openshift-infra [must-gather-9jdm7] POD openshift-logging [must-gather-9jdm7] POD openshift-metrics-server [must-gather-9jdm7] POD openshift-migration [must-gather-9jdm7] POD openshift-monitoring [must-gather-9jdm7] POD openshift-node [must-gather-9jdm7] POD openshift-sdn [must-gather-9jdm7] POD openshift-storage [must-gather-9jdm7] POD openshift-template-service-broker [must-gather-9jdm7] POD openshift-web-console [must-gather-9jdm7] POD pvc-migrate-bmark [must-gather-9jdm7] POD [must-gather-9jdm7] POD Using project "". [must-gather-9jdm7] POD Error from server (BadRequest): previous terminated container "cam" in pod "migration-controller-79c9b4fd7b-lqf6d" not found [must-gather-9jdm7] POD Error from server (BadRequest): previous terminated container "ansible" in pod "migration-operator-5967c7d5fc-qrmmx" not found [must-gather-9jdm7] POD Error from server (BadRequest): previous terminated container "mig-ui" in pod "migration-ui-79fcc7fdbc-mkpfq" not found [must-gather-9jdm7] POD Error from server (BadRequest): previous terminated container "setup-certificate-secret" in pod "restic-4cmfw" not found [must-gather-9jdm7] POD Error from server (BadRequest): previous terminated container "setup-certificate-secret" in pod "restic-lxpcj" not found [must-gather-9jdm7] POD Error from server (BadRequest): previous terminated container "setup-certificate-secret" in pod "restic-sjrx2" not found [must-gather-9jdm7] POD Error from server (BadRequest): previous terminated container "velero-plugin" in pod "velero-854749986b-2jshw" not found [must-gather-9jdm7] POD An error occurred: timed out waiting for download URL [must-gather-9jdm7] POD command terminated with exit code 1 [must-gather-9jdm7] POD An error occurred: timed out waiting for download URL [must-gather-9jdm7] POD command terminated with exit code 1 [must-gather-9jdm7] POD An error occurred: timed out waiting for download URL [must-gather-9jdm7] POD command terminated with exit code 1
Hi, based on details from the customer ticket and experiments on a dev cluster, I would: 1. Recommend CU to extend timeout for must-gather feature (600s by default) e.g. to 3600s (add parameter --timeout=3600 to the must-gather command) 2. Look deeper on listing cluster loop [1] which seems to duplicate data dump operations in case of multiple non-host clusters discovered. And provide optimization PR as a fix of this BZ. [1] https://github.com/konveyor/must-gather/blob/master/collection-scripts/gather#L4-L26 which
After looking deeper into this issue, we saw some opportunities for performance and logging enhancements to konveyor must-gather. These improvements have been merged to master and release-1.3.0 branches. These changes should included in the next round of 1.3.0 builds https://github.com/konveyor/must-gather/pull/11
Verified using MTC 1.3 openshift-migration-rhel7-operator@sha256:66efea27fa3d6498ef8c722ef9dec45ceba2a9db695b8092e0e65b5070c94d87 After testing several platforms and versions (testruns for 1.3), we have not seen this problem in our testruns. We move this issue to VERIFIED status.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Migration Toolkit for Containers (MTC) Tool image release advisory 1.3.0), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4148