Bug 1809614

This is a generic problem with `oc adm must-gather` in disconnected environments. See these docs bugs https://bugzilla.redhat.com/show_bug.cgi?id=1771435

If anything were to change in the product it'd either be on the `oc adm mirror` or `oc adm must-gather` side of things, definitely not the installer so I'm moving this to oc component.

I mirrored the must-gather image to the disconnected registry that I used for initial deployment:

oc image mirror quay.io/openshift/origin-must-gather:latest registry.ocp-edge-cluster.qe.lab.redhat.com:5000/openshift/origin-must-gather:latest

but when I run oc adm must-gather it gets stuck:


oc adm must-gather --image registry.ocp-edge-cluster.qe.lab.redhat.com:5000/openshift/origin-must-gather:latest
[must-gather      ] OUT Using must-gather plugin-in image: registry.ocp-edge-cluster.qe.lab.redhat.com:5000/openshift/origin-must-gather:latest
[must-gather      ] OUT namespace/openshift-must-gather-8hq7j created
[must-gather      ] OUT clusterrolebinding.rbac.authorization.k8s.io/must-gather-lldv8 created
[must-gather      ] OUT pod for plug-in image registry.ocp-edge-cluster.qe.lab.redhat.com:5000/openshift/origin-must-gather:latest created

pod is stuck in Init state:


openshift-must-gather-8hq7j                             must-gather-k626f                                                      0/1     Init:0/1            0          13s


[kni@provisionhost-0 ~]$ oc -n openshift-must-gather-8hq7j get pods must-gather-k626f -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks-status: ""
  creationTimestamp: "2020-03-10T18:50:34Z"
  generateName: must-gather-
  labels:
    app: must-gather
  name: must-gather-k626f
  namespace: openshift-must-gather-8hq7j
  resourceVersion: "242963"
  selfLink: /api/v1/namespaces/openshift-must-gather-8hq7j/pods/must-gather-k626f
  uid: a8c09f54-72cd-423e-949e-f16b1da35b56
spec:
  containers:
  - command:
    - /bin/bash
    - -c
    - 'trap : TERM INT; sleep infinity & wait'
    image: registry.ocp-edge-cluster.qe.lab.redhat.com:5000/openshift/origin-must-gather:latest
    imagePullPolicy: Always
    name: copy
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /must-gather
      name: must-gather-output
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-gl9wz
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: default-dockercfg-sjk9h
  initContainers:
  - command:
    - /usr/bin/gather
    image: registry.ocp-edge-cluster.qe.lab.redhat.com:5000/openshift/origin-must-gather:latest
    imagePullPolicy: Always
    name: gather
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /must-gather
      name: must-gather-output
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-gl9wz
      readOnly: true
  nodeName: master-2.ocp-edge-cluster.qe.lab.redhat.com
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 0
  tolerations:
  - operator: Exists
  volumes:
  - emptyDir: {}
    name: must-gather-output
  - name: default-token-gl9wz
    secret:
      defaultMode: 420
      secretName: default-token-gl9wz
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-03-10T18:50:34Z"
    message: 'containers with incomplete status: [gather]'
    reason: ContainersNotInitialized
    status: "False"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-03-10T18:50:34Z"
    message: 'containers with unready status: [copy]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-03-10T18:50:34Z"
    message: 'containers with unready status: [copy]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-03-10T18:50:34Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: registry.ocp-edge-cluster.qe.lab.redhat.com:5000/openshift/origin-must-gather:latest
    imageID: ""
    lastState: {}
    name: copy
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        reason: PodInitializing
  hostIP: fd2e:6f44:5dd8:c956::107
  initContainerStatuses:
  - image: registry.ocp-edge-cluster.qe.lab.redhat.com:5000/openshift/origin-must-gather:latest
    imageID: ""
    lastState: {}
    name: gather
    ready: false
    restartCount: 0
    state:
      waiting:
        reason: PodInitializing
  phase: Pending
  qosClass: BestEffort
  startTime: "2020-03-10T18:50:34Z"

The issue in previous comment seems to be caused by another BZ getting the cluster in a broken state. I could run `oc adm must-gather --image registry.ocp-edge-cluster.qe.lab.redhat.com:5000/openshift/origin-must-gather:latest` against a healthy cluster.