If no image is specified we will firstly look for tools:latest image stream in openshift namespace, if that fails only then we default to registry.redhat.io/rhel7/support-tools compare https://github.com/openshift/origin/blob/89b300063c6fd9b784c93dc6da2d5d00c754dce9/pkg/oc/cli/debug/debug.go#L756-L761 Finally, there's the option of explicitly specifying --image. Based on the above I'm closing this as not a bug.
(In reply to Maciej Szulik from comment #3) > If no image is specified we will firstly look for tools:latest image stream > in openshift namespace, if that fails only then we default to > registry.redhat.io/rhel7/support-tools > compare > https://github.com/openshift/origin/blob/ > 89b300063c6fd9b784c93dc6da2d5d00c754dce9/pkg/oc/cli/debug/debug.go#L756-L761 The problem here is that in a disconnect environment this registry is not acceptable. So unless this image is mirrored with the payload (or some other machanism), this will ALWAYS fail to pull. Until we have a good mirroring of samples (image stream content), like: `oc adm release mirror` for samples content and documented configuration changes to the samples, this will remain an issues. > > Finally, there's the option of explicitly specifying --image. This may be a workaround to this issue but it's not a solution. Should we tracking this as a bug for mirroring content?
> The problem here is that in a disconnect environment this registry is not acceptable. So unless this image is mirrored with the payload (or some other machanism), this will ALWAYS fail to pull. That Image Stream (tools:latest in openshift namespace) can be easily refined by cluster administrator so that it points to a local registry and then as explained earlier this will be the default image used for this invocation, in that case. > Until we have a good mirroring of samples (image stream content), like: `oc adm release mirror` for samples content and documented configuration changes to the samples, this will remain an issues. Mirroring images is already available, that particular image is not part of our release payload, though. I agree that we might want to document it better, but in that case that BZ should move to documentation team. Eric, do you agree?
Yes let's move this to docs, but they are going to need help defining a mirroring process for samples or RHCC images.
I believe standard functionality like `oc debug` should work as other standard features. Debugging, while not an everyday operation, is something needed at times. Why should we ask user that is already stressed enough by some problem to perform complicated actions before cluster can be debugged? That's why to me it makes more sense to have the image mirrored OOB. Is it hard to make it part of payload mirroring?
Can we default to the RHEL tools image an if it's not present, fall back to an images that is mirrored? Such as the oc cli image? We mirror that for must-gather.
Erich, that's exactly what we do today. If you don't specify any image, we default to imagestream tools:latest being present in openshift namespace. Only if that fails we reach out for support-tools from redhat's registry. So this is again a doc issue to make sure that tools:latest is properly mirrored.
Ok, do I understand correctly that users need to take special actions during installation? Or when? Who can describe the necessary steps?
*** Bug 1727849 has been marked as a duplicate of this bug. ***
Eduardo, as Maciej Szulik explained, the command automatically looks for `tools:latest` image stream in `openshift` namespace. So to resolve the issue, it seems to me that it is more straightforward to create such an image stream if it doesn't already exist. Ross Brattain figured out that multus image that is already available in cluster can be used instead of the unsupported tools image. So I think the KCS can describe how to create the necessary image stream. Presently to get the multus image I see this logic: 1. find pod with labels "app=multus" in "openshift-multus" project 2. get image pod was created from 3. create image stream `tools:latest` in `openshift` from that image I don't know what is the most straightforward way to do so. But it is an option to make things work without accessing external images.
Eduardo, I'd mention both --image flag and --image-stream flag, either is acceptable solution and worthy being explicitly mentioned in that article.
(In reply to Maciej Szulik from comment #24) > Eduardo, I'd mention both --image flag and --image-stream flag, either is > acceptable solution and worthy being explicitly mentioned in that article. The imagestream thing doesn't work for me. Those were my steps: * Configure OCP to trust the registry CA (https://docs.openshift.com/container-platform/4.4/registry/configuring-registry-operator.html#images-configuration-cas_configuring-registry-operator) $ echo -n | openssl s_client -connect provisioner.metal.e2e.local:5000 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > ca.crt $ oc create configmap registry-config --from-file=provisioner.metal.e2e.local=ca.crt -n openshift-config $ oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"registry-config"}}}' --type=merge # This is because this other BZ https://bugzilla.redhat.com/show_bug.cgi?id=1807471 $ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"managementState":"Managed"}}' $ oc patch configs.imageregistry.operator.openshift.io cluster --type merge --patch '{"spec":{"storage":{"emptyDir":{}}}}' * Create an imagestream such as: $ cat << EOF | oc apply -f - apiVersion: image.openshift.io/v1 kind: ImageStream metadata: name: tools namespace: openshift spec: tags: - name: latest importPolicy: scheduled: true from: kind: DockerImage name: provisioner.metal.e2e.local:5000/rhel7/support-tools:latest EOF Then: $ oc get is -n openshift tools -o yaml apiVersion: image.openshift.io/v1 kind: ImageStream metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"image.openshift.io/v1","kind":"ImageStream","metadata":{"annotations":{},"name":"tools","namespace":"openshift"},"spec":{"tags":[{"from":{"kind":"DockerImage","name":"provisioner.metal.e2e.local:5000/rhel7/support-tools:latest"},"importPolicy":{"scheduled":true},"name":"latest"}]}} openshift.io/image.dockerRepositoryCheck: "2020-06-24T15:02:25Z" creationTimestamp: "2020-06-24T15:00:39Z" generation: 2 name: tools namespace: openshift resourceVersion: "457098" selfLink: /apis/image.openshift.io/v1/namespaces/openshift/imagestreams/tools uid: 3eec61bb-82f5-4f1c-ac15-89bad44ee56a spec: lookupPolicy: local: false tags: - annotations: null from: kind: DockerImage name: provisioner.metal.e2e.local:5000/rhel7/support-tools:latest generation: 2 importPolicy: scheduled: true name: latest referencePolicy: type: Source status: dockerImageRepository: "" tags: - items: - created: "2020-06-24T15:02:25Z" dockerImageReference: provisioner.metal.e2e.local:5000/rhel7/support-tools@sha256:72e2c445dfd27efc68dd0e15787278517c735ff7599419746a36373f9b74cd5b generation: 2 image: sha256:72e2c445dfd27efc68dd0e15787278517c735ff7599419746a36373f9b74cd5b tag: latest * Ensure the image is not present in the host (crictl images | grep -i tools = empty) * oc -v=9 debug node/master-2 2> logs.txt See logs.txt attached. Also, if I try to specify an imagestream with --image-stream like you said: $ oc debug --image-stream=tools node/master-2 Error: unknown flag: --image-stream See 'oc debug --help' for usage. Just in case: $ oc version Client Version: 4.4.6 Server Version: 4.4.6 Kubernetes Version: v1.17.1+f63db30
Created attachment 1698624 [details] "oc -v=9 debug node/master-2" output when trying to use an imagestream
I forgot to mention that the provisioner.metal.e2e.local:5000/rhel7/support-tools:latest image was pushed into the registry as: export MYREGISTRY='provisioner.metal.e2e.local:5000' IMAGEID=$(podman pull registry.redhat.io/rhel7/support-tools:latest 2>/dev/null) podman login ${MYREGISTRY} podman push ${IMAGEID} ${MYREGISTRY}/rhel7/support-tools:latest And if I pull it manually or if I use the --image flag, it works.
Interesting, I'll verify image stream, if this is working both through the logs and locally with newer versions and report back.
oc tag -d openshift/tools:latest oc tag -n openshift $(oc get pods -n openshift-multus -l app=multus -o jsonpath='{.items[0].spec.containers[?(@.name=="kube-multus")].image}') tools:latest oc get imagetag -n openshift tools:latest seems to work
While using KCS as a workaround is fine, in the short term. Making the tools image part of the payload for easy mirroring (or as an optional mirrored artifact - in the mirror tooling/documentation) should be the focus of this bug. What traction / movement are we seeing on that front?
(In reply to Eric Rich from comment #35) > While using KCS as a workaround is fine, in the short term. Making the tools > image part of the payload for easy mirroring (or as an optional mirrored > artifact - in the mirror tooling/documentation) should be the focus of this > bug. I completely agree, documenting both --image and the need to mirror tools is necessary for completing this BZ.