oc adm must-gather has 4 options for retrieving must-gather image: 1. read image(s) from --image flag 2. read image stream present on the cluster from --image-stream flag 3. lookup built-in must-gather:latest image stream in openshift namespace 4. if all of the above fails only then it will reach out to hardcoded quay.io URL.
Linking previous discussion, although personally I'd like to see 'oc' automatically check for the must-gather pullspec referenced from the currently-running release image. Which it already has access to via: $ oc adm release info --image-for=must-gather
Hmm... I wasn't aware of that, that's reasonable to do, lemme re-open this bug and let's do it the way Trevor proposes.
Awkward caveat: if 'oc' doesn't have access to the same registry as the cluster (e.g. because you have some special tunnel set up for the Kube API that does not include registry traffic), then you can retrieve the pullspec from the ClusterVersion but will not resolve it locally. The cluster may also be using ImageContentSourcePolicies [1] to route not-actually-resolvable pullspecs to local registries. I dunno what oc would do to piggy-back on those. Would be great if the cluster offered a service over the Kubernetes API where you could pull an image by proxy. Client: Kube, I've got this pullspec... Kube: [Applies ImageContentSourcePolicies, Proxy, whatever settings to get a registry] Registry, I'd like this image... Registry: Here you go, Kube Kube: Here you go, client. Even that's a bit ugly, because ImageContentSourcePolicies is currently feeding registries.conf and being consumed by CRI-O, so the Kube API level doesn't have any idea what's going on in this space. So this is not going to be easy in situations that have ImageContentSourcePolicies, Proxy, etc. in play. But it should be possible, and we can keep chipping away at any missing pieces over time. [1]: https://github.com/openshift/api/commit/36cf2f0ae0f76a954537b236288cc72b1b6d8b79
fyi, I have noticed this last week when testing an offline deployment on Power: [must-gather ] OUT pod for plug-in image quay.io/openshift/origin-must-gather:latest created [must-gather-zg4cm] OUT gather did not start: unable to pull image: ErrImagePull: rpc error: code = Unknown desc = error pinging docker registry quay.io: Get https://quay.io/v2/: dial tcp 3.212.220.94:443: i/o timeout
https://bugzilla.redhat.com/show_bug.cgi?id=1823839#c1 and https://bugzilla.redhat.com/show_bug.cgi?id=1823839#c2 are the same, no? imagstream openshift/must-gather is same as what you see with 'oc adm release info --image-for=must-gather' for example, with a cluster launched with this release: registry.svc.ci.openshift.org/ocp/release:4.5.0-0.ci-2020-05-14-215330, must-gather uses 'Using must-gather plugin-in image: registry.svc.ci.openshift.org/ocp/4.5-2020-05-14-215330@sha256:70a34992b1f1235720d227a645565200c1aaa55af2053335186be45ffdb42a8d' that is same as you get from the oc adm release info cmd by default, then if it can't access that image will default to the 'quay.io/openshift/origin-must-gather:latest'. This bz is really about what image is used in offline environments - and in that case is it reasonable to expect users to pass `oc adm must-gather --image=myreg/myrepo/must-gather:disco` ? Perhaps instead of defaulting to that quay image user should get a message that they need to pass an --image that points to an accessible must-gather image?
> This bz is really about what image is used in offline environments - and in that case is it reasonable to expect users to pass `oc adm must-gather --image=myreg/myrepo/must-gather:disco` ? The cluster has ImageContentSourcePolicies and such available so the CVO can push deployments and such with the canonical pullspecs, and CRI-O will check down through available mirrors until it finds an image with a matching digest. It is not clear to me why we couldn't push a must-gather Pod into the cluster with the canonical, release-referenced must-gather pullspec and have it work. The issue is that it's really hard for the off-cluster 'oc' to get the release image so it can discover the release-referenced pullspecs. That also impacts usage like 'oc adm release info', where oc is asked to discover the release image from some cluster property, and then fetch the image locally to peek inside and pull out metadata. imagstream openshift/must-gather sounds like it avoids the local-oc-lookup issue. Are we clear on why it wasn't working in the disconnected environment? I'm not familiar with how imagestreams get set up from the release image references.
@esimard or @rsene, In disconnnected environment, do you have access to a must-gather image? It would be `yourregistry/yourrepo/release:tag-must-gather` and then, if so, does must-gather cmd succeed when you pass that image like so? $ oc adm must-gather --image=yourregistry/yourrepo/release:tag-must-gather In future, we might add ability to read from your ImageContentSourcePolicy from a cluster, or from an ICSP file, but please confirm you have this workaround, thanks.
In order to resolve this properly, we're going to add ImageContentSources awareness to oc, I've opened a jira for this here: https://issues.redhat.com/browse/WRKLDS-174 Please track this through jira
*** Bug 1983347 has been marked as a duplicate of this bug. ***
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days