Bug 1881694
| Summary: | Evidence of disconnected installs pulling images from the local registry instead of quay.io | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Sam Yangsao <syangsao> | |
| Component: | Node | Assignee: | Qi Wang <qiwan> | |
| Node sub component: | CRI-O | QA Contact: | MinLi <minmli> | |
| Status: | CLOSED ERRATA | Docs Contact: | ||
| Severity: | low | |||
| Priority: | medium | CC: | aos-bugs, dwalsh, jokerman, minmli, nagrawal, scuppett, tsweeney, umohnani | |
| Version: | 4.4 | |||
| Target Milestone: | --- | |||
| Target Release: | 4.8.0 | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: |
Cause: Crio logs has no information about the pull source
Consequence: Failed to tell if the image was pulled from the registry mirror.
Fix: Added info level log information to crio about the pull source of the image.
Result: The physical pull source can be shown in the info level log.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1976293 (view as bug list) | Environment: | ||
| Last Closed: | 2021-07-27 22:33:30 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1976293, 1976297 | |||
Qi, can you take a look at this. You just have to bump the existing logs from debug to info level. PR set up in containers/image to show the log https://github.com/containers/image/pull/1099 https://github.com/cri-o/cri-o/pull/4536 merged to crio release-1.20 verified on version: 4.8.0-0.nightly-2021-03-16-221720 sh-4.4# crictl version Version: 0.1.0 RuntimeName: cri-o RuntimeVersion: 1.21.0-24.rhaos4.8.git69cc531.el8-dev RuntimeApiVersion: v1alpha2 crio logs as below: Mar 17 02:52:50 minmli0317gcp1-46m6l-m-0.c.openshift-qe.internal crio[1484]: time="2021-03-17 02:52:50.178706830Z" level=info msg="Checking image status: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1926eae7cacb9c00f142ec98b00628970e974284b6ddaf9a6a086cb9af7a6c31" id=d71a23f1-17e0-40d0-b967-57f2e474bf0c name=/runtime.v1alpha2.ImageService/ImageStatus Mar 17 02:52:50 minmli0317gcp1-46m6l-m-0.c.openshift-qe.internal crio[1484]: time="2021-03-17 02:52:50.179161853Z" level=info msg="Image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1926eae7cacb9c00f142ec98b00628970e974284b6ddaf9a6a086cb9af7a6c31 not found" id=d71a23f1-17e0-40d0-b967-57f2e474bf0c name=/runtime.v1alpha2.ImageService/ImageStatus Mar 17 02:52:50 minmli0317gcp1-46m6l-m-0.c.openshift-qe.internal crio[1484]: time="2021-03-17 02:52:50.190753958Z" level=info msg="Pulling image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1926eae7cacb9c00f142ec98b00628970e974284b6ddaf9a6a086cb9af7a6c31" id=f740d165-4482-439f-aced-77c781f33731 name=/runtime.v1alpha2.ImageService/PullImage Mar 17 02:52:50 minmli0317gcp1-46m6l-m-0.c.openshift-qe.internal crio[1484]: time="2021-03-17 02:52:50.194341109Z" level=info msg="Trying to access \"minmli0317gcp1.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release@sha256:1926eae7cacb9c00f142ec98b00628970e974284b6ddaf9a6a086cb9af7a6c31\"" Mar 17 02:52:50 minmli0317gcp1-46m6l-m-0.c.openshift-qe.internal crio[1484]: time="2021-03-17 02:52:50.226788351Z" level=info msg="Trying to access \"minmli0317gcp1.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release@sha256:1926eae7cacb9c00f142ec98b00628970e974284b6ddaf9a6a086cb9af7a6c31\"" Mar 17 02:52:52 minmli0317gcp1-46m6l-m-0.c.openshift-qe.internal crio[1484]: time="2021-03-17 02:52:52.936392518Z" level=info msg="Pulled image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1926eae7cacb9c00f142ec98b00628970e974284b6ddaf9a6a086cb9af7a6c31" id=f740d165-4482-439f-aced-77c781f33731 name=/runtime.v1alpha2.ImageService/PullImage @scuppett https://bugzilla.redhat.com/show_bug.cgi?id=1976293, https://bugzilla.redhat.com/show_bug.cgi?id=1976297 Created this BZ to keep track for backporting 4.6 and 4.7. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438 |
Description of problem: We are unable to verify that the disconnected install is using a local repository to pull its images Version-Release number of the following components: 4.4 4.5 How reproducible: Always Steps to Reproduce: 1. Install OCP following [1] for a disconnected installation on 4.4 (customer) or 4.5 (my local reproducer) 2. Kick off the OpenShift installer and login to check the image pulls on the control plane nodes ... <snip> # Customer OCP 4.4.9 [root@control-plane-0 ~]# podman info host: BuildahVersion: 1.12.0-dev CgroupVersion: v1 Conmon: package: conmon-2.0.15-1.rhaos4.4.el8.x86_64 path: /usr/bin/conmon version: 'conmon version 2.0.15, commit: bf8d161cbc481375624ee6feaa8c6a31116fff61' Distribution: distribution: '"rhcos"' version: "4.4" MemFree: 6980648960 MemTotal: 16815443968 OCIRuntime: name: runc package: runc-1.0.0-68.rc10.rhaos4.4.el8.x86_64 path: /usr/bin/runc version: 'runc version spec: 1.0.1-dev' SwapFree: 0 SwapTotal: 0 arch: amd64 cpus: 4 eventlogger: journald hostname: control-plane-0.ocp-core-lab.otl.lab kernel: 4.18.0-147.8.1.el8_1.x86_64 os: linux rootless: false uptime: 13m 3.36s registries: blocked: null insecure: null search: - registry.access.redhat.com - docker.io store: ConfigFile: /etc/containers/storage.conf ContainerStore: number: 149 GraphDriverName: overlay GraphOptions: {} GraphRoot: /var/lib/containers/storage GraphStatus: Backing Filesystem: xfs Native Overlay Diff: "true" Supports d_type: "true" Using metacopy: "false" ImageStore: number: 37 RunRoot: /var/run/containers/storage VolumePath: /var/lib/containers/storage/volumes [root@control-plane-0 ~]# crictl images IMAGE TAG IMAGE ID SIZE quay.io/openshift-release-dev/ocp-release@sha256 <none> 06be4357dfb81 306MB quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 <none> b1919d4f27337 833MB quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 <none> a79ce495fb31a 365MB quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 <none> 17572d1bef427 429MB quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 <none> ce156d216c9ea 467MB quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 <none> 0715d5619b646 237MB [...] # Mine OCP 4.5.7 [root@master01 ~]# podman info host: arch: amd64 buildahVersion: 1.14.9 cgroupVersion: v1 conmon: package: conmon-2.0.20-1.rhaos4.5.el8.x86_64 path: /usr/bin/conmon version: 'conmon version 2.0.20, commit: a5deeba01836230ad02aff29709766da822d64f2' cpus: 4 distribution: distribution: '"rhcos"' version: "4.5" eventLogger: file hostname: master01.disocp4.lab.msp.redhat.com idMappings: gidmap: null uidmap: null kernel: 4.18.0-193.14.3.el8_2.x86_64 memFree: 5798805504 memTotal: 16813441024 ociRuntime: name: runc package: runc-1.0.0-70.rhaos4.5.gite677e8b.el8.x86_64 path: /usr/bin/runc version: 'runc version spec: 1.0.2' os: linux rootless: false slirp4netns: executable: "" package: "" version: "" swapFree: 0 swapTotal: 0 uptime: 4h 5m 12.47s (Approximately 0.17 days) registries: quay.io/openshift-release-dev/ocp-release: Blocked: false Insecure: false Location: quay.io/openshift-release-dev/ocp-release MirrorByDigestOnly: true Mirrors: - Insecure: false Location: registry.lab.msp.redhat.com:5000/ocp4/openshift4 Prefix: quay.io/openshift-release-dev/ocp-release quay.io/openshift-release-dev/ocp-v4.0-art-dev: Blocked: false Insecure: false Location: quay.io/openshift-release-dev/ocp-v4.0-art-dev MirrorByDigestOnly: true Mirrors: - Insecure: false Location: registry.lab.msp.redhat.com:5000/ocp4/openshift4 Prefix: quay.io/openshift-release-dev/ocp-v4.0-art-dev search: - registry.access.redhat.com - docker.io store: configFile: /etc/containers/storage.conf containerStore: number: 0 paused: 0 running: 0 stopped: 0 graphDriverName: overlay graphOptions: {} graphRoot: /var/lib/containers/storage graphStatus: Backing Filesystem: xfs Native Overlay Diff: "true" Supports d_type: "true" Using metacopy: "false" imageStore: number: 37 runRoot: /var/run/containers/storage volumePath: /var/lib/containers/storage/volumes [root@master01 ~]# crictl images IMAGE TAG IMAGE ID SIZE quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 <none> d768a2057ee34 908MB quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 <none> 3fcd563edad3b 255MB quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 <none> 47e0484c14e14 369MB quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 <none> bd254f0d16b55 430MB quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256 <none> 751b89344e723 471MB [...] </snip> Actual results: Just from looking at this quickly, `podman info` on the control plane node shows registries but not the local one in OCP 4.4 as configured, but in mine on 4.5 it does. Not sure if this was changed between releases or it just didn't accept the configuration on the initial creation of the ignition files. # 4.4 registries: blocked: null insecure: null search: - registry.access.redhat.com - docker.io # 4.5 registries: quay.io/openshift-release-dev/ocp-release: Blocked: false Insecure: false Location: quay.io/openshift-release-dev/ocp-release MirrorByDigestOnly: true Mirrors: - Insecure: false Location: registry.lab.msp.redhat.com:5000/ocp4/openshift4 Prefix: quay.io/openshift-release-dev/ocp-release quay.io/openshift-release-dev/ocp-v4.0-art-dev: Blocked: false Insecure: false Location: quay.io/openshift-release-dev/ocp-v4.0-art-dev MirrorByDigestOnly: true Mirrors: - Insecure: false Location: registry.lab.msp.redhat.com:5000/ocp4/openshift4 Prefix: quay.io/openshift-release-dev/ocp-v4.0-art-dev search: - registry.access.redhat.com - docker.io Expected results: Method or command to verify that the image is being pulled by a local repository instead of quay.io for disconnected installs. I see this [2] in regards to the image spec and that the only wait to see where the image is being pulled is by increasing the log-level of CRI-O - this is not acceptable - customer needs a simple way of verifying this whether or not the network they are installing OCP on has internet access. Additional info: Will attach their install_config.yaml as well [1] https://docs.openshift.com/container-platform/4.5/installing/installing_vsphere/installing-restricted-networks-vsphere.html [2] https://bugzilla.redhat.com/show_bug.cgi?id=1848752#c7