1881694 – Evidence of disconnected installs pulling images from the local registry instead of quay.io

Bug 1881694 - Evidence of disconnected installs pulling images from the local registry instead of quay.io

Summary: Evidence of disconnected installs pulling images from the local registry inst...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Node
Sub Component:
Version:	4.4
Hardware:	x86_64
OS:	Linux
Priority:	medium
Severity:	low
Target Milestone:	---
Target Release:	4.8.0
Assignee:	Qi Wang
QA Contact:	MinLi
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1976293 1976297
TreeView+	depends on / blocked

Reported:	2020-09-22 21:58 UTC by Sam Yangsao
Modified:	2024-10-01 16:54 UTC (History)
CC List:	8 users (show)
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	Cause: Crio logs has no information about the pull source Consequence: Failed to tell if the image was pulled from the registry mirror. Fix: Added info level log information to crio about the pull source of the image. Result: The physical pull source can be shown in the info level log.
Clone Of:
Clones:	1976293 (view as bug list)
Environment:
Last Closed:	2021-07-27 22:33:30 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	cri-o cri-o pull 4536	0	None	closed	[release-1.20] Bug 1881694 Add pull source as info level log	2021-06-25 11:30:39 UTC
Red Hat Product Errata	RHSA-2021:2438	0	None	None	None	2021-07-27 22:34:13 UTC

Description Sam Yangsao 2020-09-22 21:58:44 UTC

Description of problem:

We are unable to verify that the disconnected install is using a local repository to pull its images

Version-Release number of the following components:

4.4
4.5

How reproducible:

Always

Steps to Reproduce:

1.  Install OCP following [1] for a disconnected installation on 4.4 (customer) or 4.5 (my local reproducer)
2.  Kick off the OpenShift installer and login to check the image pulls on the control plane nodes ...

<snip>

# Customer OCP 4.4.9

[root@control-plane-0 ~]# podman info
host:
  BuildahVersion: 1.12.0-dev
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.15-1.rhaos4.4.el8.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.15, commit: bf8d161cbc481375624ee6feaa8c6a31116fff61'
  Distribution:
    distribution: '"rhcos"'
    version: "4.4"
  MemFree: 6980648960
  MemTotal: 16815443968
  OCIRuntime:
    name: runc
    package: runc-1.0.0-68.rc10.rhaos4.4.el8.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: control-plane-0.ocp-core-lab.otl.lab
  kernel: 4.18.0-147.8.1.el8_1.x86_64
  os: linux
  rootless: false
  uptime: 13m 3.36s
registries:
  blocked: null
  insecure: null
  search:
  - registry.access.redhat.com
  - docker.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 149
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 37
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

[root@control-plane-0 ~]# crictl images
IMAGE                                                   TAG                 IMAGE ID            SIZE
quay.io/openshift-release-dev/ocp-release@sha256        <none>              06be4357dfb81       306MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256   <none>              b1919d4f27337       833MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256   <none>              a79ce495fb31a       365MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256   <none>              17572d1bef427       429MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256   <none>              ce156d216c9ea       467MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256   <none>              0715d5619b646       237MB
[...]

# Mine OCP 4.5.7

[root@master01 ~]# podman info
host:
  arch: amd64
  buildahVersion: 1.14.9
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.20-1.rhaos4.5.el8.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.20, commit: a5deeba01836230ad02aff29709766da822d64f2'
  cpus: 4
  distribution:
    distribution: '"rhcos"'
    version: "4.5"
  eventLogger: file
  hostname: master01.disocp4.lab.msp.redhat.com
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.18.0-193.14.3.el8_2.x86_64
  memFree: 5798805504
  memTotal: 16813441024
  ociRuntime:
    name: runc
    package: runc-1.0.0-70.rhaos4.5.gite677e8b.el8.x86_64
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.2'
  os: linux
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 0
  swapTotal: 0
  uptime: 4h 5m 12.47s (Approximately 0.17 days)
registries:
  quay.io/openshift-release-dev/ocp-release:
    Blocked: false
    Insecure: false
    Location: quay.io/openshift-release-dev/ocp-release
    MirrorByDigestOnly: true
    Mirrors:
    - Insecure: false
      Location: registry.lab.msp.redhat.com:5000/ocp4/openshift4
    Prefix: quay.io/openshift-release-dev/ocp-release
  quay.io/openshift-release-dev/ocp-v4.0-art-dev:
    Blocked: false
    Insecure: false
    Location: quay.io/openshift-release-dev/ocp-v4.0-art-dev
    MirrorByDigestOnly: true
    Mirrors:
    - Insecure: false
      Location: registry.lab.msp.redhat.com:5000/ocp4/openshift4
    Prefix: quay.io/openshift-release-dev/ocp-v4.0-art-dev
  search:
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 37
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes

[root@master01 ~]# crictl images
IMAGE                                                   TAG                 IMAGE ID            SIZE
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256   <none>              d768a2057ee34       908MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256   <none>              3fcd563edad3b       255MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256   <none>              47e0484c14e14       369MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256   <none>              bd254f0d16b55       430MB
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256   <none>              751b89344e723       471MB
[...]

</snip>

Actual results:

Just from looking at this quickly, `podman info` on the control plane node shows registries but not the local one in OCP 4.4 as configured, but in mine on 4.5 it does.  Not sure if this was changed between releases or it just didn't accept the configuration on the initial creation of the ignition files.

# 4.4

registries:
  blocked: null
  insecure: null
  search:
  - registry.access.redhat.com
  - docker.io

# 4.5

registries:
  quay.io/openshift-release-dev/ocp-release:
    Blocked: false
    Insecure: false
    Location: quay.io/openshift-release-dev/ocp-release
    MirrorByDigestOnly: true
    Mirrors:
    - Insecure: false
      Location: registry.lab.msp.redhat.com:5000/ocp4/openshift4
    Prefix: quay.io/openshift-release-dev/ocp-release
  quay.io/openshift-release-dev/ocp-v4.0-art-dev:
    Blocked: false
    Insecure: false
    Location: quay.io/openshift-release-dev/ocp-v4.0-art-dev
    MirrorByDigestOnly: true
    Mirrors:
    - Insecure: false
      Location: registry.lab.msp.redhat.com:5000/ocp4/openshift4
    Prefix: quay.io/openshift-release-dev/ocp-v4.0-art-dev
  search:
  - registry.access.redhat.com
  - docker.io

Expected results:

Method or command to verify that the image is being pulled by a local repository instead of quay.io for disconnected installs.

I see this [2] in regards to the image spec and that the only wait to see where the image is being pulled is by increasing the log-level of CRI-O - this is not acceptable - customer needs a simple way of verifying this whether or not the network they are installing OCP on has internet access.

Additional info:

Will attach their install_config.yaml as well

[1] https://docs.openshift.com/container-platform/4.5/installing/installing_vsphere/installing-restricted-networks-vsphere.html
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1848752#c7

Comment 11 Urvashi Mohnani 2020-12-04 20:07:23 UTC

Qi, can you take a look at this. You just have to bump the existing logs from debug to info level.

Comment 12 Qi Wang 2020-12-09 20:25:01 UTC

PR set up in containers/image to show the log https://github.com/containers/image/pull/1099

Comment 14 Qi Wang 2021-02-03 16:36:50 UTC

Cri-o PR https://github.com/cri-o/cri-o/pull/4536

Comment 15 Qi Wang 2021-02-24 14:31:02 UTC

https://github.com/cri-o/cri-o/pull/4536 merged to crio release-1.20

Comment 17 MinLi 2021-03-17 07:21:35 UTC

verified on version: 4.8.0-0.nightly-2021-03-16-221720

sh-4.4# crictl version 
Version:  0.1.0
RuntimeName:  cri-o
RuntimeVersion:  1.21.0-24.rhaos4.8.git69cc531.el8-dev
RuntimeApiVersion:  v1alpha2

crio logs as below:
Mar 17 02:52:50 minmli0317gcp1-46m6l-m-0.c.openshift-qe.internal crio[1484]: time="2021-03-17 02:52:50.178706830Z" level=info msg="Checking image status: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1926eae7cacb9c00f142ec98b00628970e974284b6ddaf9a6a086cb9af7a6c31" id=d71a23f1-17e0-40d0-b967-57f2e474bf0c name=/runtime.v1alpha2.ImageService/ImageStatus
Mar 17 02:52:50 minmli0317gcp1-46m6l-m-0.c.openshift-qe.internal crio[1484]: time="2021-03-17 02:52:50.179161853Z" level=info msg="Image quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1926eae7cacb9c00f142ec98b00628970e974284b6ddaf9a6a086cb9af7a6c31 not found" id=d71a23f1-17e0-40d0-b967-57f2e474bf0c name=/runtime.v1alpha2.ImageService/ImageStatus
Mar 17 02:52:50 minmli0317gcp1-46m6l-m-0.c.openshift-qe.internal crio[1484]: time="2021-03-17 02:52:50.190753958Z" level=info msg="Pulling image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1926eae7cacb9c00f142ec98b00628970e974284b6ddaf9a6a086cb9af7a6c31" id=f740d165-4482-439f-aced-77c781f33731 name=/runtime.v1alpha2.ImageService/PullImage
Mar 17 02:52:50 minmli0317gcp1-46m6l-m-0.c.openshift-qe.internal crio[1484]: time="2021-03-17 02:52:50.194341109Z" level=info msg="Trying to access \"minmli0317gcp1.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release@sha256:1926eae7cacb9c00f142ec98b00628970e974284b6ddaf9a6a086cb9af7a6c31\""
Mar 17 02:52:50 minmli0317gcp1-46m6l-m-0.c.openshift-qe.internal crio[1484]: time="2021-03-17 02:52:50.226788351Z" level=info msg="Trying to access \"minmli0317gcp1.mirror-registry.qe.gcp.devcluster.openshift.com:5000/ocp/release@sha256:1926eae7cacb9c00f142ec98b00628970e974284b6ddaf9a6a086cb9af7a6c31\""
Mar 17 02:52:52 minmli0317gcp1-46m6l-m-0.c.openshift-qe.internal crio[1484]: time="2021-03-17 02:52:52.936392518Z" level=info msg="Pulled image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1926eae7cacb9c00f142ec98b00628970e974284b6ddaf9a6a086cb9af7a6c31" id=f740d165-4482-439f-aced-77c781f33731 name=/runtime.v1alpha2.ImageService/PullImage

Comment 20 Qi Wang 2021-06-25 17:44:08 UTC

@scuppett https://bugzilla.redhat.com/show_bug.cgi?id=1976293, https://bugzilla.redhat.com/show_bug.cgi?id=1976297 Created this BZ to keep track for backporting 4.6 and 4.7.

Comment 22 errata-xmlrpc 2021-07-27 22:33:30 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438

Note You need to log in before you can comment on or make changes to this bug.