1943315 – avoid workload disruption for ICSP changes

Bug 1943315 - avoid workload disruption for ICSP changes

Summary: avoid workload disruption for ICSP changes

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Machine Config Operator
Sub Component:
Version:	4.7
Hardware:	All
OS:	Linux
Priority:	urgent
Severity:	urgent
Target Milestone:	---
Target Release:	4.8.0
Assignee:	Sinny Kumari
QA Contact:	Michael Nguyen
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2021-03-25 19:09 UTC by Jeremy Eder
Modified:	2021-07-29 15:34 UTC (History)
CC List:	6 users (show)
Fixed In Version:
Doc Type:	Enhancement
Doc Text:	Feature: Avoid workload disruption for selected ICSP changes Reason: This feature will help our OpenShift customers and impacted team such as Service Delivery to add additional mirrors and registry without worrying about workload disruption. Result: Workload disruption will not occur for selected registries.conf changes(/etc/containers/registries.conf, e.g. ICSP changes. This includes following changes in registries.conf file: - addition of a registry with mirror-by-digest-only=true - addition of a mirror in a registry with mirror-by-digest-only=true - appending items in unqualified-search-registries list For any other changes in regsitries.conf, MCO will default to draining nodes to apply changes.
Clone Of:
Environment:
Last Closed:	2021-07-27 22:55:47 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift machine-config-operator pull 2596	0	None	open	Bug 1943315: daemon: make node drain logic smarter for ICSP changes	2021-06-03 16:07:05 UTC
Red Hat Product Errata	RHSA-2021:2438	0	None	None	None	2021-07-27 22:56:22 UTC

Comment 8 Michael Nguyen 2021-06-17 15:25:32 UTC

Verified on 4.8.0-0.nightly-2021-06-16-190035.  Added mirror to ICSP and watched `oc get mcp` and `oc get nodes`.  MCP gets updated under a minute.  Nodes get cordoned/uncordoned with no workload drain.

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-06-16-190035   True        False         63m     Cluster version is 4.8.0-0.nightly-2021-06-16-190035
$ oc get nodes
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-134-236.us-west-2.compute.internal   Ready    master   85m   v1.21.0-rc.0+120883f
ip-10-0-150-206.us-west-2.compute.internal   Ready    worker   80m   v1.21.0-rc.0+120883f
ip-10-0-164-27.us-west-2.compute.internal    Ready    master   85m   v1.21.0-rc.0+120883f
ip-10-0-183-87.us-west-2.compute.internal    Ready    worker   78m   v1.21.0-rc.0+120883f
ip-10-0-210-154.us-west-2.compute.internal   Ready    master   86m   v1.21.0-rc.0+120883f
ip-10-0-222-34.us-west-2.compute.internal    Ready    worker   79m   v1.21.0-rc.0+120883f
$ oc debug node/ip-10-0-150-206.us-west-2.compute.internal -- chroot /host cat /etc/containers/registries.conf
Starting pod/ip-10-0-150-206us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
unqualified-search-registries = ['registry.access.redhat.com', 'docker.io']

Removing debug pod ...
$ cat repositoryrepomirror.yaml 
apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: ubi8repo
spec:
  repositoryDigestMirrors:
  - mirrors:
    - example.io/example/ubi-minimal
    source: registry.access.redhat.com/ubi8/ubi-minimal 
  - mirrors:
    - example.com/example/ubi-minimal
    source: registry.access.redhat.com/ubi8/ubi-minimal
$ oc create -f repositoryrepomirror.yaml 
imagecontentsourcepolicy.operator.openshift.io/ubi8repo created
$ oc get ImageContentSourcePolicy 
NAME       AGE
ubi8repo   10s
$ oc get nodes
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-134-236.us-west-2.compute.internal   Ready    master   87m   v1.21.0-rc.0+120883f
ip-10-0-150-206.us-west-2.compute.internal   Ready    worker   81m   v1.21.0-rc.0+120883f
ip-10-0-164-27.us-west-2.compute.internal    Ready    master   87m   v1.21.0-rc.0+120883f
ip-10-0-183-87.us-west-2.compute.internal    Ready    worker   79m   v1.21.0-rc.0+120883f
ip-10-0-210-154.us-west-2.compute.internal   Ready    master   88m   v1.21.0-rc.0+120883f
ip-10-0-222-34.us-west-2.compute.internal    Ready    worker   81m   v1.21.0-rc.0+120883f
$ oc debug node/ip-10-0-150-206.us-west-2.compute.internal -- chroot /host cat /etc/containers/registries.conf
Starting pod/ip-10-0-150-206us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]

[[registry]]
  prefix = ""
  location = "registry.access.redhat.com/ubi8/ubi-minimal"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "example.com/example/ubi-minimal"

  [[registry.mirror]]
    location = "example.io/example/ubi-minimal"

Removing debug pod ...
$ oc edit ImageContentSourcePolicy 
imagecontentsourcepolicy.operator.openshift.io/ubi8repo edited
$ oc debug node/ip-10-0-150-206.us-west-2.compute.internal -- chroot /host cat /etc/containers/registries.conf
Starting pod/ip-10-0-150-206us-west-2computeinternal-debug ...
To use host binaries, run `chroot /host`
unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]

[[registry]]
  prefix = ""
  location = "registry.access.redhat.com/ubi8/ubi-minimal"
  mirror-by-digest-only = true

  [[registry.mirror]]
    location = "example.com/example/ubi-minimal"

  [[registry.mirror]]
    location = "example.io/example/ubi-minimal"

  [[registry.mirror]]
    location = "example.sample.com/example/ubi-minimal"

Removing debug pod ...


== edited last mirror to essentially delete it causes drain.  First few log messages show safe drain.  Last message shows removing a mirror which requires a drain ==

$ oc -n openshift-machine-config-operator logs machine-config-daemon-df99g -c machine-config-daemon | grep drain
I0617 15:10:19.885418    1719 drain.go:288] /etc/containers/registries.conf: changes made are safe to skip drain
I0617 15:10:19.885443    1719 update.go:581] Changes do not require drain, skipping.
I0617 15:12:35.025660    1719 drain.go:288] /etc/containers/registries.conf: changes made are safe to skip drain
I0617 15:12:35.025688    1719 update.go:581] Changes do not require drain, skipping.
I0617 15:16:41.822016    1719 drain.go:262] /etc/containers/registries.conf: mirror example.sample.com/example/ubi-minimal has been removed in registry registry.access.redhat.com/ubi8/ubi-minimal
I0617 15:16:41.834139    1719 update.go:1874] Update prepared; beginning drain
I0617 15:17:56.602840    1719 update.go:1874] drain complete
I0617 15:17:56.605564    1719 drain.go:126] Successful drain took 74.769024196 seconds

Comment 10 errata-xmlrpc 2021-07-27 22:55:47 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438

Note You need to log in before you can comment on or make changes to this bug.