Bug 1784979 - Upgrade from 4.2.9 to 4.2.10 stuck in machine-config occasionally
Summary: Upgrade from 4.2.9 to 4.2.10 stuck in machine-config occasionally
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: RHCOS
Version: 4.4
Hardware: Unspecified
OS: Unspecified
low
medium
Target Milestone: ---
: 4.4.0
Assignee: Micah Abbott
QA Contact: Michael Nguyen
URL:
Whiteboard:
Depends On:
Blocks: 1784981
TreeView+ depends on / blocked
 
Reported: 2019-12-18 20:30 UTC by Kirsten Garrison
Modified: 2020-05-04 11:21 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1781665
: 1784981 (view as bug list)
Environment:
Last Closed: 2020-05-04 11:20:43 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift machine-config-operator pull 1285 None closed Bug 1775647: pkg/daemon: randomize pivot container name 2020-02-11 20:27:08 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-04 11:21:18 UTC

Comment 1 Kirsten Garrison 2019-12-18 20:32:34 UTC
Fix was already merged into 4.4 creating 4.4 bug to facilitate backports.

Comment 3 Michael Nguyen 2019-12-19 20:54:42 UTC
Verified `ostree-container-pivot-<uuid>` is being used by podman create.

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2019-12-19-160523   True        False         32m     Cluster version is 4.4.0-0.nightly-2019-12-19-160523
$ oc get node
NAME                           STATUS   ROLES    AGE    VERSION
ip-10-0-132-200.ec2.internal   Ready    worker   110m   v1.16.2
ip-10-0-141-68.ec2.internal    Ready    master   121m   v1.16.2
ip-10-0-151-226.ec2.internal   Ready    master   121m   v1.16.2
ip-10-0-153-157.ec2.internal   Ready    worker   109m   v1.16.2
ip-10-0-171-0.ec2.internal     Ready    worker   110m   v1.16.2
ip-10-0-175-31.ec2.internal    Ready    master   121m   v1.16.2
$ oc debug node/ip-10-0-132-200.ec2.internal
Starting pod/ip-10-0-132-200ec2internal-debug ...
To use host binaries, run `chroot /host`
chroot /hostIf you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# journalctl -u machine-config-daemon-host | grep 'podman create'
Dec 19 18:59:38 ip-10-0-132-200 machine-config-daemon[1799]: I1219 18:59:38.906530    1799 rpm-ostree.go:366] Running captured: podman create --net=none --annotation=org.openshift.machineconfigoperator.pivot=true --name ostree-container-pivot-3189126b-379b-4d5e-abbc-43da5c6ed08b quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a506e8e47a92a21b085850d8f8e04beafe357e2bd51ca10a551685117ca9442
Dec 19 20:04:05 ip-10-0-132-200 machine-config-daemon[126587]: I1219 20:04:05.151125  126587 rpm-ostree.go:366] Running captured: podman create --net=none --annotation=org.openshift.machineconfigoperator.pivot=true --name ostree-container-pivot-bb9af6f4-384c-4e41-af34-713a05fb53a1 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:922c0081e73806ce176802944754d6332c8c59da98971cc95c2d4d42d1f527a8
sh-4.4# exit
exit
sh-4.2# exit
exit

Removing debug pod ...
$ oc debug node/ip-10-0-141-68.ec2.internal
Starting pod/ip-10-0-141-68ec2internal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# journalctl -u machine-config-daemon-host | grep 'podman create'
Dec 19 18:46:32 ip-10-0-141-68 machine-config-daemon[2215]: I1219 18:46:32.928119    2215 rpm-ostree.go:366] Running captured: podman create --net=none --annotation=org.openshift.machineconfigoperator.pivot=true --name ostree-container-pivot-c5167ae5-ed1b-445e-a577-58190c115ebb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a506e8e47a92a21b085850d8f8e04beafe357e2bd51ca10a551685117ca9442
Dec 19 20:03:43 ip-10-0-141-68 machine-config-daemon[192530]: I1219 20:03:43.398731  192530 rpm-ostree.go:366] Running captured: podman create --net=none --annotation=org.openshift.machineconfigoperator.pivot=true --name ostree-container-pivot-92e40e6e-8b5b-4801-9758-00da474ca2de quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:922c0081e73806ce176802944754d6332c8c59da98971cc95c2d4d42d1f527a8
sh-4.4# podman ps -a
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES
sh-4.4# exit
exit
sh-4.2# exit
exit

Removing debug pod ...

Comment 5 errata-xmlrpc 2020-05-04 11:20:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.