Bug 1784979

Summary: Upgrade from 4.2.9 to 4.2.10 stuck in machine-config occasionally
Product: OpenShift Container Platform Reporter: Kirsten Garrison <kgarriso>
Component: RHCOSAssignee: Micah Abbott <miabbott>
Status: CLOSED ERRATA QA Contact: Michael Nguyen <mnguyen>
Severity: medium Docs Contact:
Priority: low    
Version: 4.4CC: amurdaca, aos-bugs, bbreard, dustymabe, imcleod, jligon, jokerman, kgarriso, miabbott, mnguyen, nstielau, wzheng
Target Milestone: ---   
Target Release: 4.4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1781665
: 1784981 (view as bug list) Environment:
Last Closed: 2020-05-04 11:20:43 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1784981    

Comment 1 Kirsten Garrison 2019-12-18 20:32:34 UTC
Fix was already merged into 4.4 creating 4.4 bug to facilitate backports.

Comment 3 Michael Nguyen 2019-12-19 20:54:42 UTC
Verified `ostree-container-pivot-<uuid>` is being used by podman create.

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2019-12-19-160523   True        False         32m     Cluster version is 4.4.0-0.nightly-2019-12-19-160523
$ oc get node
NAME                           STATUS   ROLES    AGE    VERSION
ip-10-0-132-200.ec2.internal   Ready    worker   110m   v1.16.2
ip-10-0-141-68.ec2.internal    Ready    master   121m   v1.16.2
ip-10-0-151-226.ec2.internal   Ready    master   121m   v1.16.2
ip-10-0-153-157.ec2.internal   Ready    worker   109m   v1.16.2
ip-10-0-171-0.ec2.internal     Ready    worker   110m   v1.16.2
ip-10-0-175-31.ec2.internal    Ready    master   121m   v1.16.2
$ oc debug node/ip-10-0-132-200.ec2.internal
Starting pod/ip-10-0-132-200ec2internal-debug ...
To use host binaries, run `chroot /host`
chroot /hostIf you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# journalctl -u machine-config-daemon-host | grep 'podman create'
Dec 19 18:59:38 ip-10-0-132-200 machine-config-daemon[1799]: I1219 18:59:38.906530    1799 rpm-ostree.go:366] Running captured: podman create --net=none --annotation=org.openshift.machineconfigoperator.pivot=true --name ostree-container-pivot-3189126b-379b-4d5e-abbc-43da5c6ed08b quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a506e8e47a92a21b085850d8f8e04beafe357e2bd51ca10a551685117ca9442
Dec 19 20:04:05 ip-10-0-132-200 machine-config-daemon[126587]: I1219 20:04:05.151125  126587 rpm-ostree.go:366] Running captured: podman create --net=none --annotation=org.openshift.machineconfigoperator.pivot=true --name ostree-container-pivot-bb9af6f4-384c-4e41-af34-713a05fb53a1 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:922c0081e73806ce176802944754d6332c8c59da98971cc95c2d4d42d1f527a8
sh-4.4# exit
exit
sh-4.2# exit
exit

Removing debug pod ...
$ oc debug node/ip-10-0-141-68.ec2.internal
Starting pod/ip-10-0-141-68ec2internal-debug ...
To use host binaries, run `chroot /host`
If you don't see a command prompt, try pressing enter.
sh-4.2# chroot /host
sh-4.4# journalctl -u machine-config-daemon-host | grep 'podman create'
Dec 19 18:46:32 ip-10-0-141-68 machine-config-daemon[2215]: I1219 18:46:32.928119    2215 rpm-ostree.go:366] Running captured: podman create --net=none --annotation=org.openshift.machineconfigoperator.pivot=true --name ostree-container-pivot-c5167ae5-ed1b-445e-a577-58190c115ebb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a506e8e47a92a21b085850d8f8e04beafe357e2bd51ca10a551685117ca9442
Dec 19 20:03:43 ip-10-0-141-68 machine-config-daemon[192530]: I1219 20:03:43.398731  192530 rpm-ostree.go:366] Running captured: podman create --net=none --annotation=org.openshift.machineconfigoperator.pivot=true --name ostree-container-pivot-92e40e6e-8b5b-4801-9758-00da474ca2de quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:922c0081e73806ce176802944754d6332c8c59da98971cc95c2d4d42d1f527a8
sh-4.4# podman ps -a
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES
sh-4.4# exit
exit
sh-4.2# exit
exit

Removing debug pod ...

Comment 5 errata-xmlrpc 2020-05-04 11:20:43 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581