Bug 1955194

Summary: ocp disconnected mode machine config pool mark as degraded
Product: OpenShift Container Platform Reporter: Sebastian Scheinkman <sscheink>
Component: Machine Config OperatorAssignee: Yu Qi Zhang <jerzhang>
Status: CLOSED DUPLICATE QA Contact: Michael Nguyen <mnguyen>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.8   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-03 17:10:01 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Sebastian Scheinkman 2021-04-29 16:25:46 UTC 
Description of problem:

on disconnected mode with ImageContentSourcePolicy and mirror the nodes mode to degraded when try to install rt kernel.

logs from the machine config daemon

```
I0429 14:53:05.046167  752470 run.go:18] Running: nice -- ionice -c 3 oc image extract --path /:/run/mco-machine-os-content/os-content-215884633 --registry-config /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:826588080c4a98587f2cb45be94d9f162df2cfb3e988dd1de2e4d49de7a96aa3
error: unable to connect to image repository quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:826588080c4a98587f2cb45be94d9f162df2cfb3e988dd1de2e4d49de7a96aa3: Get "https://quay.io/v2/": dial tcp 34.224.196.162:443: connect: network is unreachable
W0429 14:53:05.101961  752470 run.go:44] nice failed: running nice -- ionice -c 3 oc image extract --path /:/run/mco-machine-os-content/os-content-215884633 --registry-config /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:826588080c4a98587f2cb45be94d9f162df2cfb3e988dd1de2e4d49de7a96aa3 failed: error: unable to connect to image repository quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:826588080c4a98587f2cb45be94d9f162df2cfb3e988dd1de2e4d49de7a96aa3: Get "https://quay.io/v2/": dial tcp 34.224.196.162:443: connect: network is unreachable
: exit status 1; retrying...
I0429 14:53:05.101990  752470 update.go:314] Falling back to using podman cp to fetch OS image content
I0429 14:53:05.102021  752470 run.go:18] Running: nice -- ionice -c 3 podman pull -q --authfile /var/lib/kubelet/config.json quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:826588080c4a98587f2cb45be94d9f162df2cfb3e988dd1de2e4d49de7a96aa3
d341d1cabe18c3ca6b1295549c2b81a09afa0bbddaeb5bde42ee72605760d77a
I0429 14:53:05.330641  752470 rpm-ostree.go:258] Running captured: podman create --net=none --annotation=org.openshift.machineconfigoperator.pivot=true --name ostree-container-pivot-77405bf3-4e66-44b3-a93f-e6bc70dd8cc1 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:826588080c4a98587f2cb45be94d9f162df2cfb3e988dd1de2e4d49de7a96aa3
I0429 14:53:05.496648  752470 run.go:18] Running: nice -- ionice -c 3 podman cp 770bc9d09df714fc17edb14ad108ab932855346abd01240161567b33dfa548b1:/ /run/mco-machine-os-content/os-content-215884633
I0429 14:53:10.086698  752470 rpm-ostree.go:258] Running captured: chcon -R -t var_run_t /run/mco-machine-os-content/os-content-215884633
I0429 14:53:10.087770  752470 update.go:278] Error changing selinux context on path /run/mco-machine-os-content/os-content-215884633  error running chcon -R -t var_run_t /run/mco-machine-os-content/os-content-215884633: chcon: cannot access '/run/mco-machine-os-content/os-content-215884633': No such file or directory

```

The problem is when the daemon tries to fallback to podman the cp command doesn't create the folder for the os-content an put the files under the parent folder then the next step for selinux fails

Error changing selinux context on path /run/mco-machine-os-content/os-content-215884633  error running chcon -R -t var_run_t /run/mco-machine-os-content/os-content-215884633: chcon: cannot access '/run/mco-machine-os-content/os-content-215884633': No such file or directory
Comment 1 Yu Qi Zhang 2021-04-29 16:42:07 UTC 
This should be a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1950112, are you still seeing this in the latest nightlies?
Comment 2 Sebastian Scheinkman 2021-04-29 17:00:47 UTC 
Hi Yu,

thanks for the comment I am running

Client Version: 4.8.0-0.nightly-2021-04-18-203506
Server Version: 4.8.0-0.nightly-2021-04-18-203506
Kubernetes Version: v1.21.0-rc.0+2993be8


I will try to redeploy my environment and check it again

Thanks!
Sebastian
Comment 3 Yu Qi Zhang 2021-05-03 17:10:01 UTC 

*** This bug has been marked as a duplicate of bug 1950112 ***