Description of problem: In crc 1.6 running a container with httpd as a systemd service requires the privileged scc. In crc 1.5 and OCP 3.11 this worked with anyuid Version-Release number of selected component (if applicable): $ crc version crc version: 1.6.0+8ef676f OpenShift version: 4.3.0 (embedded in binary) Container runtime: cri-o://1.16.2-6.dev.rhaos4.3.git9e3db66.el8 How reproducible: Always Steps to Reproduce: 1. Build the following Dockerfile FROM registry.access.redhat.com/ubi8/ubi-init RUN dnf -y install httpd; dnf clean all; systemctl enable httpd CMD [ "/sbin/init" ] 2. Deploy the container using crc 1.6 and the anyuid scc I used the following deployment: apiVersion: apps/v1 kind: Deployment metadata: name: test-httpd spec: replicas: 1 selector: matchLabels: name: test-httpd template: metadata: name: test-httpd labels: name: test-httpd spec: containers: - name: test-httpd image: image-registry.openshift-image-registry.svc:5000/test-httpd/test-httpd:latest ports: - containerPort: 80 protocol: TCP - containerPort: 8080 protocol: TCP livenessProbe: exec: command: - pidof - httpd initialDelaySeconds: 15 timeoutSeconds: 3 readinessProbe: tcpSocket: port: 80 initialDelaySeconds: 10 timeoutSeconds: 3 serviceAccountName: anyuid Actual results: The pod fails to become ready and `oc exec <pod-name> -- systemctl status httpd` returns the following error: System has not been booted with systemd as init system (PID 1). Can't operate. Failed to connect to bus: Host is down command terminated with exit code 1 Expected results: The pod becomes ready. Additional info: This previously worked for me using crc version 1.5 and a cri-o node in openshift enterprise 3.11 (container runtime: cri-o://1.11.16-0.5.dev.rhaos3.11.git3f89eba.el7-dev) Full reproduction steps including buildconfigs and a Dockerfile can be found in https://github.com/carbonin/crio-systemd-test/blob/master/README.md
Looks like a regression of https://bugzilla.redhat.com/show_bug.cgi?id=1777838 as the crc VM crio.conf plugin_dirs looks like this in 1.6: plugin_dirs = [ "/var/lib/cni/bin", ] Rather than what it feels like it should be based on https://github.com/openshift/machine-config-operator/pull/1299 But it looks (roughly) the same in 1.5 where things seem to be working for me: plugin_dir = "/var/lib/cni/bin" Possibly things worked differently at that time? Is there anything else I can look for to figure out what's going on?
@Nick Does that work with a HA (production cluster) of openshift-4.3.0 ? we don't make any changes in terms of crio configuration as part of CRC.
I don't have a production cluster at the moment, but I can try to stand one up today. Assuming it's not a crc-specific issue, what would I do? Can we change the component of this issue or should I open a new one somewhere else?
> Can we change the component of this issue or should I open a new one somewhere else? @Nick I think you should just change the component instead opening again a new issue if in the production cluster you hit the same.
Unfortunately standing up a prod cluster is not something I have the time or resources for it seems. Changing the component to "Containers" as that was the one that was used for bug 1777838. Thanks for the help Praveen.
Urvashi could you take a peak at this please?
If it's helpful I added a comment to the PR that I think caused this. There's some mention there of oci-systemd-hooks not being needed anymore, but that seems like it's obviously not the case. https://github.com/openshift/machine-config-operator/pull/1314
Hi Nick, I have opened a PR with the fix https://github.com/cri-o/cri-o/pull/3506. Will backport to 1.17 and 1.16 as well, once this gets in.
Updated the clone of this bug for 4.3 https://bugzilla.redhat.com/show_bug.cgi?id=1821510#c1 - fix is in cri-o 1.16.5.
We have a PR open that adds in new selinux labels for systemd https://github.com/cri-o/cri-o/pull/3764, so should be available in 4.5 once merged. This ensures that we don't need to use the privileged flag or run "setsebool container_manage_cgroup 1" on the node. Will work on backporting this to 4.4z as well.
The cri-o PR has been merged and the container-selinux package has been updated in rhcos 4.5. Moving to modified.
Setting to Post and assigning to Jindrich for any packaging needs (if any).
Tom, cri-o is maintained by Lokesh
the cri-o currently in 4.5 has the needed commit, moving to modified
Checked with 4.5.0-0.nightly-2020-05-31-230932, moved to verified. $ oc get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-10-0-145-122.us-east-2.compute.internal Ready master 35m v1.18.3+9e56094 10.0.145.122 <none> Red Hat Enterprise Linux CoreOS 45.81.202005291504-0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el8 ip-10-0-153-83.us-east-2.compute.internal Ready worker 25m v1.18.3+9e56094 10.0.153.83 <none> Red Hat Enterprise Linux CoreOS 45.81.202005291504-0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el8 ip-10-0-164-40.us-east-2.compute.internal Ready master 35m v1.18.3+9e56094 10.0.164.40 <none> Red Hat Enterprise Linux CoreOS 45.81.202005291504-0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el8 ip-10-0-169-28.us-east-2.compute.internal Ready worker 25m v1.18.3+9e56094 10.0.169.28 <none> Red Hat Enterprise Linux CoreOS 45.81.202005291504-0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el8 ip-10-0-205-87.us-east-2.compute.internal Ready worker 25m v1.18.3+9e56094 10.0.205.87 <none> Red Hat Enterprise Linux CoreOS 45.81.202005291504-0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el8 ip-10-0-209-112.us-east-2.compute.internal Ready master 35m v1.18.3+9e56094 10.0.209.112 <none> Red Hat Enterprise Linux CoreOS 45.81.202005291504-0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el8 $ oc new-build -D $'FROM registry.access.redhat.com/ubi8/ubi-init RUN dnf -y install httpd; dnf clean all; systemctl enable httpd CMD [ "/sbin/init" ]' --name init --> Found container image a858c9c (5 weeks old) from registry.access.redhat.com for "registry.access.redhat.com/ubi8/ubi-init" Red Hat Universal Base Image 8 Init ----------------------------------- The Universal Base Image Init is designed is designed to run an init system as PID 1 for running multi-services inside a container. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. T$ is image is maintained by Red Hat and updated regularly. Tags: base rhel8 * An image stream tag will be created as "ubi-init:latest" that will track the source image * A Docker build using a predefined Dockerfile will be created * The resulting image will be pushed to image stream tag "init:latest" * Every time "ubi-init:latest" changes a new build will be triggered --> Creating resources with label build=init ... imagestream.image.openshift.io "ubi-init" created imagestream.image.openshift.io "init" created buildconfig.build.openshift.io "init" created --> Success $ oc logs -f bc/init Replaced Dockerfile FROM image registry.access.redhat.com/ubi8/ubi-init Caching blobs under "/var/cache/blobs". Pulling image registry.access.redhat.com/ubi8/ubi-init@sha256:d09a4b71ff0676f35b1556fc941ccf627b62182d75d9742ba525c3362f397614 ... Getting image source signatures Copying blob sha256:58e1deb9693dfb1704ccce2f1cf0e4d663ac77098a7a0f699708a71549cbd924 Copying blob sha256:78afc5364ad2c981e4a4919f535aaefef9ac2f990837be01c766764e025b1f31 Copying blob sha256:f544909c6b5a5ea2b277e34d7c3ac73a1cb8f127633e8b2155d9e2bae1117fc7 Copying config sha256:a858c9c7ea130b17bad01c858a20f4392085bcc0f25aa5eeee4b16726bed5bab Writing manifest to image destination Storing signatures STEP 1: FROM registry.access.redhat.com/ubi8/ubi-init@sha256:d09a4b71ff0676f35b1556fc941ccf627b62182d75d9742ba525c3362f397614 STEP 2: RUN dnf -y install httpd; dnf clean all; systemctl enable httpd Updating Subscription Management repositories. Unable to read consumer identity Subscription Manager is operating in container mode. This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Red Hat Universal Base Image 8 (RPMs) - BaseOS 2.7 MB/s | 766 kB 00:00 Red Hat Universal Base Image 8 (RPMs) - AppStre 17 MB/s | 3.8 MB 00:00 Red Hat Universal Base Image 8 (RPMs) - CodeRea 53 kB/s | 11 kB 00:00 Dependencies resolved. =========================================================================================== Package Arch Version Repository Size =========================================================================================== Installing: httpd x86_64 2.4.37-21.module+el8.2.0+5008+cca404a3 ubi-8-appstream 1.4 M Installing dependencies: apr x86_64 1.6.3-9.el8 ubi-8-appstream 125 k apr-util x86_64 1.6.1-6.el8 ubi-8-appstream 105 k httpd-filesystem noarch 2.4.37-21.module+el8.2.0+5008+cca404a3 ubi-8-appstream 36 k httpd-tools x86_64 2.4.37-21.module+el8.2.0+5008+cca404a3 ubi-8-appstream 103 k mailcap noarch 2.1.48-3.el8 ubi-8-baseos 39 k mod_http2 x86_64 1.11.3-3.module+el8.2.0+4377+dc421495 ubi-8-appstream 158 k redhat-logos-httpd noarch 81.1-1.el8 ubi-8-baseos 26 k Installing weak dependencies: apr-util-bdb x86_64 1.6.1-6.el8 ubi-8-appstream 25 k apr-util-openssl x86_64 1.6.1-6.el8 ubi-8-appstream 27 k Enabling module streams: httpd 2.4 Transaction Summary =========================================================================================== Install 10 Packages Total download size: 2.0 M Installed size: 5.5 M Downloading Packages: (1/10): redhat-logos-httpd-81.1-1.el8.noarch.rp 268 kB/s | 26 kB 00:00 (2/10): apr-util-1.6.1-6.el8.x86_64.rpm 193 kB/s | 105 kB 00:00 (3/10): mailcap-2.1.48-3.el8.noarch.rpm 44 kB/s | 39 kB 00:00 (4/10): apr-1.6.3-9.el8.x86_64.rpm 112 kB/s | 125 kB 00:01 (5/10): httpd-2.4.37-21.module+el8.2.0+5008+cca 11 MB/s | 1.4 MB 00:00 (6/10): mod_http2-1.11.3-3.module+el8.2.0+4377+ 3.8 MB/s | 158 kB 00:00 (7/10): httpd-filesystem-2.4.37-21.module+el8.2 905 kB/s | 36 kB 00:00 (8/10): apr-util-bdb-1.6.1-6.el8.x86_64.rpm 52 kB/s | 25 kB 00:00 (9/10): httpd-tools-2.4.37-21.module+el8.2.0+50 2.1 MB/s | 103 kB 00:00 (10/10): apr-util-openssl-1.6.1-6.el8.x86_64.rp 35 kB/s | 27 kB 00:00 -------------------------------------------------------------------------------- Total 1.4 MB/s | 2.0 MB 00:01 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : apr-1.6.3-9.el8.x86_64 1/10 Running scriptlet: apr-1.6.3-9.el8.x86_64 1/10 Installing : apr-util-openssl-1.6.1-6.el8.x86_64 2/10 Installing : apr-util-bdb-1.6.1-6.el8.x86_64 3/10 Installing : apr-util-1.6.1-6.el8.x86_64 4/10 Running scriptlet: apr-util-1.6.1-6.el8.x86_64 4/10 Installing : httpd-tools-2.4.37-21.module+el8.2.0+5008+cca404a3 5/10 Running scriptlet: httpd-filesystem-2.4.37-21.module+el8.2.0+5008+cca 6/10 Installing : httpd-filesystem-2.4.37-21.module+el8.2.0+5008+cca 6/10 Installing : mailcap-2.1.48-3.el8.noarch 7/10 Installing : redhat-logos-httpd-81.1-1.el8.noarch 8/10 Installing : mod_http2-1.11.3-3.module+el8.2.0+4377+dc421495.x8 9/10 Installing : httpd-2.4.37-21.module+el8.2.0+5008+cca404a3.x86_6 10/10 Running scriptlet: httpd-2.4.37-21.module+el8.2.0+5008+cca404a3.x86_6 10/10 Verifying : redhat-logos-httpd-81.1-1.el8.noarch 1/10 Verifying : mailcap-2.1.48-3.el8.noarch 2/10 Verifying : apr-1.6.3-9.el8.x86_64 3/10 Verifying : apr-util-1.6.1-6.el8.x86_64 4/10 Verifying : apr-util-openssl-1.6.1-6.el8.x86_64 5/10 Verifying : apr-util-bdb-1.6.1-6.el8.x86_64 6/10 Verifying : httpd-2.4.37-21.module+el8.2.0+5008+cca404a3.x86_6 7/10 Verifying : mod_http2-1.11.3-3.module+el8.2.0+4377+dc421495.x8 8/10 Verifying : httpd-filesystem-2.4.37-21.module+el8.2.0+5008+cca 9/10 Verifying : httpd-tools-2.4.37-21.module+el8.2.0+5008+cca404a3 10/10 Installed products updated. Installed: apr-1.6.3-9.el8.x86_64 apr-util-1.6.1-6.el8.x86_64 apr-util-bdb-1.6.1-6.el8.x86_64 apr-util-openssl-1.6.1-6.el8.x86_64 httpd-2.4.37-21.module+el8.2.0+5008+cca404a3.x86_64 httpd-filesystem-2.4.37-21.module+el8.2.0+5008+cca404a3.noarch httpd-tools-2.4.37-21.module+el8.2.0+5008+cca404a3.x86_64 mailcap-2.1.48-3.el8.noarch mod_http2-1.11.3-3.module+el8.2.0+4377+dc421495.x86_64 redhat-logos-httpd-81.1-1.el8.noarch Complete! Updating Subscription Management repositories. Unable to read consumer identity Subscription Manager is operating in container mode. This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. 24 files removed Created symlink /etc/systemd/system/multi-user.target.wants/httpd.service → /usr/lib/systemd/system/httpd.service. time="2020-06-01T07:05:01Z" level=info msg="Image operating system mismatch: image uses \"\", expecting \"linux\"" time="2020-06-01T07:05:01Z" level=info msg="Image architecture mismatch: image uses \"\", expecting \"amd64\"" --> 7cd24bfc2f8 STEP 3: CMD ["/sbin/init"] time="2020-06-01T07:05:02Z" level=info msg="Image operating system mismatch: image uses \"\", expecting \"linux\"" time="2020-06-01T07:05:02Z" level=info msg="Image architecture mismatch: image uses \"\", expecting \"amd64\"" --> b035b88b7d0 STEP 4: ENV "OPENSHIFT_BUILD_NAME"="init-1" "OPENSHIFT_BUILD_NAMESPACE"="default" time="2020-06-01T07:05:02Z" level=info msg="Image operating system mismatch: image uses \"\", expecting \"linux\"" time="2020-06-01T07:05:02Z" level=info msg="Image architecture mismatch: image uses \"\", expecting \"amd64\"" --> ea2f3f02317 STEP 5: LABEL "io.openshift.build.name"="init-1" "io.openshift.build.namespace"="default" STEP 6: COMMIT temp.builder.openshift.io/default/init-1:4d4c89ab time="2020-06-01T07:05:02Z" level=info msg="Image operating system mismatch: image uses \"\", expecting \"linux\"" time="2020-06-01T07:05:02Z" level=info msg="Image architecture mismatch: image uses \"\", expecting \"amd64\"" --> b0d7cae3a7d b0d7cae3a7d341390a0be8a2831eeb67ce337186d1c072947bba6fd71cf5f6de Pushing image image-registry.openshift-image-registry.svc:5000/default/init:latest ... Getting image source signatures Copying blob sha256:aa47ff49eacebbb09bd5eb7e643daf5efa62ec92eb4fd2471ec708ad4d36b8bd Copying blob sha256:78afc5364ad2c981e4a4919f535aaefef9ac2f990837be01c766764e025b1f31 Copying blob sha256:f544909c6b5a5ea2b277e34d7c3ac73a1cb8f127633e8b2155d9e2bae1117fc7 Copying blob sha256:58e1deb9693dfb1704ccce2f1cf0e4d663ac77098a7a0f699708a71549cbd924 Copying config sha256:b0d7cae3a7d341390a0be8a2831eeb67ce337186d1c072947bba6fd71cf5f6de Writing manifest to image destination Storing signatures Successfully pushed image-registry.openshift-image-registry.svc:5000/default/init@sha256:81464bf0ab77da7b2c1bcaf8d8e67ad234c53ea8e73945a9304838c717eb4a18 Push successful $ oc get imagestream NAME IMAGE REPOSITORY TAGS UPDATED init image-registry.openshift-image-registry.svc:5000/default/init latest 9 seconds ago ubi-init image-registry.openshift-image-registry.svc:5000/default/ubi-init latest About a minute ago $ oc new-app -i init --> Found image b0d7cae (2 minutes old) in image stream "default/init" under tag "latest" for "init" Red Hat Universal Base Image 8 Init ----------------------------------- The Universal Base Image Init is designed is designed to run an init system as PID 1 for running multi-services inside a container. This base image is freely redistributable, but Red Hat only supports Red Ha t technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly. Tags: base rhel8 --> Creating resources ... deployment.apps "init" created --> Success Run 'oc status' to view your app. $ oc get pods NAME READY STATUS RESTARTS AGE init-1-build 0/1 Completed 0 4m42s init-776756768b-fk9tk 1/1 Running 0 29s $ oc describe pods init-776756768b-fk9tk Name: init-776756768b-fk9tk Namespace: default Priority: 0 Node: ip-10-0-153-83.us-east-2.compute.internal/10.0.153.83 Start Time: Mon, 01 Jun 2020 15:07:57 +0800 Labels: deployment=init pod-template-hash=776756768b Annotations: k8s.v1.cni.cncf.io/network-status: [{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.129.2.10" ], "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: [{ "name": "openshift-sdn", "interface": "eth0", "ips": [ "10.129.2.10" ], "default": true, "dns": {} }] Status: Running IP: 10.129.2.10 IPs: IP: 10.129.2.10 Controlled By: ReplicaSet/init-776756768b Containers: init: Container ID: cri-o://79c92b2e91ddf73cb0a7fc22c11e7c64cf0bb3942ce9a6c6f46c508534da6d93 Image: image-registry.openshift-image-registry.svc:5000/default/init@sha256:81464bf0ab77da7b2c1bcaf8d8e67ad234c53ea8e73945a9304838c717eb4a18 Image ID: image-registry.openshift-image-registry.svc:5000/default/init@sha256:81464bf0ab77da7b2c1bcaf8d8e67ad234c53ea8e73945a9304838c717eb4a18 Port: <none> Host Port: <none> State: Running Started: Mon, 01 Jun 2020 15:08:10 +0800 Ready: True Restart Count: 0 Environment: <none> Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-z7wcl (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: default-token-z7wcl: Type: Secret (a volume populated by a Secret) SecretName: default-token-z7wcl Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned default/init-776756768b-fk9tk to ip-10-0-153-83.us-east-2.compute.internal Normal AddedInterface 33s multus Add eth0 [10.129.2.10/23] Normal Pulling 33s kubelet, ip-10-0-153-83.us-east-2.compute.internal Pulling image "image-registry.openshift-image-registry.svc:5000/default/init@sha256:81464bf0ab77da7b2c1bcaf8d8e67ad234c53ea8e73945a9304838c717eb4a18" Normal Pulled 22s kubelet, ip-10-0-153-83.us-east-2.compute.internal Successfully pulled image "image-registry.openshift-image-registry.svc:5000/default/init@sha256:81464bf0ab77da7b2c1bcaf8d8e67ad234c53ea8e73945a9304838c717eb4a18" Normal Created 22s kubelet, ip-10-0-153-83.us-east-2.compute.internal Created container init Normal Started 22s kubelet, ip-10-0-153-83.us-east-2.compute.internal Started container init
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409