Bug 1807245
| Summary: | Running httpd under init requires privileged | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Nick Carboni <ncarboni> | |
| Component: | Containers | Assignee: | Peter Hunt <pehunt> | |
| Status: | CLOSED ERRATA | QA Contact: | weiwei jiang <wjiang> | |
| Severity: | low | Docs Contact: | ||
| Priority: | unspecified | |||
| Version: | 4.3.0 | CC: | aos-bugs, cfergeau, dwalsh, gbraad, jnovy, jokerman, kowen, nagrawal, prkumar, tsedmik, tsweeney, umohnani, veillard | |
| Target Milestone: | --- | |||
| Target Release: | 4.5.0 | |||
| Hardware: | x86_64 | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | ||
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 1821510 1821511 (view as bug list) | Environment: | ||
| Last Closed: | 2020-07-13 17:21:31 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1821510, 1821511 | |||
Looks like a regression of https://bugzilla.redhat.com/show_bug.cgi?id=1777838 as the crc VM crio.conf plugin_dirs looks like this in 1.6: plugin_dirs = [ "/var/lib/cni/bin", ] Rather than what it feels like it should be based on https://github.com/openshift/machine-config-operator/pull/1299 But it looks (roughly) the same in 1.5 where things seem to be working for me: plugin_dir = "/var/lib/cni/bin" Possibly things worked differently at that time? Is there anything else I can look for to figure out what's going on? @Nick Does that work with a HA (production cluster) of openshift-4.3.0 ? we don't make any changes in terms of crio configuration as part of CRC. I don't have a production cluster at the moment, but I can try to stand one up today. Assuming it's not a crc-specific issue, what would I do? Can we change the component of this issue or should I open a new one somewhere else? > Can we change the component of this issue or should I open a new one somewhere else?
@Nick I think you should just change the component instead opening again a new issue if in the production cluster you hit the same.
Unfortunately standing up a prod cluster is not something I have the time or resources for it seems. Changing the component to "Containers" as that was the one that was used for bug 1777838. Thanks for the help Praveen. Urvashi could you take a peak at this please? If it's helpful I added a comment to the PR that I think caused this. There's some mention there of oci-systemd-hooks not being needed anymore, but that seems like it's obviously not the case. https://github.com/openshift/machine-config-operator/pull/1314 Hi Nick, I have opened a PR with the fix https://github.com/cri-o/cri-o/pull/3506. Will backport to 1.17 and 1.16 as well, once this gets in. Updated the clone of this bug for 4.3 https://bugzilla.redhat.com/show_bug.cgi?id=1821510#c1 - fix is in cri-o 1.16.5. We have a PR open that adds in new selinux labels for systemd https://github.com/cri-o/cri-o/pull/3764, so should be available in 4.5 once merged. This ensures that we don't need to use the privileged flag or run "setsebool container_manage_cgroup 1" on the node. Will work on backporting this to 4.4z as well. The cri-o PR has been merged and the container-selinux package has been updated in rhcos 4.5. Moving to modified. Setting to Post and assigning to Jindrich for any packaging needs (if any). Tom, cri-o is maintained by Lokesh the cri-o currently in 4.5 has the needed commit, moving to modified Checked with 4.5.0-0.nightly-2020-05-31-230932, moved to verified.
$ oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-10-0-145-122.us-east-2.compute.internal Ready master 35m v1.18.3+9e56094 10.0.145.122 <none> Red Hat Enterprise Linux CoreOS 45.81.202005291504-0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el8
ip-10-0-153-83.us-east-2.compute.internal Ready worker 25m v1.18.3+9e56094 10.0.153.83 <none> Red Hat Enterprise Linux CoreOS 45.81.202005291504-0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el8
ip-10-0-164-40.us-east-2.compute.internal Ready master 35m v1.18.3+9e56094 10.0.164.40 <none> Red Hat Enterprise Linux CoreOS 45.81.202005291504-0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el8
ip-10-0-169-28.us-east-2.compute.internal Ready worker 25m v1.18.3+9e56094 10.0.169.28 <none> Red Hat Enterprise Linux CoreOS 45.81.202005291504-0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el8
ip-10-0-205-87.us-east-2.compute.internal Ready worker 25m v1.18.3+9e56094 10.0.205.87 <none> Red Hat Enterprise Linux CoreOS 45.81.202005291504-0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el8
ip-10-0-209-112.us-east-2.compute.internal Ready master 35m v1.18.3+9e56094 10.0.209.112 <none> Red Hat Enterprise Linux CoreOS 45.81.202005291504-0 (Ootpa) 4.18.0-147.8.1.el8_1.x86_64 cri-o://1.18.1-1.dev.rhaos4.5.git60ac541.el8
$ oc new-build -D $'FROM registry.access.redhat.com/ubi8/ubi-init
RUN dnf -y install httpd; dnf clean all; systemctl enable httpd
CMD [ "/sbin/init" ]' --name init
--> Found container image a858c9c (5 weeks old) from registry.access.redhat.com for "registry.access.redhat.com/ubi8/ubi-init"
Red Hat Universal Base Image 8 Init
-----------------------------------
The Universal Base Image Init is designed is designed to run an init system as PID 1 for running multi-services inside a container. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. T$
is image is maintained by Red Hat and updated regularly.
Tags: base rhel8
* An image stream tag will be created as "ubi-init:latest" that will track the source image
* A Docker build using a predefined Dockerfile will be created
* The resulting image will be pushed to image stream tag "init:latest"
* Every time "ubi-init:latest" changes a new build will be triggered
--> Creating resources with label build=init ...
imagestream.image.openshift.io "ubi-init" created
imagestream.image.openshift.io "init" created
buildconfig.build.openshift.io "init" created
--> Success
$ oc logs -f bc/init
Replaced Dockerfile FROM image registry.access.redhat.com/ubi8/ubi-init
Caching blobs under "/var/cache/blobs".
Pulling image registry.access.redhat.com/ubi8/ubi-init@sha256:d09a4b71ff0676f35b1556fc941ccf627b62182d75d9742ba525c3362f397614 ...
Getting image source signatures
Copying blob sha256:58e1deb9693dfb1704ccce2f1cf0e4d663ac77098a7a0f699708a71549cbd924
Copying blob sha256:78afc5364ad2c981e4a4919f535aaefef9ac2f990837be01c766764e025b1f31
Copying blob sha256:f544909c6b5a5ea2b277e34d7c3ac73a1cb8f127633e8b2155d9e2bae1117fc7
Copying config sha256:a858c9c7ea130b17bad01c858a20f4392085bcc0f25aa5eeee4b16726bed5bab
Writing manifest to image destination
Storing signatures
STEP 1: FROM registry.access.redhat.com/ubi8/ubi-init@sha256:d09a4b71ff0676f35b1556fc941ccf627b62182d75d9742ba525c3362f397614
STEP 2: RUN dnf -y install httpd; dnf clean all; systemctl enable httpd
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
Red Hat Universal Base Image 8 (RPMs) - BaseOS 2.7 MB/s | 766 kB 00:00
Red Hat Universal Base Image 8 (RPMs) - AppStre 17 MB/s | 3.8 MB 00:00
Red Hat Universal Base Image 8 (RPMs) - CodeRea 53 kB/s | 11 kB 00:00
Dependencies resolved.
===========================================================================================
Package Arch Version Repository Size
===========================================================================================
Installing:
httpd x86_64 2.4.37-21.module+el8.2.0+5008+cca404a3 ubi-8-appstream 1.4 M
Installing dependencies:
apr x86_64 1.6.3-9.el8 ubi-8-appstream 125 k
apr-util x86_64 1.6.1-6.el8 ubi-8-appstream 105 k
httpd-filesystem noarch 2.4.37-21.module+el8.2.0+5008+cca404a3 ubi-8-appstream 36 k
httpd-tools x86_64 2.4.37-21.module+el8.2.0+5008+cca404a3 ubi-8-appstream 103 k
mailcap noarch 2.1.48-3.el8 ubi-8-baseos 39 k
mod_http2 x86_64 1.11.3-3.module+el8.2.0+4377+dc421495 ubi-8-appstream 158 k
redhat-logos-httpd noarch 81.1-1.el8 ubi-8-baseos 26 k
Installing weak dependencies:
apr-util-bdb x86_64 1.6.1-6.el8 ubi-8-appstream 25 k
apr-util-openssl x86_64 1.6.1-6.el8 ubi-8-appstream 27 k
Enabling module streams:
httpd 2.4
Transaction Summary
===========================================================================================
Install 10 Packages
Total download size: 2.0 M
Installed size: 5.5 M
Downloading Packages:
(1/10): redhat-logos-httpd-81.1-1.el8.noarch.rp 268 kB/s | 26 kB 00:00
(2/10): apr-util-1.6.1-6.el8.x86_64.rpm 193 kB/s | 105 kB 00:00
(3/10): mailcap-2.1.48-3.el8.noarch.rpm 44 kB/s | 39 kB 00:00
(4/10): apr-1.6.3-9.el8.x86_64.rpm 112 kB/s | 125 kB 00:01
(5/10): httpd-2.4.37-21.module+el8.2.0+5008+cca 11 MB/s | 1.4 MB 00:00
(6/10): mod_http2-1.11.3-3.module+el8.2.0+4377+ 3.8 MB/s | 158 kB 00:00
(7/10): httpd-filesystem-2.4.37-21.module+el8.2 905 kB/s | 36 kB 00:00
(8/10): apr-util-bdb-1.6.1-6.el8.x86_64.rpm 52 kB/s | 25 kB 00:00
(9/10): httpd-tools-2.4.37-21.module+el8.2.0+50 2.1 MB/s | 103 kB 00:00
(10/10): apr-util-openssl-1.6.1-6.el8.x86_64.rp 35 kB/s | 27 kB 00:00
--------------------------------------------------------------------------------
Total 1.4 MB/s | 2.0 MB 00:01
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : apr-1.6.3-9.el8.x86_64 1/10
Running scriptlet: apr-1.6.3-9.el8.x86_64 1/10
Installing : apr-util-openssl-1.6.1-6.el8.x86_64 2/10
Installing : apr-util-bdb-1.6.1-6.el8.x86_64 3/10
Installing : apr-util-1.6.1-6.el8.x86_64 4/10
Running scriptlet: apr-util-1.6.1-6.el8.x86_64 4/10
Installing : httpd-tools-2.4.37-21.module+el8.2.0+5008+cca404a3 5/10
Running scriptlet: httpd-filesystem-2.4.37-21.module+el8.2.0+5008+cca 6/10
Installing : httpd-filesystem-2.4.37-21.module+el8.2.0+5008+cca 6/10
Installing : mailcap-2.1.48-3.el8.noarch 7/10
Installing : redhat-logos-httpd-81.1-1.el8.noarch 8/10
Installing : mod_http2-1.11.3-3.module+el8.2.0+4377+dc421495.x8 9/10
Installing : httpd-2.4.37-21.module+el8.2.0+5008+cca404a3.x86_6 10/10
Running scriptlet: httpd-2.4.37-21.module+el8.2.0+5008+cca404a3.x86_6 10/10
Verifying : redhat-logos-httpd-81.1-1.el8.noarch 1/10
Verifying : mailcap-2.1.48-3.el8.noarch 2/10
Verifying : apr-1.6.3-9.el8.x86_64 3/10
Verifying : apr-util-1.6.1-6.el8.x86_64 4/10
Verifying : apr-util-openssl-1.6.1-6.el8.x86_64 5/10
Verifying : apr-util-bdb-1.6.1-6.el8.x86_64 6/10
Verifying : httpd-2.4.37-21.module+el8.2.0+5008+cca404a3.x86_6 7/10
Verifying : mod_http2-1.11.3-3.module+el8.2.0+4377+dc421495.x8 8/10
Verifying : httpd-filesystem-2.4.37-21.module+el8.2.0+5008+cca 9/10
Verifying : httpd-tools-2.4.37-21.module+el8.2.0+5008+cca404a3 10/10
Installed products updated.
Installed:
apr-1.6.3-9.el8.x86_64
apr-util-1.6.1-6.el8.x86_64
apr-util-bdb-1.6.1-6.el8.x86_64
apr-util-openssl-1.6.1-6.el8.x86_64
httpd-2.4.37-21.module+el8.2.0+5008+cca404a3.x86_64
httpd-filesystem-2.4.37-21.module+el8.2.0+5008+cca404a3.noarch
httpd-tools-2.4.37-21.module+el8.2.0+5008+cca404a3.x86_64
mailcap-2.1.48-3.el8.noarch
mod_http2-1.11.3-3.module+el8.2.0+4377+dc421495.x86_64
redhat-logos-httpd-81.1-1.el8.noarch
Complete!
Updating Subscription Management repositories.
Unable to read consumer identity
Subscription Manager is operating in container mode.
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
24 files removed
Created symlink /etc/systemd/system/multi-user.target.wants/httpd.service → /usr/lib/systemd/system/httpd.service.
time="2020-06-01T07:05:01Z" level=info msg="Image operating system mismatch: image uses \"\", expecting \"linux\""
time="2020-06-01T07:05:01Z" level=info msg="Image architecture mismatch: image uses \"\", expecting \"amd64\""
--> 7cd24bfc2f8
STEP 3: CMD ["/sbin/init"]
time="2020-06-01T07:05:02Z" level=info msg="Image operating system mismatch: image uses \"\", expecting \"linux\""
time="2020-06-01T07:05:02Z" level=info msg="Image architecture mismatch: image uses \"\", expecting \"amd64\""
--> b035b88b7d0
STEP 4: ENV "OPENSHIFT_BUILD_NAME"="init-1" "OPENSHIFT_BUILD_NAMESPACE"="default"
time="2020-06-01T07:05:02Z" level=info msg="Image operating system mismatch: image uses \"\", expecting \"linux\""
time="2020-06-01T07:05:02Z" level=info msg="Image architecture mismatch: image uses \"\", expecting \"amd64\""
--> ea2f3f02317
STEP 5: LABEL "io.openshift.build.name"="init-1" "io.openshift.build.namespace"="default"
STEP 6: COMMIT temp.builder.openshift.io/default/init-1:4d4c89ab
time="2020-06-01T07:05:02Z" level=info msg="Image operating system mismatch: image uses \"\", expecting \"linux\""
time="2020-06-01T07:05:02Z" level=info msg="Image architecture mismatch: image uses \"\", expecting \"amd64\""
--> b0d7cae3a7d
b0d7cae3a7d341390a0be8a2831eeb67ce337186d1c072947bba6fd71cf5f6de
Pushing image image-registry.openshift-image-registry.svc:5000/default/init:latest ...
Getting image source signatures
Copying blob sha256:aa47ff49eacebbb09bd5eb7e643daf5efa62ec92eb4fd2471ec708ad4d36b8bd
Copying blob sha256:78afc5364ad2c981e4a4919f535aaefef9ac2f990837be01c766764e025b1f31
Copying blob sha256:f544909c6b5a5ea2b277e34d7c3ac73a1cb8f127633e8b2155d9e2bae1117fc7
Copying blob sha256:58e1deb9693dfb1704ccce2f1cf0e4d663ac77098a7a0f699708a71549cbd924
Copying config sha256:b0d7cae3a7d341390a0be8a2831eeb67ce337186d1c072947bba6fd71cf5f6de
Writing manifest to image destination
Storing signatures
Successfully pushed image-registry.openshift-image-registry.svc:5000/default/init@sha256:81464bf0ab77da7b2c1bcaf8d8e67ad234c53ea8e73945a9304838c717eb4a18
Push successful
$ oc get imagestream
NAME IMAGE REPOSITORY TAGS UPDATED
init image-registry.openshift-image-registry.svc:5000/default/init latest 9 seconds ago
ubi-init image-registry.openshift-image-registry.svc:5000/default/ubi-init latest About a minute ago
$ oc new-app -i init
--> Found image b0d7cae (2 minutes old) in image stream "default/init" under tag "latest" for "init"
Red Hat Universal Base Image 8 Init
-----------------------------------
The Universal Base Image Init is designed is designed to run an init system as PID 1 for running multi-services inside a container. This base image is freely redistributable, but Red Hat only supports Red Ha
t technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.
Tags: base rhel8
--> Creating resources ...
deployment.apps "init" created
--> Success
Run 'oc status' to view your app.
$ oc get pods
NAME READY STATUS RESTARTS AGE
init-1-build 0/1 Completed 0 4m42s
init-776756768b-fk9tk 1/1 Running 0 29s
$ oc describe pods init-776756768b-fk9tk
Name: init-776756768b-fk9tk
Namespace: default
Priority: 0
Node: ip-10-0-153-83.us-east-2.compute.internal/10.0.153.83
Start Time: Mon, 01 Jun 2020 15:07:57 +0800
Labels: deployment=init
pod-template-hash=776756768b
Annotations: k8s.v1.cni.cncf.io/network-status:
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.129.2.10"
],
"default": true,
"dns": {}
}]
k8s.v1.cni.cncf.io/networks-status:
[{
"name": "openshift-sdn",
"interface": "eth0",
"ips": [
"10.129.2.10"
],
"default": true,
"dns": {}
}]
Status: Running
IP: 10.129.2.10
IPs:
IP: 10.129.2.10
Controlled By: ReplicaSet/init-776756768b
Containers:
init:
Container ID: cri-o://79c92b2e91ddf73cb0a7fc22c11e7c64cf0bb3942ce9a6c6f46c508534da6d93
Image: image-registry.openshift-image-registry.svc:5000/default/init@sha256:81464bf0ab77da7b2c1bcaf8d8e67ad234c53ea8e73945a9304838c717eb4a18
Image ID: image-registry.openshift-image-registry.svc:5000/default/init@sha256:81464bf0ab77da7b2c1bcaf8d8e67ad234c53ea8e73945a9304838c717eb4a18
Port: <none>
Host Port: <none>
State: Running
Started: Mon, 01 Jun 2020 15:08:10 +0800
Ready: True
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-z7wcl (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
default-token-z7wcl:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-z7wcl
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned default/init-776756768b-fk9tk to ip-10-0-153-83.us-east-2.compute.internal
Normal AddedInterface 33s multus Add eth0 [10.129.2.10/23]
Normal Pulling 33s kubelet, ip-10-0-153-83.us-east-2.compute.internal Pulling image "image-registry.openshift-image-registry.svc:5000/default/init@sha256:81464bf0ab77da7b2c1bcaf8d8e67ad234c53ea8e73945a9304838c717eb4a18"
Normal Pulled 22s kubelet, ip-10-0-153-83.us-east-2.compute.internal Successfully pulled image "image-registry.openshift-image-registry.svc:5000/default/init@sha256:81464bf0ab77da7b2c1bcaf8d8e67ad234c53ea8e73945a9304838c717eb4a18"
Normal Created 22s kubelet, ip-10-0-153-83.us-east-2.compute.internal Created container init
Normal Started 22s kubelet, ip-10-0-153-83.us-east-2.compute.internal Started container init
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409 |
Description of problem: In crc 1.6 running a container with httpd as a systemd service requires the privileged scc. In crc 1.5 and OCP 3.11 this worked with anyuid Version-Release number of selected component (if applicable): $ crc version crc version: 1.6.0+8ef676f OpenShift version: 4.3.0 (embedded in binary) Container runtime: cri-o://1.16.2-6.dev.rhaos4.3.git9e3db66.el8 How reproducible: Always Steps to Reproduce: 1. Build the following Dockerfile FROM registry.access.redhat.com/ubi8/ubi-init RUN dnf -y install httpd; dnf clean all; systemctl enable httpd CMD [ "/sbin/init" ] 2. Deploy the container using crc 1.6 and the anyuid scc I used the following deployment: apiVersion: apps/v1 kind: Deployment metadata: name: test-httpd spec: replicas: 1 selector: matchLabels: name: test-httpd template: metadata: name: test-httpd labels: name: test-httpd spec: containers: - name: test-httpd image: image-registry.openshift-image-registry.svc:5000/test-httpd/test-httpd:latest ports: - containerPort: 80 protocol: TCP - containerPort: 8080 protocol: TCP livenessProbe: exec: command: - pidof - httpd initialDelaySeconds: 15 timeoutSeconds: 3 readinessProbe: tcpSocket: port: 80 initialDelaySeconds: 10 timeoutSeconds: 3 serviceAccountName: anyuid Actual results: The pod fails to become ready and `oc exec <pod-name> -- systemctl status httpd` returns the following error: System has not been booted with systemd as init system (PID 1). Can't operate. Failed to connect to bus: Host is down command terminated with exit code 1 Expected results: The pod becomes ready. Additional info: This previously worked for me using crc version 1.5 and a cri-o node in openshift enterprise 3.11 (container runtime: cri-o://1.11.16-0.5.dev.rhaos3.11.git3f89eba.el7-dev) Full reproduction steps including buildconfigs and a Dockerfile can be found in https://github.com/carbonin/crio-systemd-test/blob/master/README.md