Bug 1959564 - Test verify /run filesystem contents failing
Summary: Test verify /run filesystem contents failing
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 4.8
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.8.0
Assignee: Gabe Montero
QA Contact: XiuJuan Wang
URL:
Whiteboard:
Depends On:
Blocks: 1963115
TreeView+ depends on / blocked
 
Reported: 2021-05-11 19:08 UTC by ravig
Modified: 2021-07-27 23:08 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-27 23:08:01 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 26156 0 None closed Bug 1959564: builds: skip /run verification tests 2021-05-25 15:57:02 UTC
Github openshift origin pull 26164 0 None closed Bug 1959564: move verify run test from tools to cli image to avoid additional /run content 2021-05-25 15:57:04 UTC
Github openshift origin pull 26181 0 None open Bug 1959564: narrow verify /run test to /run/secrets 2021-05-25 16:08:40 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:08:18 UTC

Description ravig 2021-05-11 19:08:10 UTC
Description of problem:

https://search.ci.openshift.org/?search=+verify+%2Frun+filesystem+contents&maxAge=48h&context=1&type=all&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job

CI jobs are failing because of the above test failing

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 Adam Kaplan 2021-05-11 20:03:21 UTC
`/run` on a UBI8 container now seems to contain a lot more stuff:

```
/run:
console
cryptsetup
faillock
lock
log
rhsm
secrets
sepermit
setrans
systemd
user

/run/console:

/run/cryptsetup:

/run/faillock:

/run/lock:
subsys

/run/lock/subsys:

/run/log:

/run/rhsm:

/run/secrets:
rhsm

/run/secrets/rhsm:
ca

/run/secrets/rhsm/ca:
redhat-entitlement-authority.pem
redhat-uep.pem

/run/sepermit:

/run/setrans:

/run/systemd:
ask-password
machines
seats
sessions
shutdown
users

/run/systemd/ask-password:

/run/systemd/machines:

/run/systemd/seats:

/run/systemd/sessions:

/run/systemd/shutdown:

/run/systemd/users:

```

See https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_oc/821/pull-ci-openshift-oc-master-e2e-aws/1391875953382133760

Comment 6 Scott McCarty 2021-05-13 13:29:10 UTC
I don't think this is from a change to UBI: 

[root@keith-dc2-crunchtools-com ~]# podman run -it ubi8 bash
[root@fc2074814725 /]# find /run/
/run/
/run/lock
/run/.containerenv
/run/secrets
/run/secrets/rhsm
/run/secrets/rhsm/syspurpose
/run/secrets/rhsm/syspurpose/valid_fields.json
/run/secrets/rhsm/syspurpose/syspurpose.json
/run/secrets/rhsm/rhsm.conf.kat-backup
/run/secrets/rhsm/rhsm.conf
/run/secrets/rhsm/logging.conf
/run/secrets/rhsm/facts
/run/secrets/rhsm/facts/uuid.facts
/run/secrets/rhsm/facts/katello.facts
/run/secrets/rhsm/ca
/run/secrets/rhsm/ca/redhat-uep.pem
/run/secrets/rhsm/ca/redhat-entitlement-authority.pem
/run/secrets/rhsm/ca/katello-server-ca.pem
/run/secrets/rhsm/ca/katello-default-ca.pem
/run/secrets/redhat.repo
/run/secrets/etc-pki-entitlement
/run/secrets/etc-pki-entitlement/6470214438861842971.pem
/run/secrets/etc-pki-entitlement/6470214438861842971-key.pem

Comment 7 Adam Kaplan 2021-05-13 17:13:10 UTC
@Scott I believe the node team is checking which changes to RHCOS or cri-o added the extra content to /run.

Comment 8 Peter Hunt 2021-05-13 17:30:06 UTC
I asked Scott to check because I can't think of any CRI-O changes that would cause this. but I've just added another person who's confused about this :D

Comment 9 Daniel Walsh 2021-05-13 18:09:39 UTC
This looks like a bad image, could someone give the exact image this came from.

Comment 11 Adam Kaplan 2021-05-14 15:26:50 UTC
@Dan so it looks like in the test we run a Dockerfile build with the origin-tools image [1]. 

It appears this image was updated to include the `stress-ng` and `fio` packages - when I pull that image from quay, the console, systemd, and other directories are present. We can safely rule out node/cri-o as the root cause of this issue.

Why those added the extra bits to /run, and why it took over two weeks for these changes to show up in CI, is a mystery to me [2].
I don't see these extra bits in the `oc` image, which leads me to believe that these bits are specific to the tools image.

Moving this back to the Build team so we make our test more resilient.

[1] https://quay.io/repository/openshift/origin-tools?tag=latest&tab=tags
[2] https://github.com/openshift/oc/pull/771

Comment 12 Adam Kaplan 2021-05-17 14:24:22 UTC
Note that the cli image doesn't have this extra content in /run, so we can use this image in our tests instead:

```
$ podman run --rm -i -t quay.io/openshift/origin-cli:latest /bin/bash
[root@1fdfbfe41dcb /]# ls /run
lock  rhsm  secrets
[root@1fdfbfe41dcb /]# 
```

Comment 14 XiuJuan Wang 2021-05-20 02:35:33 UTC
Has verified on pr, manually change the bug status.

Comment 15 Scott Dodson 2021-05-25 14:16:53 UTC
Additional changes coming for this, back to post

Comment 18 errata-xmlrpc 2021-07-27 23:08:01 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.