Bug 1906511 - Root reprovisioning tests flaking often in CI
Summary: Root reprovisioning tests flaking often in CI
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: RHCOS
Version: 4.7
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.7.0
Assignee: Jonathan Lebon
QA Contact: Michael Nguyen
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-12-10 16:59 UTC by Jonathan Lebon
Modified: 2021-02-24 15:42 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-02-24 15:41:57 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:42:24 UTC

Description Jonathan Lebon 2020-12-10 16:59:48 UTC
We're seeing the Tang tests using the new LUKS path flaking often in CI (yet they're solid locally). Likely some I/O latency sensitive races in there.

Comment 1 Jonathan Lebon 2020-12-11 16:54:49 UTC
OK, this is actually pretty easy to reproduce in a pod on the PSI privileged cluster. It also affects more than just Tang tests.
If you just run a VM directly with e.g. a TPM2-pinning Ignition config:

```
$ kola qemuexec -i tpm2.ign --ignition-direct --memory 4096
```

You'll find that it always gets killed at the save stage:

```
[***   ] A start job is running for Ignition…: Save Partitions (17s / no limit)Error: signal: killed
2020-12-11T16:04:24Z cli: signal: killed
```

Debugging further, it looks like the qemu process is getting killed because it's too resource hungry. Bumping my cosa-shell to request 2 CPUs makes it solid.

Filed: https://gitlab.cee.redhat.com/coreos/rhcos-psi/-/merge_requests/135

Comment 2 Jonathan Lebon 2020-12-14 19:55:16 UTC
MRs merged. kola denylist revert in https://github.com/openshift/os/pull/468.

Comment 5 Michael Nguyen 2021-01-05 22:09:05 UTC
Test looks green for recent builds.  Closing as verified.

Comment 7 errata-xmlrpc 2021-02-24 15:41:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633


Note You need to log in before you can comment on or make changes to this bug.