Bug 1906511

Summary: Root reprovisioning tests flaking often in CI
Product: OpenShift Container Platform Reporter: Jonathan Lebon <jlebon>
Component: RHCOSAssignee: Jonathan Lebon <jlebon>
Status: CLOSED ERRATA QA Contact: Michael Nguyen <mnguyen>
Severity: high Docs Contact:
Priority: high    
Version: 4.7CC: bbreard, imcleod, jligon, miabbott, nstielau
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:41:57 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jonathan Lebon 2020-12-10 16:59:48 UTC
We're seeing the Tang tests using the new LUKS path flaking often in CI (yet they're solid locally). Likely some I/O latency sensitive races in there.

Comment 1 Jonathan Lebon 2020-12-11 16:54:49 UTC
OK, this is actually pretty easy to reproduce in a pod on the PSI privileged cluster. It also affects more than just Tang tests.
If you just run a VM directly with e.g. a TPM2-pinning Ignition config:

```
$ kola qemuexec -i tpm2.ign --ignition-direct --memory 4096
```

You'll find that it always gets killed at the save stage:

```
[***   ] A start job is running for Ignition…: Save Partitions (17s / no limit)Error: signal: killed
2020-12-11T16:04:24Z cli: signal: killed
```

Debugging further, it looks like the qemu process is getting killed because it's too resource hungry. Bumping my cosa-shell to request 2 CPUs makes it solid.

Filed: https://gitlab.cee.redhat.com/coreos/rhcos-psi/-/merge_requests/135

Comment 2 Jonathan Lebon 2020-12-14 19:55:16 UTC
MRs merged. kola denylist revert in https://github.com/openshift/os/pull/468.

Comment 5 Michael Nguyen 2021-01-05 22:09:05 UTC
Test looks green for recent builds.  Closing as verified.

Comment 7 errata-xmlrpc 2021-02-24 15:41:57 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633