Bug 1866445

Summary: kola --basic-qemu-scenarios scenario fail on ppc64le & s390x
Product: OpenShift Container Platform Reporter: Andy McCrae <amccrae>
Component: RHCOSAssignee: Andy McCrae <amccrae>
Status: CLOSED ERRATA QA Contact: Michael Nguyen <mnguyen>
Severity: low Docs Contact:
Priority: high    
Version: 4.5CC: bbreard, imcleod, jligon, miabbott, nstielau, rravanel, smilner, travier, walters
Target Milestone: ---   
Target Release: 4.7.0   
Hardware: s390x   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Cause: There is no support in ppc64le and s390x for NVME emulation in KVM in RHEL 8.2 Consequence: The kola --basic-qemu-scenarios utilising NVME emulation fail. Workaround (if any): Disabled the tests for NVME emulation on ppc64le and s390x Result: Tests succeed
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-02-24 15:15:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Andy McCrae 2020-08-05 15:08:30 UTC
Description of problem:
Since https://gitlab.cee.redhat.com/coreos/redhat-coreos/-/merge_requests/1073/ merged to change from basic --> basic-qemu-scenarios the RHCOS builds for 4.5 are failing for both ppc64le and s390x.


Version-Release number of selected component (if applicable):
4.5

How reproducible:
Consistently


Steps to Reproduce:
1. Run the build pipeline for ppc64le or s390x

Actual results:

Fails at the the cosa kola run --basic-qemu-scenarios step


Expected results:

Tests succeed and build completes.

Additional info:
For s390x we see the following failure:
qemu-system-s390x: -device nvme,drive=d1,serial=primary-disk,bootindex=1: 'nvme' is not a valid device model name

For ppc64le we see the same test timeout, but with TFTP errors in the console.txt for the basic test:
  Receiving data:  0 KBytes
  TFTP error: TFTP access violation

E3407: Load failed

Comment 1 Micah Abbott 2020-08-05 19:44:35 UTC
Andy, I'm setting you as the assignee, but lean on the CoreOS team for help.  If you don't want to be the assignee, let me know.

Comment 2 Renata Ravanelli 2020-08-06 15:51:33 UTC
Just updating here with some info:

RHEL does not support QEMU nvme emulation, while Fedora 31 does have support only for qemu-system-ppc64, but not for qemu-system-s390x. For s390x there is no workaround. 

In the power case it could be related to some bug, mainly because the QEMU team is not running testing against it due no support in RHEL for nvme.

I already spoke with the QEMU team, since they don't have many people it could take some time to check this bug.

We also need to keep in mind that there is not support for nvme in RHEL, for this reason they will check it as low priority.



How to reproduce it:

In a RHEL 7.8 run a container with Fedora 31 and try to run qemu-system-ppc64 with nvme.

Result:


Console output:
  Welcome to Open Firmware
  Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
  This program and the accompanying materials are made available
  under the terms of the BSD License available at
  http://www.opensource.org/licenses/bsd-license.php
Trying to load:  from: /pci@800000020000000/pci8086,5845@4 ...
E3405: No such device
Trying to load:  from: disk ...
E3405: No such device
Trying to load:  from: /vdevice/v-scsi@71000002/disk@8200000000000000 ... No medium !
E3405: No such device
Trying to load:  from: /pci@800000020000000/ethernet@3 ...
 Initializing NIC
  Reading MAC address from device: 52:54:00:12:34:56
  Requesting information via DHCP: done
  Using IPv4 address: 10.0.2.15
  Requesting file "" via TFTP from 10.0.2.2
Trying pxelinux.cfg files...
  TFTP error: TFTP access violation
  TFTP error: TFTP access violation
  TFTP error: TFTP access violation
  TFTP error: TFTP access violation
  TFTP error: TFTP access violation
  TFTP error: TFTP access violation
  TFTP error: TFTP access violation
  TFTP error: TFTP access violation
  TFTP error: TFTP access violation
  TFTP error: TFTP access violation
  TFTP error: TFTP access violation

Comment 3 Timothée Ravier 2020-08-11 14:13:28 UTC
The change from https://gitlab.cee.redhat.com/coreos/redhat-coreos/-/merge_requests/1073/ enables more tests beyond the basic ones. If those tests are not compatible/supported on ppc64le and s390x maybe we should consider disabling them for those platforms? I can create a PR for that.

Comment 4 Timothée Ravier 2020-08-12 15:10:26 UTC
Discard my previous comment as this has already been done in https://gitlab.cee.redhat.com/coreos/redhat-coreos/-/merge_requests/1086.

Comment 5 Micah Abbott 2020-09-14 19:52:07 UTC
It's unlikely this will be resolved in time for 4.6 GA; moving to 4.7.0

Comment 9 errata-xmlrpc 2021-02-24 15:15:21 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633