Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2002215

Summary: Multipath day1 not working on s390x
Product: OpenShift Container Platform Reporter: jschinta
Component: RHCOSAssignee: jschinta
Status: CLOSED ERRATA QA Contact: HuijingHei <hhei>
Severity: medium Docs Contact:
Priority: high    
Version: 4.9CC: aos-bugs, bgilbert, danili, dornelas, hhei, jligon, jschinta, miabbott, mrussell, nstielau, psundara
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: s390x   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 2009652 (view as bug list) Environment:
Last Closed: 2022-03-10 16:08:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2004596    
Bug Blocks: 1980363, 2009652    

Description jschinta 2021-09-08 09:39:11 UTC
Description of problem:
When enabling multipath during firstboot with ignition-kargs, it fails and boots into the emergency shell instead.

Version-Release number of selected component (if applicable):
4.9

How reproducible:
Append multipath kargs through ignition during firstboot

Steps to Reproduce:
1. Create ignition with multipath kargs
2. Install rhcos with ignition-file
3.

Actual results:
Boot into emergency shell

Expected results:
Boot rhcos with multipath from day1

Additional info:
The Problem is a missing zipl helper script in the initramfs, preventing zipl to recognize the device-mapper and running correctly. The fix is already upstream.

Comment 1 jschinta 2021-09-09 13:19:16 UTC
When the fix lands in rhcos 4.9 we also need to enable the test again in kola https://github.com/openshift/os/pull/617

Comment 2 Prashanth Sundararaman 2021-09-13 16:23:54 UTC
the fix to coreos-installer has landed: https://github.com/coreos/coreos-installer/pull/613. there needs to be a new tag created in the coreos-installer and we need rpms built off of that in rhcos/fcos for this test to work.

Comment 3 Prashanth Sundararaman 2021-09-13 16:26:06 UTC
Jan,

Could you work with the coreos team to maybe bump coreos-installer by creating a new tag or by backporting the patch in the current release? 

Thanks
Prashanth

Comment 4 Dan Li 2021-09-13 16:34:49 UTC
Re-assigning to Jan per Comment 3, let me know if the assignment is incorrect.

Comment 5 Dan Li 2021-09-20 18:13:07 UTC
Hi @jschintag would it be okay to assign this bug under your name as the assignee per work stated in Comment 3? If so, do you think you will continue to work on this bug in the next sprint (starting September 27th)? If so, we might want to add "reviewed-in-sprint". Thanks!

Comment 6 jschinta 2021-09-21 08:49:38 UTC
Hi Dan, 
Sure, i was planning on doing the backports in the next Sprint.

Comment 7 jschinta 2021-09-27 08:47:28 UTC
I created the backports of the fix for coreos-installer to rhcos 4.10 and 4.9

4.10: https://src.osci.redhat.com/rpms/coreos-installer/pull-request/30
4.9: https://src.osci.redhat.com/rpms/coreos-installer/pull-request/31

Comment 8 jschinta 2021-09-30 12:48:15 UTC
The RPM is build and included in nightlys. Tested with rhcos-49.84.202109300002-0-qemu.s390x.qcow2
PRs to re-enable the test in rhcos:
https://github.com/openshift/os/pull/639
https://github.com/openshift/os/pull/640

Comment 10 Micah Abbott 2021-09-30 18:08:57 UTC
In order to follow OCP Backport policy, we must target this fix for 4.10 first, then create the necessary backport BZs for older versions.

RHCOS 410.84.202109292202-0 on s390x includes coreos-installer-0.10.0-2.rhaos4.10.el8 with the fix, so it should be simple enough to move this along.

Comment 11 RHCOS Bug Bot 2021-10-01 07:30:24 UTC
This bug has been reported fixed in a new RHCOS build.  Do not move this bug to MODIFIED until the fix has landed in a new bootimage.

Comment 12 Michael Nguyen 2021-10-12 18:17:12 UTC
Ran multipath.day1 test and it passed. Test details are here: https://github.com/coreos/coreos-assembler/blob/main/mantle/kola/tests/misc/multipath.go

[coreos-assembler]$ kola run --qemu-image rhcos-410.84.202110080602-0-qemu.s390x.qcow2 multipath.day1
⚠️  Skipping kola test pattern "fips.enable*":
  👉 https://bugzilla.redhat.com/show_bug.cgi?id=1782026
⚠️  Skipping kola test pattern "ext.config.var-mount":
  👉 https://github.com/ibm-s390-tools/s390-tools/pull/82
⚠️  Skipping kola test pattern "coreos.ignition.journald-log":
  👉 https://github.com/coreos/coreos-assembler/issues/1173
=== RUN   multipath.day1
--- PASS: multipath.day1 (330.88s)
PASS, output in tmp/kola/qemu-unpriv-2021-10-12-1804-37

Comment 13 RHCOS Bug Bot 2021-10-20 17:53:45 UTC
The fix for this bug has landed in a bootimage bump, as tracked in bug 2004596 (now in status MODIFIED).  Moving this bug to MODIFIED.

Comment 15 HuijingHei 2021-10-26 14:01:01 UTC
Verify passed with RHCOS 410.84.202110141003-0

$ sudo kola run --qemu-image rhcos-410.84.202110141003-0-qemu.s390x.qcow2 multipath.day1
=== RUN   multipath.day1
--- PASS: multipath.day1 (420.41s)
PASS, output in _kola_temp/qemu-unpriv-2021-10-26-1347-32

Comment 17 jschinta 2022-02-07 07:56:46 UTC
I can't set it myself, but Doc type for this Bug should be "No Doc Update"

Comment 18 HuijingHei 2022-02-07 08:08:04 UTC
(In reply to jschinta from comment #17)
> I can't set it myself, but Doc type for this Bug should be "No Doc Update"

Set Doc type to "No Doc Update"

Comment 21 errata-xmlrpc 2022-03-10 16:08:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056