Bug 1955621

Summary: Install of OCP 4 with FIPS enabled on IBM z/Architecture fails
Product: OpenShift Container Platform Reporter: Holger Wolf <Holger.Wolf>
Component: RHCOSAssignee: Nikita Dubrovskii (IBM) <ndubrovs>
Status: CLOSED DEFERRED QA Contact: Michael Nguyen <mnguyen>
Severity: high Docs Contact:
Priority: low    
Version: 4.7CC: amulmule, dgilmore, dornelas, erich, fnovak, jligon, jschinta, jwiedman, matt.mondics, miabbott, mrussell, ndubrovs, nstielau, pwnovak, wolfgang.voesch
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: s390x   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-05-18 19:53:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
console-log
none
Console Log
none
Installation log with patched rhcos image none

Description Holger Wolf 2021-04-30 14:41:40 UTC
Created attachment 1777825 [details]
console-log

Thanks for reporting your issue!

In order for the CoreOS team to be able to quickly and successfully triage your issue, please fill out the following template as completely as possible.

Be ready for follow-up questions and please respond in a timely manner.

If we can't reproduce a bug, we might close your issue.

---

OCP Version at Install Time: 4.7 
RHCOS Version at Install Time:
OCP Version after Upgrade (if applicable):
RHCOS Version after Upgrade (if applicable):
Platform: Z
Architecture: s390x


What are you trying to do? What is your use case?
Installing a worker in fips mode

What happened? What went wrong or what did you expect?
Displaying logs from failed units: rhcos-fips.service
-- Logs begin at Thu 2021-04-29 20:48:46 UTC, end at Thu 2021-04-29 20:48:51 UTC. --
Apr 29 20:48:49 systemd[1]: Starting Check for FIPS mode...
Apr 29 20:48:50 rhcos-fips[763]: Found /etc/ignition-machine-config-encapsulated.json in Ignition config
Apr 29 20:48:50 rhcos-fips[763]: FIPS mode required; updating BLS entries
Apr 29 20:48:50 rhcos-fips[763]: Appending 'fips=1 boot=LABEL=boot' to /run/rhcos-fips/sysroot/boot/loader/entries/ostree-1-rhcos.conf
Apr 29 20:48:50 rhcos-fips[763]: /usr/sbin/rhcos-fips: line 78: zipl: command not found
Apr 29 20:48:50 systemd[1]: [0;1;39m[0;1;31m[0;1;39mrhcos-fips.service: Main process exited, code=exited, status=127/n/a[0m
Apr 29 20:48:50 systemd[1]: [0;1;39m[0;1;31m[0;1;39mrhcos-fips.service: Failed with result 'exit-code'.[0m
Apr 29 20:48:50 systemd[1]: [0;1;31m[0;1;39m[0;1;31mFailed to start Check for FIPS mode.[0m
Apr 29 20:48:50 systemd[1]: rhcos-fips.service: Triggering OnFailure= dependencies.
Press Enter for emergency shell or wait 5 minutes for reboot.                
Press Enter for emergency shell or wait 4 minutes 45 seconds for reboot.      
Press Enter for emergency shell or wait 4 minutes 30 seconds for reboot.      
Press Enter for emergency shell or wait 4 minutes 15 seconds for reboot.
Press Enter for emergency shell or wait 3 minutes 30 seconds for reboot.      
Press Enter for emergency shell or wait 3 minutes 15 seconds for reboot.  

What are the steps to reproduce your issue? Please try to reduce these steps to something that can be reproduced with a single RHCOS node.


If you're having problems booting/installing RHCOS, please provide:
- the full contents of the serial console showing disk initialization, network configuration, and Ignition stage (see https://access.redhat.com/articles/7212 for information about configuring your serial console)
- Ignition JSON
- output of `journalctl -b`
console log attached.

If you're having problems post-upgrade, please provide:
- A complete must-gather (`oc adm must-gather`)


If you're having SELinux related issues, please provide:
- The full `/var/log/audit/audit.log` file
- Were any SELinux modules or booleans changed from the default configuration?
- The output of `ostree admin config-diff | grep selinux/targeted` on impacted nodes


Please add anything else that might be useful, for example:
- kernel command line (`cat /proc/cmdline`)
- contents of `/etc/NetworkManager/system-connections/`
- contents of `/etc/sysconfig/network-scripts/`

Comment 1 Holger Wolf 2021-04-30 14:43:51 UTC
FYI ndubrovs is looking into from a z perspective and will have a patch.

Comment 2 Micah Abbott 2021-04-30 14:50:52 UTC
Per comment #1, I'm assigning this to Nikita

Comment 3 Paul Novak 2021-04-30 15:05:55 UTC
Created attachment 1777833 [details]
Console Log

Comment 4 Paul Novak 2021-04-30 15:25:23 UTC
OCP Version at Install Time: 4.7.9
RHCOS Version at Install Time: 4.7.7
OCP Version after Upgrade (if applicable):
RHCOS Version after Upgrade (if applicable):
Platform: Z
Architecture: s390x

query cplevel
z/VM Version 7 Release 1.0, service level 2001 (64-bit)
Generated at 08/28/20 14:35:31 EDT
IPL at 12/15/20 01:09:52 EDT
Ready; T=0.01/0.01 11:24:37

Comment 5 Nikita Dubrovskii (IBM) 2021-05-03 06:43:10 UTC
Hi all. I've made a fix, now i'm testing it

Comment 6 Nikita Dubrovskii (IBM) 2021-05-03 17:35:02 UTC
PR: https://github.com/openshift/os/pull/546
 
Because of broken local rpm mirror wasn't able to build rhcos-47 and test it. So work is still in progress

Comment 8 Nikita Dubrovskii (IBM) 2021-05-06 12:54:11 UTC
Created attachment 1780295 [details]
Installation log with patched rhcos image

Log was captured on s390x zVM machine during installation of rhcos-47.83 image built with:
- https://github.com/openshift/os/pull/546
- https://github.com/nikita-dubrovskii/s390-tools/commit/70dde0e7d64f39a6f4747306c15ebc053ba01b19

Comment 9 Dennis Gilmore 2021-05-18 19:30:07 UTC
Setting the priority to low as we will be evaluating FIPS support in OCP for Z in an upcoming quarter

Comment 10 Micah Abbott 2021-05-18 19:53:10 UTC
After discussing this with the OCP Multi Arch team, it appears there has been a misunderstanding.  FIPS compliance on OCP running on s390x has never been supported.

There is an open RFE for this work being tracked here - https://issues.redhat.com/browse/MULTIARCH-690

Since this would be a feature level change and not a bug fix, I am going to close this BZ as DEFERRED.

Please follow the RFE for progress on supporting FIPS + OCP s390x.