Bug 1955621 - Install of OCP 4 with FIPS enabled on IBM z/Architecture fails
Summary: Install of OCP 4 with FIPS enabled on IBM z/Architecture fails
Keywords:
Status: ASSIGNED
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: RHCOS
Version: 4.7
Hardware: s390x
OS: Unspecified
high
high
Target Milestone: ---
: 4.8.0
Assignee: Nikita Dubrovskii (IBM)
QA Contact: Michael Nguyen
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-30 14:41 UTC by Holger Wolf
Modified: 2021-05-06 12:54 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)
console-log (224.16 KB, text/plain)
2021-04-30 14:41 UTC, Holger Wolf
no flags Details
Console Log (145.37 KB, text/plain)
2021-04-30 15:05 UTC, Paul Novak
no flags Details
Installation log with patched rhcos image (117.66 KB, text/plain)
2021-05-06 12:54 UTC, Nikita Dubrovskii (IBM)
no flags Details

Description Holger Wolf 2021-04-30 14:41:40 UTC
Created attachment 1777825 [details]
console-log

Thanks for reporting your issue!

In order for the CoreOS team to be able to quickly and successfully triage your issue, please fill out the following template as completely as possible.

Be ready for follow-up questions and please respond in a timely manner.

If we can't reproduce a bug, we might close your issue.

---

OCP Version at Install Time: 4.7 
RHCOS Version at Install Time:
OCP Version after Upgrade (if applicable):
RHCOS Version after Upgrade (if applicable):
Platform: Z
Architecture: s390x


What are you trying to do? What is your use case?
Installing a worker in fips mode

What happened? What went wrong or what did you expect?
Displaying logs from failed units: rhcos-fips.service
-- Logs begin at Thu 2021-04-29 20:48:46 UTC, end at Thu 2021-04-29 20:48:51 UTC. --
Apr 29 20:48:49 systemd[1]: Starting Check for FIPS mode...
Apr 29 20:48:50 rhcos-fips[763]: Found /etc/ignition-machine-config-encapsulated.json in Ignition config
Apr 29 20:48:50 rhcos-fips[763]: FIPS mode required; updating BLS entries
Apr 29 20:48:50 rhcos-fips[763]: Appending 'fips=1 boot=LABEL=boot' to /run/rhcos-fips/sysroot/boot/loader/entries/ostree-1-rhcos.conf
Apr 29 20:48:50 rhcos-fips[763]: /usr/sbin/rhcos-fips: line 78: zipl: command not found
Apr 29 20:48:50 systemd[1]: [0;1;39m[0;1;31m[0;1;39mrhcos-fips.service: Main process exited, code=exited, status=127/n/a[0m
Apr 29 20:48:50 systemd[1]: [0;1;39m[0;1;31m[0;1;39mrhcos-fips.service: Failed with result 'exit-code'.[0m
Apr 29 20:48:50 systemd[1]: [0;1;31m[0;1;39m[0;1;31mFailed to start Check for FIPS mode.[0m
Apr 29 20:48:50 systemd[1]: rhcos-fips.service: Triggering OnFailure= dependencies.
Press Enter for emergency shell or wait 5 minutes for reboot.                
Press Enter for emergency shell or wait 4 minutes 45 seconds for reboot.      
Press Enter for emergency shell or wait 4 minutes 30 seconds for reboot.      
Press Enter for emergency shell or wait 4 minutes 15 seconds for reboot.
Press Enter for emergency shell or wait 3 minutes 30 seconds for reboot.      
Press Enter for emergency shell or wait 3 minutes 15 seconds for reboot.  

What are the steps to reproduce your issue? Please try to reduce these steps to something that can be reproduced with a single RHCOS node.


If you're having problems booting/installing RHCOS, please provide:
- the full contents of the serial console showing disk initialization, network configuration, and Ignition stage (see https://access.redhat.com/articles/7212 for information about configuring your serial console)
- Ignition JSON
- output of `journalctl -b`
console log attached.

If you're having problems post-upgrade, please provide:
- A complete must-gather (`oc adm must-gather`)


If you're having SELinux related issues, please provide:
- The full `/var/log/audit/audit.log` file
- Were any SELinux modules or booleans changed from the default configuration?
- The output of `ostree admin config-diff | grep selinux/targeted` on impacted nodes


Please add anything else that might be useful, for example:
- kernel command line (`cat /proc/cmdline`)
- contents of `/etc/NetworkManager/system-connections/`
- contents of `/etc/sysconfig/network-scripts/`

Comment 1 Holger Wolf 2021-04-30 14:43:51 UTC
FYI ndubrovs@redhat.com is looking into from a z perspective and will have a patch.

Comment 2 Micah Abbott 2021-04-30 14:50:52 UTC
Per comment #1, I'm assigning this to Nikita

Comment 3 Paul Novak 2021-04-30 15:05:55 UTC
Created attachment 1777833 [details]
Console Log

Comment 4 Paul Novak 2021-04-30 15:25:23 UTC
OCP Version at Install Time: 4.7.9
RHCOS Version at Install Time: 4.7.7
OCP Version after Upgrade (if applicable):
RHCOS Version after Upgrade (if applicable):
Platform: Z
Architecture: s390x

query cplevel
z/VM Version 7 Release 1.0, service level 2001 (64-bit)
Generated at 08/28/20 14:35:31 EDT
IPL at 12/15/20 01:09:52 EDT
Ready; T=0.01/0.01 11:24:37

Comment 5 Nikita Dubrovskii (IBM) 2021-05-03 06:43:10 UTC
Hi all. I've made a fix, now i'm testing it

Comment 6 Nikita Dubrovskii (IBM) 2021-05-03 17:35:02 UTC
PR: https://github.com/openshift/os/pull/546
 
Because of broken local rpm mirror wasn't able to build rhcos-47 and test it. So work is still in progress

Comment 8 Nikita Dubrovskii (IBM) 2021-05-06 12:54:11 UTC
Created attachment 1780295 [details]
Installation log with patched rhcos image

Log was captured on s390x zVM machine during installation of rhcos-47.83 image built with:
- https://github.com/openshift/os/pull/546
- https://github.com/nikita-dubrovskii/s390-tools/commit/70dde0e7d64f39a6f4747306c15ebc053ba01b19


Note You need to log in before you can comment on or make changes to this bug.