Bug 2031637

Looks like lszdev is not in s390utils-core. According to this comment a while ago: https://github.com/coreos/fedora-coreos-config/pull/756#issuecomment-754699901 Dan Horak did move fdasd and lszdev to s390utils-core, but looks like it is only present in version 2.16 which is part of RHEL 8.5

Dan,

Can this change be backported to 8.4 s390utils-core to fix this issue?

Thanks
Prashanth

rhcos uses the 8.4 EUS rpms to build: http://rhsm-pulp.corp.redhat.com/content/eus/rhel8/8.4/s390x/baseos/os/Packages/s/ and they have an older version of s390utils-core which does not contain fdasd and lszdev

Setting a needinfo for Dan per Comment 1

Folks,

FYI and for OCP on Z support documentation purposes: OCP 4.10 on Z RHCOS build 410.84.202112132002-0 fails with the same issue.

Thank you,
Kyle

(In reply to Prashanth Sundararaman from comment #1)
> Looks like lszdev is not in s390utils-core. According to this comment a
> while ago:
> https://github.com/coreos/fedora-coreos-config/pull/756#issuecomment-
> 754699901 Dan Horak did move fdasd and lszdev to s390utils-core, but looks
> like it is only present in version 2.16 which is part of RHEL 8.5
> 
> Dan,
> 
> Can this change be backported to 8.4 s390utils-core to fix this issue?

yes, that shouldn't be a big issue. Could you somehow clone this bug into a RHEL bug for s390utils? I will then clone it further for the 8.4.0.z zstream.

Thanks Dan. opened https://bugzilla.redhat.com/show_bug.cgi?id=2032486 against RHEL.

vmcp command is now also missing in RHCOS which was available in older RHCOS release. Could you also add that?

yes, we can do that, vmcp is moving to core in 8.6.0 via bug #2021071

i'm wondering  now whether this is a blocker. we can also use an older bootimage to get the install to work.

Kyle,

Could you use an older 4.10 rhcos image for the installation or even a 4.9 rhcos image to install the 4.10 payload ? if that works, we can remove the blocker flag.

Thanks
Prashanth

Prashanth,

Thank you for the assistance.  

1. To work around this issue for zVM environments, we have been using the last/latest OCP 4.10 on Z RHCOS build that does boot properly for the bootstrap, master/control, and worker/compute nodes, as it contains the required s390x tools.

2. This latest OCP 4.10 on Z RHCOS build that does boot properly is the OCP 4.10 on Z RHCOS 410.84.202112062233-0 build.

3. As you know, although using the RHCOS 410.84.202112062233-0 build to boot the bootstrap, master/control, and worker/compute nodes is a workaround for the issue, in doing so we do not exercise any changes to the OCP 4.10 on Z RHCOS builds released after this RHCOS 410.84.202112062233-0 build for the boot process.  

4. Any boot related changes introduced into these post OCP 4.10 on Z RHCOS 410.84.202112062233-0 builds will not be tested until a subsequent OCP 4.10 on Z RHCOS build is made available with the required s390x tools to properly boot the bootstrap, master/control, and worker/compute nodes.

Thank you,
Kyle

Hi Kyle,

Thanks for testing it and getting back so quickly. I certainly understand your concern about missing boot related changes, but those are far and few. Also after install, the os will be pivoted to the one present in the OCP payload as part of the machine-os-content so all your libraries will be up to date after install.

I am removing the blocker flag for this as there is a workaround.

Thanks
Prashanth

Kyle,

We have latest 4.10 rhcos images with the version of s390utils-core containing the missing binaries: https://releases-rhcos-art.cloud.privileged.psi.redhat.com/?stream=releases/rhcos-4.10-s390x&release=410.84.202112212202-0#410.84.202112212202-0 . If you could test this out that would be great.

Thanks
Prashanth

Prashanth,

Thanks for the update and OCP 4.10 RHCOS build.  I'll test today with this OCP 4.10 RHCOS build and provide an update.

Thank you,
Kyle

Prashanth,

Using the OCP 4.10 on Z RHCOS 410.84.202112212202-0 build to boot the bootstrap, master/control, and worker/compute nodes, we have successfully installed the 5 following OCP 4.10 on Z builds in a zVM environment.  These 5 OCP 4.10 on Z builds subsequently installed their required OCP 4.10 on Z RHCOS builds as part of the normal installation process.

1. OCP 4.10.0-0.nightly-s390x-2021-12-16-185334
2. OCP 4.10.0-0.nightly-s390x-2021-12-18-034912
3. OCP 4.10.0-0.nightly-s390x-2021-12-20-215258
4. OCP 4.10.0-0.nightly-s390x-2021-12-21-231942
5. OCP 4.10.0-0.nightly-s390x-2021-12-22-053640


Thank you,
Kyle

thanks Kyle. I'll keep this open till https://bugzilla.redhat.com/show_bug.cgi?id=2032507 makes it but i will lower the severity.

Prashanth,


Thank you.  Sounds good.

Thank you,
Kyle

Hi Dan, do you know if https://bugzilla.redhat.com/show_bug.cgi?id=2032507 will move to ON_QA before the end of the current OpenShift sprint (January 8th)? If not, I'd like to add "reviewed-in-sprint" flag to this bug

I think it's unlikely to meet the January 8th date, I expect progress during the next week.

Thank you Dan. Setting the flag.

Hi Dan, do you know if the related bug BZ 2032507 would move to ON_QA before the end of the current OpenShift sprint (January 29th)? If not, I'd like to add "reviewed-in-sprint" flag to indicate that the team has reviewed the progress of this bug during this sprint.

(In reply to Dan Li from comment #20)
> Hi Dan, do you know if the related bug BZ 2032507 would move to ON_QA before
> the end of the current OpenShift sprint (January 29th)? If not, I'd like to
> add "reviewed-in-sprint" flag to indicate that the team has reviewed the
> progress of this bug during this sprint.

I don't think so, it's waiting on our QA to do first level of verification.

Thanks Dan. Adding reviewed-in-sprint.

Folks,

In testing the latest OCP 4.10 FC build, 4.10.0-fc.4, with the corresponding RHCOS 410.84.202201280202-0 build, the vmcp command and s390utils-base rpm are not contained within this build.

1. The s390utils-base fdasd and lszdev commands are contained within the RHCOS 410.84.202201280202-0 build, in the s390utils-core-2.16.0-2.el8.s390x rpm.

2. The s390utils-base vmcp command is not contained within the RHCOS 410.84.202201280202-0 build, as the s390utils-base rpm is not contained within this RHCOS build, and the vmcp command has not been copied to the s390utils-core-2.16.0-2.el8.s390x rpm.

   [root@master-0 bin]# which vmcp
   /usr/bin/which: no vmcp in (/root/.local/bin:/root/bin:/sbin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin)
   [root@master-0 bin]# vmcp
   -bash: vmcp: command not found
   [root@master-0 bin]#


   [root@master-0 bin]# rpm -qa | grep s390utils
   s390utils-core-2.16.0-2.el8.s390x
   [root@master-0 bin]#


3. Given the s390utils-base rpm is not contained within the RHCOS 410.84.202201280202-0 build, other important s390utils commands are missing.  For example, but not limited to:
   (1) /usr/sbin/chccwdev
   (2) /usr/sbin/chchp
   (3) /usr/sbin/chcpumf
   (4) /usr/sbin/chshut
   (5) /usr/sbin/chzcrypt
   (6) /usr/sbin/dasdstat
   (7) /usr/sbin/dasdview
   (8) /usr/sbin/dbginfo.sh
   (9) /usr/sbin/lschp
  (10) /usr/sbin/lscpumf
  (11) /usr/sbin/lscss
  (12) /usr/sbin/lsdasd
  (13) /usr/sbin/lsluns
  (14) /usr/sbin/lsqeth
  (15) /usr/sbin/lstape
  (16) /usr/sbin/lszcrypt
  (17) /usr/sbin/lszfcp


4. For Client debug and support purposes, and general test purposes, it would be very helpful if the full list of s390utils-base commands could be included in the OCP 4.10 on Z RHCOS builds, as with the OCP 4.9 on Z and previous OCP 4.x on Z releases' RHCOS builds.


5. The current workaround for not having the full list of s390utils-base commands within the OCP 4.10 on Z RHCOS builds is to copy them (scp) over from the bastion and/or other RHEL 8.4/8.5 server with the s390utils-base-2.16.0-2.el8.s390x rpm or other appropriate s390utils-base rpm level.  This is problematic, though, as the s390utils-base rpm level will potentially not always be compatible with the   


6. For comparison purposes, the latest OCP 4.9 build, 4.9.19, with its corresponding RHCOS 49.84.202201262102-0 build contains both the s390utils-core and s390utils-base rpms, including the fdasd, lszdev, and vmcp commands in the s390utils-base-2.15.1-5.el8.s390x rpm.

   [root@master-0 ~]# rpm -qa | grep s390utils
   s390utils-core-2.15.1-5.el8.s390x
   s390utils-base-2.15.1-5.el8.s390x
   [root@master-0 ~]#

   [root@master-0 ~]# rpm -qf `which lsdasd `
   s390utils-base-2.15.1-5.el8.s390x
   [root@master-0 ~]# rpm -qf `which fdasd `
   s390utils-base-2.15.1-5.el8.s390x
   [root@master-0 ~]# rpm -qf `which lszdev `
   s390utils-base-2.15.1-5.el8.s390x
   [root@master-0 ~]# rpm -qf `which vmcp `
   s390utils-base-2.15.1-5.el8.s390x
   [root@master-0 ~]#



   [root@master-0 ~]# which vmcp
   /sbin/vmcp
   [root@master-0 ~]#

   [root@master-0 ~]# vmcp
   Usage: vmcp [OPTIONS] CP-command

   z/VM CP command interface

   OPTIONS
    -k, --keepcase     Do not convert CP-command string to uppercase
    -b, --buffer SIZE  Specify buffer size in bytes, kilobytes (k) or megabytes
                       (M). SIZE range from 4096 to 1048576 bytes
    -h, --help         Print this help, then exit
    -v, --version      Print version information, then exit
   [root@master-0 ~]#



Thank you,
Kyle

Prashanth,

Just a folllow-up to our discussion earlier today to document the clarification that
1. Although the vmcp command is missing from the latest OCP 4.10 on Z RHCOS builds, the actual OCP 4.10 on Z installs continue to succeed since the December 21, 2021 RHCOS update, as documented above in comments 12 through 16, as they contain the lszdev command.

2. I will provide a list here of the requested/highly recommended s390utils-base commands to be included in the s390utils-core rpm for RHCOS 4.10 and beyond, as the s390utils-base rpm will no longer be included in the RHCOS 4.10 and beyond builds.


Thank you,
Kyle

Thanks Kyle - could you actually create a new BZ and assign it to the RHEL component with the commands we would need ?

Prashanth,

Thanks.  I've opened Red Hat RHEL 8.4 bugzilla https://bugzilla.redhat.com/show_bug.cgi?id=2052687 for this issue.

Thank you,
Kyle

Hi Dan, as BZ 2032507 has moved to ON_QA, do you know if this bug would progress past ON_QA before the end of this current OpenShift sprint (Feb. 19th)? If not, I'd like to add "reviewed-in-sprint" flag to indicate that the team has reviewed the progress of this bug during this sprint.

Setting reviewed-in-sprint, as dependent bug has just been verified but it may take it longer for this bug to reach ON_QA

Hi Dan, as BZ 2032507 has moved to VERIFIED, should this bug be moved to a further state than ASSIGNED? If not, I'd like to add "reviewed-in-sprint" flag to indicate that the team has reviewed the progress of this bug during this sprint.

I think it depends how the OCP team needs to track the progress. The RHEL update made a progress, so we know it will be released as part the 8.4.0 batch update 8 (on 2022-04-19). AFAIK to be consumed until then it still needs some manual action by the OCP/CoreOS team.

I had a chat with the team and we think that since the RHEL bug (BZ 2032507) has been verified, we can close this bug for now.

Note for future viewers of this bug, that the fix for this bug will ship with the 8.6 s390utils-core.