Description of problem: The cluster does not come up installing it on zVM with RoCE cards. The bootstrap node comes up, and is detected by the ansible script. Ssh to the core OS works but shows: --- This is the bootstrap node; it will be destroyed when the master is fully up. The primary service is "bootkube.service". To watch its status, run e.g. journalctl -b -f -u bootkube.service [systemd] Failed Units: 2 rdma-load-modules rdma-load-modules However pinging the internet and the bastion is successful. those modules are only needed for infiniband and rdma which are not supported for now but we should still either not load them or load them without errors Version-Release number of selected component (if applicable): LGA 06.12.2019 Additional info: N/A
Can you get what failed to load and the error messages? Our support for s390 is weak at best and will likely need some fixes upstream to be solid.
this is not s390x specific iirc. Modules seem to be missing. [core@bootstrap-0 ~]$ journalctl -u rdma-load-modules -- Logs begin at Mon 2019-12-16 09:05:43 UTC, end at Wed 2019-12-18 11:26:22 UTC. -- Dec 16 09:05:54 localhost systemd[1]: Starting Load RDMA modules from /etc/rdma/modules/infiniband.conf... Dec 16 09:05:55 localhost systemd-modules-load[1250]: Failed to find module 'ib_ipoib' Dec 16 09:05:55 localhost systemd-modules-load[1250]: Failed to find module 'ib_umad' Dec 16 09:05:55 localhost systemd[1]: rdma-load-modules: Main process exited, code=exited, status=1/FAILURE Dec 16 09:05:55 localhost systemd[1]: rdma-load-modules: Failed with result 'exit-code'. Dec 16 09:05:55 localhost systemd[1]: Failed to start Load RDMA modules from /etc/rdma/modules/infiniband.conf. [core@bootstrap-0 ~]$ journalctl -u rdma-load-modules -- Logs begin at Mon 2019-12-16 09:05:43 UTC, end at Wed 2019-12-18 11:26:31 UTC. -- Dec 16 09:05:54 localhost systemd[1]: Starting Load RDMA modules from /etc/rdma/modules/rdma.conf... Dec 16 09:05:55 localhost systemd-modules-load[1257]: Failed to find module 'ib_iser' Dec 16 09:05:55 localhost systemd-modules-load[1257]: Inserted module 'rdma_ucm' Dec 16 09:05:55 localhost systemd-modules-load[1257]: Inserted module 'rpcrdma' Dec 16 09:05:55 localhost systemd[1]: rdma-load-modules: Main process exited, code=exited, status=1/FAILURE Dec 16 09:05:55 localhost systemd[1]: rdma-load-modules: Failed with result 'exit-code'. Dec 16 09:05:55 localhost systemd[1]: Failed to start Load RDMA modules from /etc/rdma/modules/rdma.conf.
> Dec 16 09:05:55 localhost systemd-modules-load[1257]: Failed to find module 'ib_iser' It looks like ib_iser isn't in the s390x package: $ rpm -qpl kernel-modules-4.18.0-147.4.1.el8_1.x86_64.rpm | grep ib_iser /lib/modules/4.18.0-147.4.1.el8_1.x86_64/kernel/drivers/infiniband/ulp/iser/ib_iser.ko.xz /lib/modules/4.18.0-147.4.1.el8_1.x86_64/kernel/drivers/infiniband/ulp/isert/ib_isert.ko.xz $ rpm -qpl kernel-modules-4.18.0-147.4.1.el8_1.s390x.rpm | grep ib_iser $
re: ddutile, is the ib_iser driver supported on s390x? I'd recommend asking SteveB if rdma on s390 is supported in general, and if so, which IB ulp's should be enabled/built. We have no s390-RDMA systems to test with, thus, the regression/test of RDMA on s390 would have to be done by IBM.
(In reply to Don Dutile (Red Hat) from comment #5) > re: ddutile, is the ib_iser driver supported on s390x? > > I'd recommend asking SteveB if rdma on s390 is supported in general, and if > so, which IB ulp's should be enabled/built. > We have no s390-RDMA systems to test with, thus, the regression/test of RDMA > on s390 would have to be done by IBM. sbest? Is RDMA supported on s390? P.
(In reply to Prarit Bhargava from comment #6) > (In reply to Don Dutile (Red Hat) from comment #5) > > re: ddutile, is the ib_iser driver supported on s390x? > > > > I'd recommend asking SteveB if rdma on s390 is supported in general, and if > > so, which IB ulp's should be enabled/built. > > We have no s390-RDMA systems to test with, thus, the regression/test of RDMA > > on s390 would have to be done by IBM. > > sbest? Is RDMA supported on s390? > > P. why does this need to be private IBM is on the ticket they should be able to answer it.
We see the same issue with the OCP version 4.3.0-nightly OCP: Server Version: 4.3.0-0.nightly-s390x-2020-03-09-183623 Kubernetes Version: v1.16.2 RHCOS: Kernel Version: 4.18.0-147.5.1.el8_1.s390x OS Image: Red Hat Enterprise Linux CoreOS 43.81.202003091812.0 (Ootpa) Container Runtime Version: cri-o://1.16.3-22.dev.rhaos4.3.git11c04e3.el8
(In reply to Prarit Bhargava from comment #6) > (In reply to Don Dutile (Red Hat) from comment #5) > > re: ddutile, is the ib_iser driver supported on s390x? > > > > I'd recommend asking SteveB if rdma on s390 is supported in general, and if > > so, which IB ulp's should be enabled/built. > > We have no s390-RDMA systems to test with, thus, the regression/test of RDMA > > on s390 would have to be done by IBM. > > sbest? Is RDMA supported on s390? > > P. Wolfgang, are you able to provide an answer for Prarit's question?
Shared Memory Communications over RDMA (SMC-R/SMC-D) is supported with RoCE on s390x. Currently it's not supported for ocp on s390x as it isn't fully transparent and requires the application to use a preloaded library
(In reply to Alexander Klein from comment #11) I cleared the need info request.
(In reply to Alexander Klein from comment #11) > Shared Memory Communications over RDMA (SMC-R/SMC-D) is supported with RoCE > on s390x. > Currently it's not supported for ocp on s390x as it isn't fully transparent > and requires the application to use a preloaded library Which preloaded libraries are required? Are we talking about only ib_iser or are there others needed also?
To use smc-d/r on ibm z you have to either: - use the AF_SMC address family in the system calls of your application or - use the preload library via environment variable export LD_PRELOAD=ld_pre_smc.so if the application cannot be changed.
Just few comments, the preload lib is libsmc-preload.so, since smc-tools 1.1.0. And to be clear, the preload lib is open-source, part of the smc-tools project (https://www.ibm.com/developerworks/linux/linux390/smc-tools.html).
Adding "UpcomingSprint" as this bug is unlikely to be fixed during this sprint.
Adding UpcomingSprint as it won't be resolved this sprint.
Adding "UpcomingSprint" label as the team is fixing other bugs and do not have the bandwidth to resolve this bug. Also changing target release to 4.7 as this bug will not make it into 4.6
Adding "UpcomingSprint" label as the team is fixing other bugs and do not have the bandwidth to resolve this bug.
Re-assigning to Muhammad to investigate further
Adding UpcomingSprint as team will not resolve this bug during this sprint
Hi Muhammad, do you think this bug should target 4.7.0 (January 28th) for its fix? If there is no set Target Release, I'd like to set the "Target Release" value to "---"
Hi Dan, yes please set it to ---
Adding UpcomingSprint as team will not resolve this bug during this sprint. Hardware access request has been made and we are awaiting for the allocation.
Adding "Reviewed-in-Sprint" flag as this bug will not be resolved during this sprint
Hi Muhammad, do you think this bug will be resolved before the end of this sprint (Feb 27th)? If not, can we set "Reviewed-in-Sprint" flag to "+"?
Setting the flag after chatting with Muhammad. This bug will not be fixed before end of this sprint.
An initial fix has been provided, but a more proper solution is needed which is to be done in April.
Add "Reviewed-in-Sprint" based on Comment 30
Hi Dan, Sorry for getting back so late on this. But one thing we need an answer here is: I see the IB ulp modules are disabled in the Kernel config for s390x: # grep -rn "CONFIG_INFINIBAND_ISER" /lib/modules/4.18.0-240.el8.s390x/ /lib/modules/4.18.0-240.el8.s390x/config:2359:# CONFIG_INFINIBAND_ISER is not set /lib/modules/4.18.0-240.el8.s390x/config:2360:# CONFIG_INFINIBAND_ISERT is not set Why rdma service is then trying to load the ulp modules? Please suggest, how to prevent the rdma service to not to load those modules on platforms where it is not supported? I try to edit the /etc/rdma/rdma.conf and change following: # Load IPoIB IPOIB_LOAD=no # Load SRP (SCSI Remote Protocol initiator support) module SRP_LOAD=no # Load SRPT (SCSI Remote Protocol target support) module SRPT_LOAD=no # Load iSER (iSCSI over RDMA initiator support) module ISER_LOAD=no However the service is still trying to load the modules. -Muhammad
Hi Muhammad, I don't have an answer for this technical question. @Carvel @Dan Horak do you guys have any insights? Alternatively you can ask your question on the multi-arch Slack channel.
Do we know where are the 2 units (rdma-load-modules and rdma-load-modules) coming from? Are they from RHEL or is it something CoreOS specific?
The services are coming from RHEL and I think it is only used in CoreOS because of the SR-IOV feature which is not available on s390x. Also whether OpenShift makes use of IB_ISERT,IB_IPoIB modules is not clear. Perhaps somewhere in RHEL but not necessarily with CoreOS.
Hi Munammad, I am not sure if Dennis is the right person to flag with any technical questions. In addition to that, do you think this bug will be resolved before the end of the current sprint? If not, can we set the "Reviewed-in-Sprint" flag?
Turning off needinfo from Dennis after chatting with Muhammad
Hi Muhammad, do you think this bug will be resolved before the end of the current sprint (May 1st)? If not, can we set the "Reviewed-in-Sprint" flag?
Hi Dan, yes please set the flag.
Changing the assignee to Prashanth and adding "reviewed-in-sprint" as this bug will be looked at but likely won't be resolved this sprint.
filed a PR for a fix in rdma-core: https://github.com/linux-rdma/rdma-core/pull/1015
so i closed the previous PR i opened because the packaging issue is not general. RedHat's s390x kernel excludes infiniband support. Also on further investigation, the libpcap library is pulling in the rdma-core package because libpcap now has support for sniffing rdma packets. I opened a BZ against libpcap to disable rdma sniffing for s390x and also opened an associated PR: https://bugzilla.redhat.com/show_bug.cgi?id=1964108
Setting Reviewed-in-Sprint as Prashanth is OOTO and this bug is unlikely to be resolved by the end of the sprint
Hi Prashanth, do you think this bug will be resolved before the end of the current sprint (July 24th)? If not, can we add the "reviewed-in-sprint" flag?
Hi Prashanth, do you think this bug will be resolved before the end of this sprint (August 14th)? If not, could you review this bug and set the "reviewed-in-sprint" flag to "+" if appropriate?
Hi Prashanth, do you think this bug will be resolved before the end of the current sprint (September 4th)? If not, can we set the "reviewed-in-sprint" flag?
This bug is reproducible in the latest 4.9 builds per bug triage team.
Adding "reviewed-in-sprint" as the bug will not be resolved before the end of this sprint.
Hi Prashanth, do you think this bug will be resolved before the end of the current sprint (September 25th)? If not, can we set the "reviewed-in-sprint" flag?
Adding "reviewed-in-sprint"
Hi Prashanth, do you think this bug will be resolved before the end of the current sprint (October 16th)? If not, I'd like to add "reviewed-in-sprint" flag.
Adding reviewed-in-sprint. Hi Prashanth - seems BZ 1964108 is ready for your validation - wanted to let you know so that we may be able to validate and fix this bug.
Hi Prashanth, do you think this bug will be resolved before the end of the current sprint (November 6th)? If not, I'd like to add "reviewed-in-sprint" flag
Hi Holger, after discussion this bug further with Prashanth, I decided to assign you as the assignee for this original bug. The summary is that this bug fix is in the RDMA Package in RHEL-8.5 (see BZ 1964108 for details). If we need the bug for 4.10, we might need to raise additional request for backport. Another option is to wait for RHEL 9.0 in RHCOs. It is up to you and team on how you'd like to pursue this bug.
Hi Holger, will this bug be continued to be examined after this sprint (ends on November 27th)? If it will be, then I'd like to set "reviewed-in-sprint" to indicate that work will continue.
Adding "reviewed-in-sprint" as the dependent bug is going through zstream evaluation and it's unlikely that this bug will reach ON_QA before the end of this sprint. Please correct me if the assumption is incorrect.
The dependent bug is still undergoing z-stream evaluation though there was some updates before the holidays. Also, Holger is OOTO and this bug is a non-blocker. Setting reviewed-in-sprint.
Hi Holger, will this bug be continued to be examined in the next sprint (after this week)? If it will be, then I'd like to set the "reviewed-in-sprint" flag.
Folks, FYI. 1. We have seen this issue with OCP 4.10 on Z RoCE installations, including for the most recent publicly available builds tested: 1. OCP 4.10.0-0.nightly-s390x-2022-01-13-230756 2. OCP 4.10.0-0.nightly-s390x-2022-01-14-035349 3. OCP 4.10.0-0.nightly-s390x-2022-01-17-023143 4. OCP 4.10.0-0.nightly-s390x-2022-01-17-171822 5. OCP 4.10.0-fc.2 2. This issue does not seem to impact the installs or functionality of these OCP 4.10 on Z clusters. Thank you, Kyle
Adding "reviewed-in-sprint" flag for now as Kyle has provided comment above and it's not likely that this will be fixed this sprint. Doug will have further discussions with the Z team regarding triaging of this bug.
Hi Holger, will this bug be continued to be examined in the next sprint (after February 19th)? If it will be, then I'd like to set the "reviewed-in-sprint" flag.
Hi Dan, the fix for this bug is in RHEL-8.5, so we are waiting for RHCOS to be rebased on RHEL-8.5 or RHEL-8.6 before we can verify it on OCP.
Thank you Muhammad, setting the flag.
Hi Muhammad, do you know if the rebase has happened or if we are still waiting for the rebase? If we are still waiting, I'd like to set reviewed-in-sprint
Hi Dan, rebase has happened starting with the latest 4.11 nightly. But the problem is still seen. So we should set reviewed-in-sprint flag :)
Thanks Muhammad - adding reviewed-in-sprint
OCP 4.11 is still on RHEL 8.4 the rebase will be soon, but has not yet happened, I think it is expected that this is not yet resolved
Hi Muhammad, I'm doing my sprintly check-in - since the rebase has not yet happened per Comment 68, would it be right to assume that this bug will continue on in the next sprint?
Hi Dan, yes, please add the flag.
Thanks Muhammad, setting the flag.
Hi Muhammad, do you know if this bug is ready to be verified during this sprint? It seems that the rebase is still ongoing so if we are not ready to verify, I'd like to set the reviewed-in-sprint flag.
Hi Dan, please add the flag.
Thank you Muhammad. Adding the flag.
Hi Muhammad, is this bug still waiting for rebase to be validated? If so, we might want to wait until later this week to evaluate whether "reviewed-in-sprint" flag needs to be set (as rebase is still ongoing).
Rebase is complete and nightlies have been updated to the latest RHCOS. Muhammad is working on this but do not anticipate completion before end of this sprint. Setting reivewed-in-sprint
Correction from above Comment 76 - rebase is ongoing (not complete)
RHCOS was rebased to RHEL-8.6 for a while and I have verified that this BZ is resolved with RHEL-8.6 based RHCOS. # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.0-0.nightly-s390x-2022-05-10-223025 True False 7d14h Cluster version is 4.11.0-0.nightly-s390x-2022-05-10-223025 # ssh core.lnxne.boe The authenticity of host 'worker-0.ocp-m1314001.lnxne.boe (172.23.231.21)' can't be established. ECDSA key fingerprint is SHA256:5jyk8gefS8Tur+MmGd8l9LlIX+fu1Kv2jURCNINkfiM. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added 'worker-0.ocp-m1314001.lnxne.boe,172.23.231.21' (ECDSA) to the list of known hosts. Red Hat Enterprise Linux CoreOS 411.86.202205030351-0 Part of OpenShift 4.11, RHCOS is a Kubernetes native operating system managed by the Machine Config Operator (`clusteroperator/machine-config`). WARNING: Direct SSH access to machines is not recommended; instead, make configuration changes via `machineconfig` objects: https://docs.openshift.com/container-platform/4.11/architecture/architecture-rhcos.html --- [core@worker-0 ~]$ ip a .... 814: enP261s69: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 82:06:2d:0c:b8:b0 brd ff:ff:ff:ff:ff:ff inet6 fe80::2e48:c747:240c:9440/64 scope link noprefixroute valid_lft forever preferred_lft forever 815: enP261s69d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 82:06:2d:0c:b8:b1 brd ff:ff:ff:ff:ff:ff inet6 fe80::f442:3710:c37e:92ae/64 scope link noprefixroute valid_lft forever preferred_lft forever Thanks Prashanth!
Per Muhammad's Comment 78, I'm marking this bug as VERIFIED.
Marking closed. This was verified in 4.11.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days