Description of problem:
1. When attempting to install a zVM hosted OCP 4.9 on Z cluster with any of the 3 current builds from August 12, 2021, the installations fail with either the master nodes not booting successfully from the bootstrap node (for builds 4.9.0-0.nightly-s390x-2021-08-12-033000 and 4.9.0-0.nightly-s390x-2021-08-12-04400) or not successfully completing installation (for build 4.9.0-0.nightly-s390x-2021-08-12-055355).
  
2. Using the (current) latest OCP 4.9 build from August 12, 2021, build 4.9.0-0.nightly-s390x-2021-08-12-055355, the master nodes install but fail to/never join the OCP cluster.

3. Here is an install of the 4.9.0-0.nightly-s390x-2021-08-12-055355 build with the bootstrap core userid login, which shows a "crio.service" failure and the following unhealthy cluster information as displayed by the
"journalctl -b -f -u release-image.service -u bootkube.service" command:


This is the bootstrap node; it will be destroyed when the master is fully up.

The primary services are release-image.service followed by bootkube.service. To watch their status, run e.g.

  journalctl -b -f -u release-image.service -u bootkube.service
[systemd]
Failed Units: 1
  crio.service
[core@bootstrap-0 ~]$ journalctl -b -f -u release-image.service -u bootkube.service
-- Logs begin at Thu 2021-08-12 07:53:37 UTC. --
Aug 12 08:02:08 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2409]: Error: unhealthy cluster
Aug 12 08:02:09 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2409]: etcdctl failed. Retrying in 5 seconds...
Aug 12 08:02:19 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2409]: {"level":"warn","ts":1628755339.550245,"logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001a2000/#initially=[https://localhost:2379]","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp [::1]:2379: connect: connection refused\""}
Aug 12 08:02:19 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2409]: https://localhost:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Aug 12 08:02:19 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2409]: Error: unhealthy cluster
Aug 12 08:02:19 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2409]: etcdctl failed. Retrying in 5 seconds...
Aug 12 08:02:30 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2409]: {"level":"warn","ts":1628755350.320205,"logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000550380/#initially=[https://localhost:2379]","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp [::1]:2379: connect: connection refused\""}
Aug 12 08:02:30 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2409]: https://localhost:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Aug 12 08:02:30 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2409]: Error: unhealthy cluster
Aug 12 08:02:30 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2409]: etcdctl failed. Retrying in 5 seconds...
q
^C
[core@bootstrap-0 ~]$


4. Here is a second install of the 4.9.0-0.nightly-s390x-2021-08-12-055355 build with the bootstrap core userid login, which shows a "crio.service" failure and the following unhealthy cluster information as displayed by the
"journalctl -b -f -u release-image.service -u bootkube.service" command:

This is the bootstrap node; it will be destroyed when the master is fully up.

The primary services are release-image.service followed by bootkube.service. To watch their status, run e.g.

  journalctl -b -f -u release-image.service -u bootkube.service
[systemd]
Failed Units: 1
  crio.service
[core@bootstrap-0 ~]$ journalctl -b -f -u release-image.service -u bootkube.service
-- Logs begin at Thu 2021-08-12 08:15:58 UTC. --
Aug 12 08:20:56 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2388]: Error: unhealthy cluster
Aug 12 08:20:57 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2388]: etcdctl failed. Retrying in 5 seconds...
Aug 12 08:21:07 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2388]: {"level":"warn","ts":1628756467.5806026,"logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000394a80/#initially=[https://localhost:2379]","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp [::1]:2379: connect: connection refused\""}
Aug 12 08:21:07 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2388]: https://localhost:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Aug 12 08:21:07 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2388]: Error: unhealthy cluster
Aug 12 08:21:07 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2388]: etcdctl failed. Retrying in 5 seconds...
Aug 12 08:21:18 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2388]: {"level":"warn","ts":1628756478.3589513,"logger":"client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0003e4700/#initially=[https://localhost:2379]","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp [::1]:2379: connect: connection refused\""}
Aug 12 08:21:18 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2388]: https://localhost:2379 is unhealthy: failed to commit proposal: context deadline exceeded
Aug 12 08:21:18 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2388]: Error: unhealthy cluster
Aug 12 08:21:18 bootstrap-0.pok-220.ocptest.pok.stglabs.ibm.com bootkube.sh[2388]: etcdctl failed. Retrying in 5 seconds...
^C
[core@bootstrap-0 ~]$




Version-Release number of selected component (if applicable):
1. 4.9.0-0.nightly-s390x-2021-08-12-033000
2. 4.9.0-0.nightly-s390x-2021-08-12-044003
3. 4.9.0-0.nightly-s390x-2021-08-12-055355

How reproducible:
Consistently reproducible

Steps to Reproduce:
1. Initiate installation of a z/VM hosted OCP 4.9.0 on Z cluster using the OCP 4.9.0-0.nightly-s390x-2021-08-12-055355 build and the corresponding RHCOS 49.84.202108100912-0 build.


Actual results:
Failed installation of z/VM hosted OCP 4.9 on Z cluster.  Installation does not proceed beyond the master/control plane node installation phase, with no master/control plane nodes joining the OCP cluster.

Expected results:
Successful installation of the z/VM hosted OCP 4.9 on Z cluster.

Additional info:


Thank you.