Bug 1879510

*** Bug 1879511 has been marked as a duplicate of this bug. ***

looks like using the preview core OS is not repeatable.  Worked one time, but over night the cluster went unhealthy.

Still doing some testing.

Hi Ray,

Can you provide some logs showing the failures and the version of the the UPI installer in use? Also, when you way you're using EL8 and Ubuntu 20 VMs can you provide some background around that as well? For example, are you using the linked initramfs/kernel but trying to boot RHEL8 or Ubuntu 20 or do they just happen to be on the VMs and are going to be overwritten?

For the bootstrap, masters, and workers, they are deploying into empty disks.


The boot using the initramfs and kernel we pull from Redhat.

There are no Ubuntu vms (edited) 

I guess that I can't update the bugzilla.

The upi level is the nightly build that you downloaded, and we can't get logs. We can't access the system while it is building and it doesn't put anything on the disk, so there are no logs to pull. Let us know if you have specific ways to get a log.

Hi Ray,

Is it fair to assume the nightly build that you downloaded is one of the nightly OpenShift Installer builds from the 10th of this month as well? There has been quite a few fixes in nightlies since then and I think it would be worth trying the latest available artifacts. If you're still hitting an issue after that we can go from there and try to see why the VMs are unable to pull down required content.

> but fails to apply/act on the APPEND boot options

To verify, you mean append via PXE and not append via QEMU (or another method) correct?

> The upi level is the nightly build that you downloaded, and we can't get logs. We can't access the system while it is building and it doesn't put anything on the disk, so there are no logs to pull. Let us know if you have specific ways to get a log.

Is there anything on the QEMU console that indicates an issue?

for RHCOS latest is 9/10  https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest/

for client https://mirror.openshift.com/pub/openshift-v4/clients/ocp-dev-preview/latest/

and using the 4.5 RHCOS is not working to install 4.6 either.

Charlie Andrews from infrastructure team is providing as much detail as he can at this point.  Here is his response to last update.

1 qemu kernel boot passed thorough append section to booted kernel

2 installer kernel seems to boot normally for upi and sits after 2-8 seconds of kernel boot 
log messages but no actual errors.  This is like normal.  However it does not connect to any 
of the http provided links in the append and does not install anything to the block device.

I should be noted that everything broken when we were force to move off the rhcos-4.6.0-0.nightly-2020-08-26-093617, so one key would be to look at the changes put into rhcos versions since this version was deleted.

Can we get some action on this case. It is blocking any forward development. It is very evident that changes have gone into the rhcos  after rhcos-4.6.0-0.nightly-2020-08-26-093617 that have broken baremetal.

>  It is very evident that changes have gone into the rhcos  after rhcos-4.6.0-0.nightly-2020-08-26-093617 that have broken baremetal.

Note we have multiple tests for baremetal-style installations, among them:

"kola testiso"
https://github.com/coreos/coreos-assembler/blob/2851c6518fda7d25e51253f5275b4a546c2df0d0/mantle/cmd/kola/testiso.go
is run against every build - we don't ship a CoreOS build unless it passes that, and while it doesn't cover
everything it definitely uses the "APPEND" directive with pxe:
https://github.com/coreos/coreos-assembler/blob/db85a3c3c40ba38b5a92228a83d62f2d4507cbf5/mantle/platform/metal.go#L356

There's also e2e-metal-ipi which does PXE installs on Packet.net

Can you include a bit more information about what exactly you're providing in the PXE configuration?

> It is very evident that changes have gone into the rhcos  after rhcos-4.6.0-0.nightly-2020-08-26-093617 that have broken baremetal.

I didn't look closely but it's possible you're being bit by the Ignition Spec 3 transition for 4.6:
https://github.com/openshift/enhancements/blob/master/enhancements/machine-config/ignition-spec-dual-support.md
https://github.com/openshift/installer/pull/3871 merged in late July though, not August, so it may not be that.

> fails to apply/act on the APPEND boot options and never downloads the metal/ignition files from http webserver to actually continue the install.

Note that there have been multiple changes to the bare metal install workflow in 4.6. One of them is that you no longer need to specify a metal image. It is now part of the live images already.

We're still ironing out the documentation for OpenShift 4.6, but there is a first draft up at https://github.com/openshift/openshift-docs/pull/25709 which you may find useful. Notably, there is now also a rootfs image that you need to either append as a secondary initrd in your PXE config, or point to it over HTTP via `coreos.live.rootfs_url`.

So in the end, you normally should only need to specify `coreos.inst.install_dev`, `coreos.inst.ignition_url`, and perhaps `coreos.live.rootfs_url` as mentioned above.

You may also find the upstream documentation for this useful:
- https://docs.fedoraproject.org/en-US/fedora-coreos/bare-metal/#_installing_from_pxe
- https://docs.fedoraproject.org/en-US/fedora-coreos/bare-metal/#_pxe_rootfs_image

(RHCOS has already adapted to the rootfs image approach.)

I am back from vacation.  Just a quick update.

Andrew from infrastructure team was able to install the latest 4.6 nightly (4.6.0-0.nightly-2020-09-26-140139). However it required a hard reboot of the masters during the bootstrapping, and a hard reboot of the workers after bootstrapping was done.

Created attachment 1717282 [details]
1 master spinning

A hard reboot of all 3 allowed the bootstrapping to finish.

See image 2 for more info on workers.

Created attachment 1717283 [details]
workers stuck waiting on api server

Despite having been pinging previously. Rebooting them during the bootstrapping didn't help, but rebooting them after bootstrapping was finished (without any workers) caused them to come up properly. From there the cluster built to completion.

Any thoughts on the need for hard reboots to make the install work?

We need the full journal and/or console logs from the masters/workers in order to determine what happened leading up to the master spinning as shown in comment #13.  Without that data we can only speculate what is happening.

got all journal logs collected, file is 584MB.  Looking for a place to upload it.

here is a link to the tar ball.  https://drive.google.com/file/d/1snsV9VixRhvaR2Rs-6rXrICDdXYQcQzF/view?usp=sharing

Unfortunately, the journals collected don't show the machines booting and what that initial bringup of the machines look like:

```
$ head -n3 *journal.log
==> 10.17.108.192.journal.log <==
-- Logs begin at Mon 2020-09-28 18:05:30 UTC, end at Tue 2020-09-29 16:19:45 UTC. --
Sep 28 18:05:30 master0.o46-test-custom1.cp.fyre.ibm.com hyperkube[1588]: I0928 18:05:30.260778    1588 factory.go:209] Factory "systemd" can handle container "/system.slice/run-ipcns-95043b8c\\x2dfe2d\\x2d4661\\x2d820a\\x2d9101768fd53b.mount", but ignoring.
Sep 28 18:05:30 master0.o46-test-custom1.cp.fyre.ibm.com hyperkube[1588]: I0928 18:05:30.260787    1588 manager.go:917] ignoring container "/system.slice/run-ipcns-95043b8c\\x2dfe2d\\x2d4661\\x2d820a\\x2d9101768fd53b.mount"

==> 10.17.109.192.journal.log <==
-- Logs begin at Mon 2020-09-28 22:37:48 UTC, end at Tue 2020-09-29 16:20:56 UTC. --
Sep 28 22:37:48 master1.o46-test-custom1.cp.fyre.ibm.com hyperkube[1586]: I0928 22:37:48.308033    1586 factory.go:209] Factory "systemd" can handle container "/system.slice/var-lib-containers-storage-overlay-85255018139a2038ede20345fb72ae19752da648fe98f90f7b39f5064ec82c4a-merged.mount", but ignoring.
Sep 28 22:37:48 master1.o46-test-custom1.cp.fyre.ibm.com hyperkube[1586]: I0928 22:37:48.308045    1586 manager.go:917] ignoring container "/system.slice/var-lib-containers-storage-overlay-85255018139a2038ede20345fb72ae19752da648fe98f90f7b39f5064ec82c4a-merged.mount"

==> 10.17.110.192.journal.log <==
-- Logs begin at Mon 2020-09-28 16:05:13 UTC, end at Tue 2020-09-29 16:22:24 UTC. --
Sep 28 16:22:24 master2.o46-test-custom1.cp.fyre.ibm.com hyperkube[1586]: I0928 16:22:24.618166    1586 handler.go:181] Unable to get Process Stats: couldn't open cpu cgroup procs file /sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c8fea7_45dc_420e_bd4e_a244452aa9c8.slice/crio-cbe17ee93b353fc64684cb389502d9c69838db840ea18ef290d84df4d37afceb.scope/cgroup.procs : open /sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c8fea7_45dc_420e_bd4e_a244452aa9c8.slice/crio-cbe17ee93b353fc64684cb389502d9c69838db840ea18ef290d84df4d37afceb.scope/cgroup.procs: no such file or directory
Sep 28 16:22:24 master2.o46-test-custom1.cp.fyre.ibm.com hyperkube[1586]: I0928 16:22:24.654010    1586 kubelet.go:1891] SyncLoop (SYNC): 1 pods; recyler-pod-master2.o46-test-custom1.cp.fyre.ibm.com_openshift-infra(866d346b936b398c387c029de0e7ebd8)

==> 10.17.111.192.journal.log <==
-- Logs begin at Sun 2020-09-27 10:33:11 UTC, end at Tue 2020-09-29 16:23:40 UTC. --
Sep 28 16:23:40 worker0.o46-test-custom1.cp.fyre.ibm.com hyperkube[1683]: I0928 16:23:40.647528    1683 prober.go:159] Exec-Probe Pod: thanos-querier-b8c6d846d-v6nlp, Container: thanos-query, Command: [sh -c if [ -x "$(command -v curl)" ]; then curl http://localhost:9090/-/healthy; elif [ -x "$(command -v wget)" ]; then wget --quiet --tries=1 --spider http://localhost:9090/-/healthy; else exit 1; fi]
Sep 28 16:23:40 worker0.o46-test-custom1.cp.fyre.ibm.com hyperkube[1683]: I0928 16:23:40.702142    1683 exec.go:60] Exec probe response: "OK  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r100     2  100     2    0     0   2000      0 --:--:-- --:--:-- --:--:--  2000\n"

==> 10.17.112.192.journal.log <==
-- Logs begin at Sun 2020-09-27 15:16:53 UTC, end at Tue 2020-09-29 16:24:03 UTC. --
Sep 28 16:24:03 worker1.o46-test-custom1.cp.fyre.ibm.com hyperkube[1671]: I0928 16:24:03.796370    1671 handler.go:181] Unable to get Process Stats: couldn't open cpu cgroup procs file /sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce458ed6_2c75_4d03_8ec7_93199bf23b26.slice/crio-6edb3e4d31cf81dd9724fc2397a32e99bca79afc95456d818c9459316410516d.scope/cgroup.procs : open /sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce458ed6_2c75_4d03_8ec7_93199bf23b26.slice/crio-6edb3e4d31cf81dd9724fc2397a32e99bca79afc95456d818c9459316410516d.scope/cgroup.procs: no such file or directory
Sep 28 16:24:03 worker1.o46-test-custom1.cp.fyre.ibm.com hyperkube[1671]: I0928 16:24:03.910172    1671 httplog.go:94] "HTTP" verb="GET" URI="/metrics" latency="5.368388ms" userAgent="Prometheus/2.21.0" srcIP="10.17.111.192:42516" resp=200

==> 10.17.113.192.journal.log <==
-- Logs begin at Sun 2020-09-27 20:34:09 UTC, end at Tue 2020-09-29 16:24:31 UTC. --
Sep 28 16:24:32 worker2.o46-test-custom1.cp.fyre.ibm.com hyperkube[1698]: I0928 16:24:32.156470    1698 prober.go:159] Exec-Probe Pod: redhat-operators-gkvrq, Container: registry-server, Command: [grpc_health_probe -addr=:50051]
Sep 28 16:24:32 worker2.o46-test-custom1.cp.fyre.ibm.com hyperkube[1698]: I0928 16:24:32.207150    1698 exec.go:60] Exec probe response: "status: SERVING\n"
```


We need the journals from the initial boot that show what is happening with Ignition and the early boot phases of the nodes.  The journals should start similarly like this:

```
-- Logs begin at Tue 2020-09-29 18:31:22 UTC, end at Tue 2020-09-29 18:32:40 UTC. --
Sep 29 18:31:22 localhost kernel: Linux version 4.18.0-193.23.1.el8_2.x86_64 (mockbuild.eng.bos.redhat.com) (gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)) #1 SMP Thu Sep 17 15:34:16 EDT 2020
Sep 29 18:31:22 localhost kernel: Command line: BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-69818ae3a83104d3913e68baa0bbf130f784ab304a6a41592ffa8650cccdf0af/vmlinuz-4.18.0-193.23.1.el8_2.x86_64 rhcos.root=crypt_rootfs random.trust_cpu=on console=tty0 console=ttyS0,115200n8 ignition.platform.id=qemu rd.luks.options=discard ig0
Sep 29 18:31:22 localhost kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
```

And should include evidence that Ignition is running; for example:

```
$ journalctl -b | grep Ignition
Sep 29 18:31:24 localhost systemd[1]: Starting Ignition (setup user config)...
Sep 29 18:31:24 localhost systemd[1]: Starting Ignition (setup base config)...
Sep 29 18:31:24 localhost systemd[1]: Started Ignition (setup base config).
Sep 29 18:31:24 localhost systemd[1]: Started Ignition (setup user config).
Sep 29 18:31:24 localhost systemd[1]: Starting Ignition (fetch-offline)...
Sep 29 18:31:24 localhost ignition[754]: Ignition 2.6.0
Sep 29 18:31:24 localhost systemd[1]: Started Ignition (fetch-offline).
Sep 29 18:31:24 localhost ignition[754]: Ignition finished successfully
Sep 29 18:31:24 localhost rhcos-fips[767]: No /etc/ignition-machine-config-encapsulated.json found in Ignition config
Sep 29 18:31:24 localhost systemd[1]: Starting Ignition (disks)...
Sep 29 18:31:24 localhost ignition[797]: Ignition 2.6.0
Sep 29 18:31:24 localhost ignition[797]: Ignition finished successfully
Sep 29 18:31:24 localhost systemd[1]: Started Ignition (disks).
Sep 29 18:31:25 cosa-devsh systemd[1]: Starting Ignition OSTree: Mount (firstboot) /sysroot...
Sep 29 18:31:25 cosa-devsh systemd[1]: Started Ignition OSTree: Mount (firstboot) /sysroot.
Sep 29 18:31:25 cosa-devsh systemd[1]: Reached target Ignition Boot Disk Setup.
Sep 29 18:31:25 cosa-devsh systemd[1]: Starting Ignition (mount)...
Sep 29 18:31:25 cosa-devsh ignition[943]: INFO     : Ignition 2.6.0
Sep 29 18:31:25 cosa-devsh systemd[1]: Started Ignition (mount).
Sep 29 18:31:25 cosa-devsh ignition[943]: INFO     : Ignition finished successfully
Sep 29 18:31:26 cosa-devsh systemd[1]: Starting Ignition (files)...
Sep 29 18:31:26 cosa-devsh ignition[974]: INFO     : Ignition 2.6.0
Sep 29 18:31:27 cosa-devsh ignition[974]: INFO     : Ignition finished successfully
Sep 29 18:31:27 cosa-devsh systemd[1]: Started Ignition (files).
Sep 29 18:31:27 cosa-devsh systemd[1]: Reached target Ignition Complete.
```

d(In reply to Ray Ashworth from comment #13)
> Created attachment 1717282 [details]
> 1 master spinning
> 
> A hard reboot of all 3 allowed the bootstrapping to finish.
> 
> See image 2 for more info on workers.

Those messages being printed to the console are likely the result of containers starting/stopping.  For example, when starting a simple Fedora container on my test RHCOS node, I can see similar messages in the journal:

```
Sep 29 18:44:40 cosa-devsh sudo[1775]:     core : TTY=pts/0 ; PWD=/var/home/core ; USER=root ; COMMAND=/bin/podman run -it registry.fedoraproject.org/fedora:32
Sep 29 18:44:40 cosa-devsh sudo[1775]: pam_systemd(sudo:session): Cannot create session: Already running in a session or user slice
Sep 29 18:44:40 cosa-devsh sudo[1775]: pam_unix(sudo:session): session opened for user root by core(uid=0)
Sep 29 18:44:50 cosa-devsh kernel: bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
Sep 29 18:44:50 cosa-devsh NetworkManager[1453]: <info>  [1601405090.9631] manager: (cni-podman0): new Bridge device (/org/freedesktop/NetworkManager/Devices/3)
Sep 29 18:44:50 cosa-devsh systemd-udevd[1875]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Sep 29 18:44:50 cosa-devsh systemd-udevd[1875]: Could not generate persistent MAC address for cni-podman0: No such file or directory
Sep 29 18:44:50 cosa-devsh kernel: IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
Sep 29 18:44:50 cosa-devsh kernel: IPv6: ADDRCONF(NETDEV_UP): vethbb854f86: link is not ready


$ journalctl -b | grep vethbb854f86
Sep 29 18:44:50 cosa-devsh kernel: IPv6: ADDRCONF(NETDEV_UP): vethbb854f86: link is not ready
Sep 29 18:44:50 cosa-devsh kernel: IPv6: ADDRCONF(NETDEV_CHANGE): vethbb854f86: link becomes ready
Sep 29 18:44:50 cosa-devsh kernel: cni-podman0: port 1(vethbb854f86) entered blocking state
Sep 29 18:44:50 cosa-devsh kernel: cni-podman0: port 1(vethbb854f86) entered disabled state
Sep 29 18:44:50 cosa-devsh kernel: device vethbb854f86 entered promiscuous mode
Sep 29 18:44:50 cosa-devsh kernel: cni-podman0: port 1(vethbb854f86) entered blocking state
Sep 29 18:44:50 cosa-devsh kernel: cni-podman0: port 1(vethbb854f86) entered forwarding state
Sep 29 18:44:50 cosa-devsh systemd-udevd[1878]: Could not generate persistent MAC address for vethbb854f86: No such file or directory
Sep 29 18:44:50 cosa-devsh NetworkManager[1453]: <info>  [1601405090.9849] manager: (vethbb854f86): new Veth device (/org/freedesktop/NetworkManager/Devices/4)
Sep 29 18:44:50 cosa-devsh NetworkManager[1453]: <info>  [1601405090.9857] device (vethbb854f86): carrier: link connected
Sep 29 18:45:34 cosa-devsh kernel: cni-podman0: port 1(vethbb854f86) entered disabled state
Sep 29 18:45:34 cosa-devsh kernel: device vethbb854f86 left promiscuous mode
Sep 29 18:45:34 cosa-devsh kernel: cni-podman0: port 1(vethbb854f86) entered disabled state
Sep 29 18:45:34 cosa-devsh NetworkManager[1453]: <info>  [1601405134.9314] device (vethbb854f86): released from master device cni-podman0

```

This report is looking a lot like this other report https://bugzilla.redhat.com/show_bug.cgi?id=1882515

Can you take a look at that other report and confirm that they are same or different.

Since VMWare and KVM/Qemu are both broken.

> This report is looking a lot like this other report https://bugzilla.redhat.com/show_bug.cgi?id=1882515
>
> Can you take a look at that other report and confirm that they are same or different.

It's hard to tell without access to the logs for the first boot. Could you provide those?

If it is indeed the case, then adding virtio-rng to the VMs should also help.

Would it be possible to schedule a bluejean session with the architecture team to discuss our environment in detail.

It appears that RH provided a OVA that worked on the VMWare side.  The question was asked if we could get the matching kernal, initfsram, and metal files to test in our KVM environment.

reading https://bugzilla.redhat.com/show_bug.cgi?id=1882515 it looks like a fix will be coming out in 2 to 3 days.  I am guessing it might help this report as well.  Can someone confirm that.

> I am guessing it might help this report as well.  Can someone confirm that.

Are you deploying the VMs on top of the same hardware (Sandy Bridge hypervisors)?

Do you see a warning in the serial console like:

> crypto/rand: blocked for 60 seconds waiting to read random data from the kernel

this is a different infrastructure using KVM/Qemu, not VMWare based, however hardware is newer that the VMWare environment and we have not seen the crypto/rand message.

The correlation is that both the VMware and bare metal install process worked when we were able to use rhcos-4.6.0-0.nightly-2020-08-26-093617, they both stopped working when we were no longer able to get rhcos-4.6.0-0.nightly-2020-08-26-093617 and had to use 4.6.0-0.nightly-2020-09-10-195619. 

Luckily, we had saved off the ova for rhcos-4.6.0-0.nightly-2020-08-26-093617 for vmware, but not so lucky we did not save off the various files from the same build for bare metal.

> this is a different infrastructure using KVM/Qemu, not VMWare based, however hardware is newer that the VMWare environment and we have not seen the crypto/rand message.

Hmm, it doesn't sound like the same issue then. You would see the initial boot take a long time in the initramfs with the fetch-offline stage appearing stuck.

Though given that it's KVM/qemu, as good practice you should make sure that you're adding a virtio-rng device. See https://github.com/coreos/coreos-assembler/blob/fe9ae781357f781c42f74f0cf26011046a9a286e/mantle/platform/qemu.go#L1190-L1191 for example CLI args to qemu. If you're using libvirt to create these VMs, see https://libvirt.org/formatdomain.html#random-number-generator-device (though make sure to use `/dev/urandom`).

If that is somehow the same issue, the above should help.

(In reply to Ray Ashworth from comment #24)
> reading https://bugzilla.redhat.com/show_bug.cgi?id=1882515 it looks like a
> fix will be coming out in 2 to 3 days.  I am guessing it might help this
> report as well.  Can someone confirm that.

Just in case the change to Ignition may be useful for this BZ:

The new version of Ignition (2.6.0-5.rhaos4.6.git947598e.el8) landed as part of RHCOS 46.82.202010011740-0; if you have access to the RHCOS builds at https://releases-rhcos-art.cloud.privileged.psi.redhat.com/?stream=releases/rhcos-4.6&release=46.82.202010011740-0, you can try out that build again while waiting for the `openshift-install` PR (https://github.com/openshift/installer/pull/4234) to merge.

I should have another update later today.

added in the use of rootfs

{
  "kernel_url": "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest-4.6/rhcos-4.6.0-0.nightly-2020-09-29-013537-x86_64-live-kernel-x86_64",
  "initramfs_url": "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest-4.6/rhcos-4.6.0-0.nightly-2020-09-29-013537-x86_64-live-initramfs.x86_64.img",
  "rootfs_url": "https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/pre-release/latest-4.6/rhcos-4.6.0-0.nightly-2020-09-29-013537-x86_64-live-rootfs.x86_64.img",
  "install_url": "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp-dev-preview/latest-4.6/openshift-install-linux.tar.gz",
  "client_url": "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp-dev-preview/latest-4.6/openshift-client-linux.tar.gz"
}

getting farther, but still need to reboot workers to get install to complete.

as a work around we are building in the worker reboots to help the install complete.  However this issue should get fixed at some point.

Latest deployment attempt is using latest RHCOS 9/29 and 4.6 RC.0

we are seeing success with the following links.  No worker reboot needed.

  "kernel_url": "https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.6/46.82.202010011740-0/x86_64/rhcos-46.82.202010011740-0-live-kernel-x86_64",
  "initramfs_url": "https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.6/46.82.202010011740-0/x86_64/rhcos-46.82.202010011740-0-live-initramfs.x86_64.img",
  "metal_url":"https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.6/46.82.202010011740-0/x86_64/rhcos-46.82.202010011740-0-metal.x86_64.raw.gz",
  "rootfs_url": "https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.6/46.82.202010011740-0/x86_64/rhcos-46.82.202010011740-0-live-rootfs.x86_64.img",
  "install_url": "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/candidate-4.6/openshift-install-linux-4.6.0-rc.1.tar.gz",
  "client_url": "https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/candidate-4.6/openshift-client-linux-4.6.0-rc.1.tar.gz",

we've added an auto reboot of the workers if they end up not pingable.  Using about links shows the auto reboot was not needed on 1 test so far.  2 more tests are running as I type this message.

Using the above releases-art url's are working for coreos.  We will update to the public mirror url when availalbe.

This report may be closed.

Good to hear, thanks!

The public mirror versions should be inbound soon.