Our f34 and f35 container composes have been failing. They are made on f35 aarch64 hardware in armv7 vm's. The failure is because the install in the vm can't fetch anything from the net, because: [ 25.816331] dracut-initqueue[1055]: Warning: can't find installer main image path in .treeinfo [ 25.872371] dracut-initqueue[1175]: % Total % Received % Xferd Average Speed Time Time Time Current [ 25.873249] dracut-initqueue[1175]: Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 [ 25.948377] dracut-initqueue[1175]: curl: (60) SSL certificate problem: certificate is not yet valid ... [ 26.126181] dracut-initqueue[1178]: Warning: Downloading 'https://kojipkgs.fedoraproject.org/compose/35/latest-Fedora-35/compose/Everything/armhfp/os//LiveOS/squashfs.img' failed! [ 26.136862] dracut-initqueue[1055]: Warning: anaconda: failed to fetch stage2 from https://kojipkgs.fedoraproject.org/compose/35/latest-Fedora-35/compose/Everything/armhfp/os/ The date in the vm seems to be (by looking at dmesg -T): [Thu Oct 7 23:59:50 2021] Booting Linux on physical CPU 0x0 [Thu Oct 7 23:59:50 2021] Linux version 5.14.10-300.fc35.armv7hl (mockbuild.fedo raproject.org) (gcc (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1), GNU ld version 2.37-10.fc35) #1 SMP Thu Oct 7 22:07:42 UTC 2021 ... of course our ssl cert wasn't valid back in oct of 2021. ;( The host qemu commandline: qemu 664847 127 4.0 5664596 664560 ? Sl 21:24 9:46 /usr/bin/qemu-system-aarch64 -name guest=factory-build-1ba4e4c1-c87d-4253-8a26-5d6f12bde64e,debug-threads=on -S -object {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-factory-build-1ba4e4/master-key.aes"} -blockdev {"driver":"file","filename":"/usr/share/edk2/arm/QEMU_EFI-pflash.raw","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"} -blockdev {"driver":"file","filename":"/var/lib/libvirt/qemu/nvram/factory-build-1ba4e4c1-c87d-4253-8a26-5d6f12bde64e_VARS.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"} -machine virt-6.1,accel=kvm,usb=off,dump-guest-core=off,gic-version=2,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,memory-backend=mach-virt.ram -cpu host,aarch64=off -m 4096 -object {"qom-type":"memory-backend-ram","id":"mach-virt.ram","size":4294967296} -overcommit mem-lock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 60fbfcff-b46e-433a-9d18-96f79ad4cadc -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=34,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-reboot -no-acpi -boot strict=on -kernel /var/tmp/koji/tasks/6205/86046205/scratch_images/factory-build-1ba4e4c1-c87d-4253-8a26-5d6f12bde64e-kernel -initrd /var/tmp/koji/tasks/6205/86046205/scratch_images/factory-build-1ba4e4c1-c87d-4253-8a26-5d6f12bde64e-ramdisk -append inst.method=https://kojipkgs.fedoraproject.org/compose/35/latest-Fedora-35/compose/Everything/armhfp/os/ inst.ks=file:/ks.cfg -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 -device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 -device pcie-pci-bridge,id=pci.3,bus=pci.1,addr=0x0 -device pcie-root-port,port=0xa,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x2 -device pcie-root-port,port=0xb,chassis=5,id=pci.5,bus=pcie.0,addr=0x1.0x3 -device pcie-root-port,port=0xc,chassis=6,id=pci.6,bus=pcie.0,addr=0x1.0x4 -device pcie-root-port,port=0xd,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x5 -device pcie-root-port,port=0xe,chassis=8,id=pci.8,bus=pcie.0,addr=0x1.0x6 -device piix3-usb-uhci,id=usb,bus=pci.3,addr=0x1 -device virtio-serial-pci,id=virtio-serial0,bus=pci.4,addr=0x0 -blockdev {"driver":"file","filename":"/var/tmp/koji/tasks/6205/86046205/output_image/1ba4e4c1-c87d-4253-8a26-5d6f12bde64e.body","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":false,"driver":"raw","file":"libvirt-1-storage"} -device virtio-blk-pci,bus=pci.5,addr=0x0,drive=libvirt-1-format,id=virtio-disk0,bootindex=1 -netdev tap,fd=36,id=hostnet0,vhost=on,vhostfd=37 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:14:3b:0d,bus=pci.2,addr=0x0 -chardev pty,id=charserial0 -serial chardev:charserial0 -chardev socket,id=charserial1,host=127.0.0.1,port=55401,server=on,wait=off -serial chardev:charserial1 -chardev socket,id=charchannel0,host=127.0.0.1,port=58873,server=on,wait=off -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.fedoraproject.anaconda.log.0 -device usb-tablet,id=input0,bus=usb.0,port=1 -device usb-kbd,id=input1,bus=usb.0,port=2 -audiodev id=audio1,driver=none -vnc 127.0.0.1:0,audiodev=audio1 -device virtio-vga,id=video0,max_outputs=1,bus=pci.7,addr=0x0 -object {"qom-type":"rng-random","id":"objrng0","filename":"/dev/random"} -device virtio-rng-pci,rng=objrng0,id=rng0,max-bytes=1024,period=1000,bus=pci.6,addr=0x0 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on Linux buildhw-a64-06.iad2.fedoraproject.org 5.16.18-200.fc35.aarch64 #1 SMP Mon Mar 28 13:53:06 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux qemu-6.1.0-14.fc35.aarch64 edk2-arm-20211126gitbb1bba3d7767-1.fc35.noarch This only happens on 32bit arm. I would think it would get the default time from the host, but it sure doesn't seem so. ;( aarch64_host% date Thu Apr 21 09:37:03 PM UTC 2022 Happy to gather more info, etc.
Any errors or warnings about clocks in either the guest or host dmesg? Also what is the clocksource inside the guest? For me: $ cat /sys/devices/system/clocksource/clocksource0/current_clocksource arch_sys_counter
Also you might want to try to see if later qemu fixes this by selectively updating qemu from F36 or Rawhide. qemu 7.0 should be added to Rawhide soon.
In guest: sh-5.1# cat /sys/devices/system/clocksource/clocksource0/current_clocksource arch_sys_counter In dmesg I only see: [Thu Oct 7 23:59:50 2021] arch_timer: cp15 timer(s) running at 50.00MHz (virt). [Thu Oct 7 23:59:50 2021] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xb88127 36b, max_idle_ns: 440795202655 ns [Thu Oct 7 23:59:50 2021] sched_clock: 56 bits at 50MHz, resolution 20ns, wraps every 4398046511100n s [Thu Oct 7 23:59:50 2021] Switching to timer-based delay loop, resolution 20ns ... [Thu Oct 7 23:59:50 2021] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns : 1911260446275000 ns ... [Thu Oct 7 23:59:50 2021] clocksource: Switched to clocksource arch_sys_counter
ok. I tried qemu-7 from rawhide. No change. I tried the rawhide edk2-arm package. No change.
Just FYI here, note that this is causing f34/f35 daily container updates/builds to fail... so those branches aren't getting any container updates. ;( Workarounds welcome.
I asked around inside Red Hat in case anyone had any idea, but no one recognised the problem. Is there a simple reproducer? eg. Downloading the armv7 cloud installer image and just booting it should work, right?
IIUC Fedora has been using 32-bit arm vs on 64-bit aarch64 host for a long time and presumably this has worked correctly historically. With that in mind, I'm pretty interested in what changed in Fedora's infra that coincided with this data problem arising. If there was either a host or guest software update, and there's a log of changed packages, that would be very useful in giving a pointer to narrow down where to look for the root cause.
So, what changed is on 2022-03-02 we updated the *.fedoraproject.org ssl cert. The old one was a cert from 2020 that expired 2022-03-07, ie it was valid for 2020-03-07 to 2022-03-07. The new one is only valid _after_ Thu, 27 Jan 2022. So, this likely was happening for a long time, but the cert was just valid in the crazy 2021 time it got. :( As far as reproducer, booting a netinstall in a vm I would think would work?
(In reply to Kevin Fenzi from comment #8) > As far as reproducer, booting a netinstall in a vm I would think would work? I tried lots of things last night but couldn't get either a serial console or a display to appear. (Dammit Arm, standardize the serial port ...) If you have a qemu incantation that actually works I can go for the bisection.
ok. I'll try and get a reproducer... right now it's a stack o doom: cron job calls pungi, which calls koji image-build, which calls imagefactory, which calls oz, which calls libvirtd to make a guest, which uses qemu. :( It may be possible to short circut to imagefactory at least. - Install imagefactory imagefactory-plugins-Docker oz on a aarch64 machine that can run 32bit arm vm guests - get the tdl-armhfp.xml and koji-f35-build-86738497-base.ks files: https://kojipkgs.fedoraproject.org//work/tasks/8497/86738497/koji-f35-build-86738497-base.ks https://kojipkgs.fedoraproject.org//work/tasks/8497/86738497/tdl-armhfp.xml - run: imagefactory --debug base_image tdl-armvhfp.xml --file-parameter install_script koji-f35-build-86738497-base.ks --parameter offline_icicle true - in another window wait for the vm to appear in libvirt and 'virsh console' on it.
I was thinking something more along the lines of this (but not broken). qemu-system-aarch64 -machine virt-6.1,accel=kvm,usb=off,dump-guest-core=off,gic-version=2 -cpu host,aarch64=off -m 2048 -drive file=Fedora-Server-netinst-armhfp-34-1.2.iso,snapshot=on,if=none -device virtio-scsi -boot menu=on -rtc base=utc -device virtio-vga,id=video0,max_outputs=1 -device virtio-serial-pci,id=virtio-serial0 Also, are you running this on aarch64 host? (I was trying these commands on a Raspberry Pi 4B).
Yes, the host is aarch64... lenovo emag My understanding is that not many aarch64 socs support 32bit arm virt these days. I have no idea on the PI4B. ;(
Pi4 does support 32 bit guests using KVM. The command line given in comment 0 uses -machine virt-6.1,accel=kvm,... which I believe implies that it must be using KVM for the guest (not TCG) as otherwise qemu would print an error. I can't actually verify that because I don't have any non-32-bit-KVM-capable aarch64 hardware. Anyway this does matter because the KVM & TCG paths are wildly different and I wouldn't want to be bisecting the wrong code.
(In reply to Richard W.M. Jones from comment #13) > Pi4 does support 32 bit guests using KVM. > > The command line given in comment 0 uses -machine virt-6.1,accel=kvm,... > which I believe implies that it must be using KVM for the guest (not TCG) Yup, the command line in comment 0, like comment 11, shows that KVM is in use. comment 11 is missing edk2 though. I think it's worth experimenting with and without edk2, because with edk2 the guest may be getting time and date from an EFI runtime call.
(In reply to Kevin Fenzi from comment #12) > Yes, the host is aarch64... lenovo emag > My understanding is that not many aarch64 socs support 32bit arm virt these > days. I have no idea on the PI4B. ;( RPi4 is fine, with caveats, it's more enterprise servers ATM that don't support 32bit emulation at EL1 (AKA running a VM).
Does the VM have a RTC passed, I see "-rtc base=utc" but not an explicit rtc. I'm guessing qemu sets the clock on the virtal RTC to set the time but someone who knows the depths of qemu and how it deals with that would be useful.
This is now breaking the F36 IoT compose. https://koji.fedoraproject.org/koji/taskinfo?taskID=91252992
@pwhalen how do you know it's this issue? I can't tell from the screenshot in from koji link. Is there more guest log data somewhere? We need someone to distill this to a reproducer that doesn't involve koji. Below is libvirt XML that oz is generating in Paul's koji build. kernel + initrd are from this compose link: https://kojipkgs.fedoraproject.org/compose/36/Fedora-36-20220504.1/compose/Everything/armhfp/os/ <domain type="kvm"> <name>factory-build-2bd05547-282d-4d7d-8082-9d8a7e8b13cf</name> <memory>4194304</memory> <currentMemory>4194304</currentMemory> <uuid>82d29831-29c0-473e-bb4e-29709cc09587</uuid> <clock offset="utc"/> <vcpu>4</vcpu> <features/> <cpu mode="host-passthrough"/> <os> <type arch="armv7l" machine="virt">hvm</type> <kernel>/var/tmp/koji/tasks/2992/91252992/scratch_images/factory-build-2bd05547-282d-4d7d-8082-9d8a7e8b13cf-kernel</kernel> <initrd>/var/tmp/koji/tasks/2992/91252992/scratch_images/factory-build-2bd05547-282d-4d7d-8082-9d8a7e8b13cf-ramdisk</initrd> <cmdline>inst.method=https://kojipkgs.fedoraproject.org/compose/36/Fedora-36-20220504.1/compose/Everything/armhfp/os/ inst.ks=file:/ks.cfg</cmdline> <loader readonly="yes" type="pflash">/usr/share/edk2/arm/QEMU_EFI-pflash.raw</loader> <nvram template="/usr/share/edk2/arm/vars-template-pflash.raw"/> </os> <on_poweroff>destroy</on_poweroff> <on_reboot>destroy</on_reboot> <on_crash>destroy</on_crash> <devices> <graphics port="-1" type="vnc"/> <interface type="bridge"> <source bridge="virbr0"/> <mac address="52:54:00:9a:76:5b"/> <model type="virtio"/> </interface> <input bus="usb" type="tablet"/> <controller type="usb" index="0"/> <input type="keyboard" bus="usb"/> <serial type="pty"> <target port="0"/> </serial> <serial type="tcp"> <source mode="bind" host="127.0.0.1" service="36878"/> <protocol type="raw"/> <target port="1"/> </serial> <channel type="tcp"> <source mode="bind" host="127.0.0.1" service="39057"/> <target type="virtio" name="org.fedoraproject.anaconda.log.0"/> </channel> <rng model="virtio"> <rate bytes="1024" period="1000"/> <backend model="random">/dev/random</backend> </rng> <disk device="disk" type="file"> <target dev="vda" bus="virtio"/> <source file="/var/tmp/koji/tasks/2992/91252992/output_image/2bd05547-282d-4d7d-8082-9d8a7e8b13cf.body"/> <driver name="qemu" type="raw"/> </disk> </devices> </domain>
(In reply to Cole Robinson from comment #18) > @pwhalen how do you know it's this issue? I can't tell from the screenshot > in from koji link. Is there more guest log data somewhere? Reproduced running imagefactory locally. > > We need someone to distill this to a reproducer that doesn't involve koji. > Below is libvirt XML that oz is generating in Paul's koji build. kernel + > initrd are from this compose link: > https://kojipkgs.fedoraproject.org/compose/36/Fedora-36-20220504.1/compose/ > Everything/armhfp/os/ It's also reproducible on an aarch64 host (F36 or F37): virt-install \ --name f36-armhfp --ram 3072 --arch armv7l --os-variant fedora22 \ --disk /var/lib/libvirt/images/f36-armhfp.raw,bus=virtio,format=raw,size=40 \ --location=https://dl.fedoraproject.org/pub/fedora/linux/releases/36/Everything/armhfp/os/ .. [ 22.893099] dracut-initqueue[1019]: Warning: can't find installer main image path in .treeinfo [ 22.936160] dracut-initqueue[1155]: % Total % Received % Xferd Average Speed Time Time Time Current [ 22.936383] dracut-initqueue[1155]: Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 [ 23.636210] dracut-initqueue[1155]: curl: (60) SSL certificate problem: certificate is not yet valid [ 23.636539] dracut-initqueue[1155]: More details here: https://curl.se/docs/sslcerts.html [ 23.639358] dracut-initqueue[1155]: curl failed to verify the legitimacy of the server and therefore could not [ 23.645626] dracut-initqueue[1155]: establish a secure connection to it. To learn more about this situation and [ 23.648785] dracut-initqueue[1155]: how to fix it, please visit the web page mentioned above. [ 23.662379] dracut-initqueue[1149]: Warning: Downloading 'https://dl.fedoraproject.org/pub/fedora/linux/releases/36/Everything/armhfp/os//images/install.img' failed!
rjones can you give pwhalen's reproducer a spin on your aarch64 hardware?(In reply to Paul Whalen from comment #19) > (In reply to Cole Robinson from comment #18) > > @pwhalen how do you know it's this issue? I can't tell from the screenshot > > in from koji link. Is there more guest log data somewhere? > > Reproduced running imagefactory locally. > Ok thanks. So reproducing locally confirmed the date is wrong, which will break ssl validation. > > It's also reproducible on an aarch64 host (F36 or F37): > > virt-install \ > --name f36-armhfp --ram 3072 --arch armv7l --os-variant fedora22 > \ > --disk > /var/lib/libvirt/images/f36-armhfp.raw,bus=virtio,format=raw,size=40 \ > > --location=https://dl.fedoraproject.org/pub/fedora/linux/releases/36/ > Everything/armhfp/os/ > rjones can you give that a spin on your aarch64 hardware? ccing abologna too since I think he often looks at aarch64 virt issues
$ sudo -E virt-install --name tmp-f36-armhfp --ram 3072 --arch armv7l --os-variant fedora22 --disk /var/lib/libvirt/images/tmp-f36-armhfp.raw,bus=virtio,format=raw,size=40 --location=https://dl.fedoraproject.org/pub/fedora/linux/releases/36/Everything/armhfp/os/ Starting install... Retrieving 'vmlinuz' | 7.5 MB 00:01 ... Retrieving 'initrd.img' | 69 MB 00:13 ... Allocating 'tmp-f36-armhfp.raw' | 0 B 00:00 ... Removing disk 'tmp-f36-armhfp.raw' | 0 B 00:00 ERROR unsupported configuration: TPM model 'tpm-tis' is only available for x86 and aarch64 guests Domain installation does not appear to have been successful. If it was, you can restart your domain by running: virsh --connect qemu:///system start tmp-f36-armhfp otherwise, please restart your installation. --- Can you try a more recent --os-variant? fedora35 or fedora36 should work.
The TPM error is https://bugzilla.redhat.com/show_bug.cgi?id=2078995 Cole, is there a fixed virt-manager package for that?
(In reply to Richard W.M. Jones from comment #22) > The TPM error is https://bugzilla.redhat.com/show_bug.cgi?id=2078995 > Cole, is there a fixed virt-manager package for that? Working on it. In the mean time you can avoid it with `--tpm none`
This message is a reminder that Fedora Linux 36 is nearing its end of life. Fedora will stop maintaining and issuing updates for Fedora Linux 36 on 2023-05-16. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a 'version' of '36'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, change the 'version' to a later Fedora Linux version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see it. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora Linux 36 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora Linux, you are encouraged to change the 'version' to a later version prior to this bug being closed.
Fedora Linux 36 entered end-of-life (EOL) status on 2023-05-16. Fedora Linux 36 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora Linux please feel free to reopen this bug against that version. Note that the version field may be hidden. Click the "Show advanced fields" button if you do not see the version field. If you are unable to reopen this bug, please file a new report against an active release. Thank you for reporting this bug and we are sorry it could not be fixed.