Created attachment 1522677 [details] journalctl --no-hostname -b | grep -v sshd > dmesg.txt 1. Please describe the problem: after updating fedora 29 from 4.18.16-300.fc29.ppc64le to 4.20.3-200.fc29.ppc64le Fedora did not boot anymore. Rebooting with Kernel 4.18 worked Stopped boot process after crng: [ 9.658698] random: crng done (trusting CPU's manufacturer) [ 143.141563] dracut-initqueue[449]: Warning: dracut-initqueue timeout - starting timeout scripts 2. What is the Version-Release number of the kernel: 4.20.3-200 3. Did it work previously in Fedora? If so, what kernel version did the issue *first* appear? Old kernels are available for download at https://koji.fedoraproject.org/koji/packageinfo?packageID=8 : 4. Can you reproduce this issue? If so, please provide the steps to reproduce the issue below: yes, reboot using Kernel 4.20.3-200 5. Does this problem occur with the latest Rawhide kernel? To install the Rawhide kernel, run ``sudo dnf install fedora-repos-rawhide`` followed by ``sudo dnf update --enablerepo=rawhide kernel``: this actually fails: warning: /var/cache/dnf/rawhide-1c578a7639193129/packages/kernel-5.0.0-0.rc3.git0.1.fc30.ppc64le.rpm: Header V3 RSA/SHA256 Signature, key ID cfc659b9: NOKEY Fedora - Rawhide - Developmental packages for the next Fedora release 1.6 kB/s | 1.6 kB 00:01 Importing GPG key 0x429476B4: Userid : "Fedora 29 (29) <fedora-29>" Fingerprint: 5A03 B4DD 8254 ECA0 2FDA 1637 A20A A56B 4294 76B4 From : /etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-29-ppc64le Is this ok [y/N]: y Key imported successfully Import of key(s) didn't help, wrong key(s)? Public key for kernel-5.0.0-0.rc3.git0.1.fc30.ppc64le.rpm is not installed. Failing package is: kernel-5.0.0-0.rc3.git0.1.fc30.ppc64le GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-29-ppc64le Public key for kernel-core-5.0.0-0.rc3.git0.1.fc30.ppc64le.rpm is not installed. Failing package is: kernel-core-5.0.0-0.rc3.git0.1.fc30.ppc64le GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-29-ppc64le Public key for kernel-modules-5.0.0-0.rc3.git0.1.fc30.ppc64le.rpm is not installed. Failing package is: kernel-modules-5.0.0-0.rc3.git0.1.fc30.ppc64le GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-29-ppc64le The downloaded packages were saved in cache until the next successful transaction. You can remove cached packages by executing 'dnf clean packages'. Error: GPG check FAILED 6. Are you running any modules that not shipped with directly Fedora's kernel?: no 7. Please attach the kernel logs. You can get the complete kernel log for a boot with ``journalctl --no-hostname -k > dmesg.txt``. If the issue occurred on a previous boot, use the journalctl ``-b`` flag. this actually does not work either and shows the log of kernel 4.18 only, but I've nevertheless attached the output
Even I am seeing similar issue on ppc64le with kernel-4.20.3-200.fc29.ppc64le . I am trying to install a fresh vm using Atomic Host iso from https://kojipkgs.fedoraproject.org/compose/updates/Fedora-29-updates-20190122.0/compose/AtomicHost/ppc64le/iso/Fedora-AtomicHost-ostree-ppc64le-29-20190122.0.iso . Vm install hangs at following steps during boot: # virt-install --name fah-20190122 --ram 2048 --vcpus 1 --os-type=linux --os-variant=fedora26 --disk path=/var/lib/libvirt/images/f29_20190122.qcow2,size=10,bus=virtio,format=qcow2 --network bridge=virbr0 --nographic --check path_in_use=off -c Fedora-AtomicHost-ostree-ppc64le-29-20190122.0.iso ... OF stdout device is: /vdevice/vty@30000000 Preparing to boot Linux version 4.20.3-200.fc29.ppc64le (mockbuild.fedoraproject.org) (gcc version 8.2.1 20181215 (Red Hat 8.2.1-6) (GCC)) #1 SMP Thu Jan 17 15:04:41 UTC 2019 Detected machine type: 0000000000000101 command line: BOOT_IMAGE=/ppc/ppc64/vmlinuz ro Max number of cores passed to firmware: 1024 (NR_CPUS = 1024) Calling ibm,client-architecture-support... done memory layout at init: memory_limit : 0000000000000000 (16 MB aligned) alloc_bottom : 00000000070b0000 alloc_top : 0000000030000000 alloc_top_hi : 0000000080000000 rmo_top : 0000000030000000 ram_top : 0000000080000000 instantiating rtas at 0x000000002fff0000... done prom_hold_cpus: skipped copying OF device tree... Building dt strings... Building dt structure... Device tree strings 0x00000000070c0000 -> 0x00000000070c0ad5 Device tree struct 0x00000000070d0000 -> 0x00000000070e0000 Quiescing Open Firmware ... Booting Linux via __start() @ 0x0000000002000000 ... kernel 4.20.3-200.fc29 was introduced in Atomic Host compose Fedora-29-updates-20190122.0 . Atomic Host iso from compose Fedora-29-updates-20190121.0(https://kojipkgs.fedoraproject.org/compose/updates/Fedora-29-updates-20190121.0/compose/AtomicHost/ppc64le/iso/Fedora-AtomicHost-ostree-ppc64le-29-20190121.0.iso) contains kernel 4.19.15-300.fc29 which boots perfectly fine. By looking at rpm-ostree diff for commit from Fedora-29-updates-20190121.0 and Fedora-29-updates-20190122.0 , it also shows that only kernel was updated: $ rpm-ostree db diff cdcbea2ccac7804770be806befd30895457de080d1525ee6050a5bebdfeefeb7 dfa61b4b9c11713dca18df38df705e9f742e78e3f561d10faf1c9265fea331e3 ostree diff commit old: cdcbea2ccac7804770be806befd30895457de080d1525ee6050a5bebdfeefeb7 ostree diff commit new: dfa61b4b9c11713dca18df38df705e9f742e78e3f561d10faf1c9265fea331e3 Upgraded: kernel 4.19.15-300.fc29 -> 4.20.3-200.fc29 kernel-core 4.19.15-300.fc29 -> 4.20.3-200.fc29 kernel-modules 4.19.15-300.fc29 -> 4.20.3-200.fc29
kernel-4.20.6-200.fc29.ppc64le boots fine in my VM, Power9 HW, qemu 3.1.0 on host
(In reply to Dan Horák from comment #2) > kernel-4.20.6-200.fc29.ppc64le boots fine in my VM, Power9 HW, qemu 3.1.0 on > host Tried booting image with kernel-4.20.6-200.fc29.ppc64le (https://kojipkgs.fedoraproject.org/compose/updates/Fedora-29-updates-20190205.0/compose/AtomicHost/ppc64le/images/Fedora-AtomicHost-29-20190205.0.ppc64le.qcow2) , same hang result for me. I am using Power8 guest. Dan, can you try on Power8 and let us know the result?
Christoph, Where are you seeing the issue, on Power 8 or Power9?
My machine is a Power 8 (S822)
FYI: I have an openQA test of f29, running on a f29 P8 machine, and do not have net install problem of last delivered kernel, the guest properly rebooted with kernel kernel-4.20.6-200.fc29.ppc64le
(In reply to Michel Normand from comment #6) > FYI: I have an openQA test of f29, running on a f29 P8 machine, and do not > have net install problem of last delivered kernel, the guest properly > rebooted with kernel kernel-4.20.6-200.fc29.ppc64le Thanks Michel for the information. I also observed that I have a successfully installed and booted F28/F29 ppc64le vm (P8 with kernel 4.20.6-100.fc{28,29}.ppc64le ) with netboot. But on those vm, when I using virt-install to boot image for example qcow2 image it hangs. Observed same with latest FAH qcow2 ( https://kojipkgs.fedoraproject.org/compose/updates/Fedora-29-updates-20190205.0/compose/AtomicHost/ppc64le/images/Fedora-AtomicHost-29-20190205.0.ppc64le.qcow2) and F29-cloud-base (https://kojipkgs.fedoraproject.org/compose/cloud/Fedora-Cloud-29-20190205.0/compose/Cloud/ppc64le/images/Fedora-Cloud-Base-29-20190205.0.ppc64le.qcow2) command I used was something like: $ virt-install --name VM_NAME --ram 1024 --vcpus 1 --disk /var/lib/libvirt/images/IMAGE.ppc64le.qcow2 --os-type linux --os-variant fedora26 --network bridge=virbr0 --nographics --cdrom /var/lib/libvirt/images/init.iso This works fine on same host with older qcow2 image which has kernel 4.19.* Also, on same host I booted a working Atomic Host qcow2 image (https://kojipkgs.fedoraproject.org/compose/updates/Fedora-29-updates-20190119.0/compose/AtomicHost/ppc64le/images/Fedora-AtomicHost-29-20190119.0.ppc64le.qcow2) which boots perfectly fine. Later on updated just kernel packages to 4.20.6-100.fc29.ppc64le and after reboot vm hangs at same place. Can you try using any of these scenarios mentioned above using virt-install and let us know?
Just a note, on Atomic Host system you can override kernel (base package) using: $ sudo rpm-ostree override replace pkg1 pkg2 ...
For the record I have hit this(or the issue Sinny is describing) with nested virtualization on P8 machine. It hangs after kexecing from the fw with one CPU core used by the qemu at 100%. Phy Host RHEL 7 kernel-3.10.0-957.1.3.el7.ppc64le qemu-2.12.0-18.el7_6.1.ppc64le Virt Host Fedora 29 kernel-4.20.6-200.fc29.ppc64le qemu-2.12.0-3.fc39 (run in privileged container) Virt Guest Fedora 29 kernel-4.20.6-200.fc29.ppc64le ... Booting from memory... OF stdout device is: /vdevice/vty@71000000 Preparing to boot Linux version 4.20.6-200.fc29.ppc64le (mockbuild.fedoraproject.org) (gcc version 8.2.1 20181215 (Red Hat 8.2.1-6) (GCC)) #1 SMP Thu Jan 31 15:31:01 UTC 2019 Detected machine type: 0000000000000101 command line: panic=1 console=hvc0 console=ttyS0 edd=off udevtimeout=6000 udev.event-timeout=6000 no_timer_check printk.time=1 cgroup_disable=memory usbcore.nousb cryptomgr.notests tsc=reliable 8250.nr_uarts=1 root=/dev/sdb selinux=0 guestfs_verbose=1 TERM=xterm Max number of cores passed to firmware: 1024 (NR_CPUS = 1024) Calling ibm,client-architecture-support... done memory layout at init: memory_limit : 0000000000000000 (16 MB aligned) alloc_bottom : 0000000001cc0000 alloc_top : 0000000030000000 alloc_top_hi : 0000000040000000 rmo_top : 0000000030000000 ram_top : 0000000040000000 instantiating rtas at 0x000000002fff0000... done prom_hold_cpus: skipped copying OF device tree... Building dt strings... Building dt structure... Device tree strings 0x0000000001ed0000 -> 0x0000000001ed0aa6 Device tree struct 0x0000000001ee0000 -> 0x0000000001ef0000 Quiescing Open Firmware ... Booting Linux via __start() @ 0x0000000000400000 ...
My environment is PowerVM (so no nested virtualization) on a IBM 8247-22L. The LPAR is running with 2 cores and 8GB RAM.
It's possible that the problem is related to disk or multipathd: Here's what I see, when I'm booting with kernel 4.20: [...] [ 2.046685] sr 0:0:1:0: Attached scsi generic sg0 type 5 [ 2.046981] sd 0:0:2:0: Attached scsi generic sg1 type 0 [ 2.047598] sd 0:0:2:0: [sda] 62914560 512-byte logical blocks: (32.2 GB/30.0 GiB) [ 2.047669] sd 0:0:2:0: [sda] Write Protect is off [ 2.047753] sd 0:0:2:0: [sda] Cache data unavailable [ 2.047760] sd 0:0:2:0: [sda] Assuming drive cache: write through [ 2.049168] sda: sda1 sda2 sda3 [ 2.050343] sd 0:0:2:0: [sda] Attached SCSI disk [ 2.072695] sd 1:0:1:0: Attached scsi generic sg2 type 0 [ 2.072945] sd 1:0:1:0: [sdb] 62914560 512-byte logical blocks: (32.2 GB/30.0 GiB) [ 2.073017] sd 1:0:1:0: [sdb] Write Protect is off [ 2.073087] sd 1:0:1:0: [sdb] Cache data unavailable [ 2.073094] sd 1:0:1:0: [sdb] Assuming drive cache: write through [ 2.074453] sdb: sdb1 sdb2 sdb3 [ 2.075586] sd 1:0:1:0: [sdb] Attached SCSI disk [ OK ] Started udev Wait for Complete Device Initialization. Starting Device-Mapper Multipath Device Controller... [ OK ] Started Device-Mapper Multipath Device Controller. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 2.194757] device-mapper: multipath service-time: version 0.3.0 loaded [ 2.195095] device-mapper: table: table load rejected: not all devices are blk-mq request-stackable [ 2.195108] device-mapper: table: unable to determine table type [ 2.198454] device-mapper: table: table load rejected: not all devices are blk-mq request-stackable [ 2.198471] device-mapper: table: unable to determine table type [ 8.988724] random: crng init done [ 8.988741] random: 7 urandom warning(s) missed due to ratelimiting [ 8.988762] random: crng done (trusting CPU's manufacturer) ---> stops here, later I'm falling to an emergency shell: dracut:/# multipathd -k multipathd> list paths hcil dev dev_t pri dm_st chk_st dev_st next_check 0:0:2:0 sda 8:0 1 undef undef unknown orphan 1:0:1:0 sdb 8:16 1 undef undef unknown orphan (no fdisk or lsblk in this shell) ***** This shows up when (successfully) booting with kernel 4.18: [...] [ 2.295306] sr 0:0:1:0: Attached scsi generic sg0 type 5 [ 2.295555] sd 0:0:2:0: Attached scsi generic sg1 type 0 [ 2.296082] sd 0:0:2:0: [sda] 62914560 512-byte logical blocks: (32.2 GB/30.0 GiB) [ 2.296160] sd 0:0:2:0: [sda] Write Protect is off [ 2.296230] sd 0:0:2:0: [sda] Cache data unavailable [ 2.296236] sd 0:0:2:0: [sda] Assuming drive cache: write through [ 2.297716] sda: sda1 sda2 sda3 [ 2.298969] sd 0:0:2:0: [sda] Attached SCSI disk [ 2.303265] ibmveth 30000003 env3: renamed from eth1 [ 2.339388] sd 1:0:1:0: Attached scsi generic sg2 type 0 [ 2.339917] sd 1:0:1:0: [sdb] 62914560 512-byte logical blocks: (32.2 GB/30.0 GiB) [ 2.339987] sd 1:0:1:0: [sdb] Write Protect is off [ 2.340054] sd 1:0:1:0: [sdb] Cache data unavailable [ 2.340060] sd 1:0:1:0: [sdb] Assuming drive cache: write through [ 2.341450] sdb: sdb1 sdb2 sdb3 [ 2.342595] sd 1:0:1:0: [sdb] Attached SCSI disk [ OK ] Started udev Wait for Complete Device Initialization. Starting Device-Mapper Multipath Device Controller... [ OK ] Started Device-Mapper Multipath Device Controller. [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Create Volatile Files and Directories. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 2.452935] device-mapper: multipath service-time: version 0.3.0 loaded [ OK ] Found device /dev/mapper/fedora-root. [ OK ] Reached target Initrd Root Device. [ OK ] Started dracut initqueue hook. Starting File System Check on /dev/mapper/fedora-root... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ OK ] Started File System Check on /dev/mapper/fedora-root. Mounting /sysroot... [ 3.524325] SGI XFS with ACLs, security attributes, scrub, no debug enabled [ 3.533152] random: crng init done [ 3.533176] random: 7 urandom warning(s) missed due to ratelimiting [ 3.533210] random: crng done (trusting CPU's manufacturer) [ 3.534153] XFS (dm-4): Mounting V5 Filesystem [ 3.573470] XFS (dm-4): Ending clean mount [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Started Cleaning Up and Shutting Down Daemons. Starting Setup Virtual Console... Starting Plymouth switch root service... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Sockets. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local File Systems (Pre). Stopping Device-Mapper Multipath Device Controller... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Timers. [ OK ] Stopped target Slices. [ OK ] Stopped Device-Mapper Multipath Device Controller. [ OK ] Started Plymouth switch root service. [ OK ] Stopped udev Wait for Complete Device Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Started Setup Virtual Console. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required staâ¦vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped dracut ask for additional cmdline parameters. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 4.330963] systemd-journald[252]: Received SIGTERM from PID 1 (systemd). [ 4.361458] systemd: 13 output lines suppressed due to ratelimiting [...]
multipath output on working system: [root@fed29 ~]# multipathd -k multipathd> list paths hcil dev dev_t pri dm_st chk_st dev_st next_check 0:0:2:0 sda 8:0 1 active ready running XXXXXXX... 15/20 1:0:1:0 sdb 8:16 1 active ready running XXXXXXX... 15/20 multipathd>
For info: the PowerVM setup can be done with one or more virtual IO servers (VIOS). If using multiple VIOS these servers provide the same LUN/disk via multiple paths, in my case two VIOS/paths.
Isn't anything useful in /var/log/libvirt/qemu/<vm name>?
Created attachment 1528049 [details] qemu log from FAH 20190205 /var/log/libvirt/qemu/fah log fron one of the virt-insatll attempt which hangs
From what has been posted, this seems like two separate issues. One that Christoph describes on the LPAR(later in the boot) and other that me and Sinny are observing in VMs(kexecing kernel, possibly also qemu/kvm bug). IMO only link is that something has changed within recent kernel. For the record nothing is in the /var/log/libvirt/qemu/ directory for me and when investigating the qemu process with gdb it seems that it is stuck busy polling/waiting for something. #0 0x00007fffb5194364 in __GI_ppoll (fds=0x202, nfds=6, timeout=<optimized out>, sigmask=0x0) at ../sysdeps/unix/sysv/linux/ppoll.c:39 #1 0x00000001235d982c in ppoll (__ss=0x0, __timeout=0x0, __nfds=<optimized out>, __fds=<optimized out>) at /usr/include/bits/poll2.h:77 #2 qemu_poll_ns (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at /usr/src/debug/qemu-2.12.0-3.fc29.ppc64le/util/qemu-timer.c:322 #3 0x00000001235daf94 in os_host_main_loop_wait (timeout=<optimized out>) at /usr/src/debug/qemu-2.12.0-3.fc29.ppc64le/util/main-loop.c:258 #4 main_loop_wait (nonblocking=<optimized out>) at /usr/src/debug/qemu-2.12.0-3.fc29.ppc64le/util/main-loop.c:522 #5 0x0000000123019a40 in main_loop () at /usr/src/debug/qemu-2.12.0-3.fc29.ppc64le/vl.c:1943 #6 main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /usr/src/debug/qemu-2.12.0-3.fc29.ppc64le/vl.c:4734
I have the same problem as Christoph on a baremetal P8 machine with multipath (8247-22L). I have a f29 I updated and got kernel-4.20.6-200.fc29.ppc64le. on the console: [ OK ] Started Create Volatile Files and Directories. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 72.504522] device-mapper: multipath service-time: version 0.3.0 loaded [ 72.504764] device-mapper: table: table load rejected: not all devices are blk-mq request-stackable [ 72.504785] device-mapper: table: unable to determine table type [ 72.507232] device-mapper: table: table load rejected: not all devices are blk-mq request-stackable [ 72.507277] device-mapper: table: unable to determine table type [ 72.511441] device-mapper: table: table load rejected: not all devices are blk-mq request-stackable [ 72.511507] device-mapper: table: unable to determine table type [ 72.515748] device-mapper: table: table load rejected: not all devices are blk-mq request-stackable [ 72.515801] device-mapper: table: unable to determine table type [ 149.761624] dracut-initqueue[1348]: Warning: dracut-initqueue timeout - starting timeout scripts [ 150.341936] dracut-initqueue[1348]: Warning: dracut-initqueue timeout - starting timeout scripts [ 150.921886] dracut-initqueue[1348]: Warning: dracut-initqueue timeout - starting timeout scripts ... [ 218.221867] dracut-initqueue[1348]: Warning: Could not boot. dracut:/tmp# multipathd -k multipathd> list paths hcil dev dev_t pri dm_st chk_st dev_st next_check 0:2:0:0 sda 8:0 50 undef undef unknown orphan 0:2:1:0 sdb 8:16 50 undef undef unknown orphan 1:2:0:0 sdc 8:32 10 undef undef unknown orphan 1:2:1:0 sdd 8:48 10 undef undef unknown orphan This machine f29 boot correctly with 4.19.6-300.fc29.ppc64le kernel.
Right, I'm changing the title of this bug to mention the multipath storage. And we will open another bug to track the nested virt issues.
For the record I have opened bug for the LPAR nested virt. BZ#1676475
(In reply to Jakub Čajka from comment #19) > For the record I have opened bug for the LPAR nested virt. BZ#1676475 Thanks Jakub!
removing the need-info as as previous comment#7 tracked by new bug#1676475 I just commented in.
I am seeing a similar issue after upgrading from Fedora27 that was using 4.18.19-100.fc27.ppc64 to Fedora28 using 4.20.7-100.fc28.ppc64 I found this post https://lore.kernel.org/lkml/20181105135157.GA11485@redhat.com/ and tried adding scsi_mod.use_blk_mq=Y to the kernel commandline. Doing so enabled my ppc64 system to boot and find the multipath devices with the 4.20 kernel.
Matt, this actually also worked with my Fedora 29 installation. Thanks for the hint!
Is there any reason why this would be ppc64le specific? Sounds like a generic problem in the block layer to me.
In fact this seems to be the same issue as in bz#1669235
*** This bug has been marked as a duplicate of bug 1669235 ***