Description of problem: [ 80.584148] systemd-udevd[599]: seq 1294 '/devices/pci0000:00/0000:00:02.0' is taking a long time [ 80.586967] systemd-udevd[599]: seq 1314 '/devices/pci0000:00/0000:00:07.0' is taking a long time [ 80.589218] systemd-udevd[599]: seq 1313 '/devices/pci0000:00/0000:00:06.0' is taking a long time [ 80.591405] systemd-udevd[599]: seq 1312 '/devices/pci0000:00/0000:00:05.0' is taking a long time [ 80.593637] systemd-udevd[599]: seq 1311 '/devices/pci0000:00/0000:00:04.0' is taking a long time [ 135.583224] systemd-journald[166]: Sent WATCHDOG=1 notification. [ 140.887325] systemd[1]: systemd-udev-settle.service: Main process exited, code=exited, status=1/FAILURE [ 140.890072] systemd[1]: systemd-udev-settle.service: Failed with result 'exit-code'. [ 140.893607] systemd[1]: Failed to start udev Wait for Complete Device Initialization. [FAILED] Failed to start udev Wait for Complete Device Initialization. [...] [ 200.593723] systemd-udevd[599]: seq 1294 '/devices/pci0000:00/0000:00:02.0' killed [ 200.596953] systemd-udevd[599]: seq 1314 '/devices/pci0000:00/0000:00:07.0' killed [ 200.599980] systemd-udevd[599]: seq 1311 '/devices/pci0000:00/0000:00:04.0' killed [ 200.602119] systemd-udevd[599]: seq 1313 '/devices/pci0000:00/0000:00:06.0' killed [ 200.605124] systemd-udevd[599]: seq 1312 '/devices/pci0000:00/0000:00:05.0' killed [ 200.613744] systemd-udevd[599]: worker [695] terminated by signal 9 (KILL) [ 200.616156] systemd-udevd[599]: worker [698] terminated by signal 9 (KILL) [ 200.618147] systemd-udevd[599]: worker [701] terminated by signal 9 (KILL) [ 200.620114] systemd-udevd[599]: worker [704] terminated by signal 9 (KILL) [ 200.622079] systemd-udevd[599]: worker [707] terminated by signal 9 (KILL) [ 200.623967] systemd-udevd[599]: worker [707] failed while handling '/devices/pci0000:00/0000:00:02.0' [ 200.647780] virtio_blk: probe of virtio2 failed with error -2 [ 200.662916] virtio_console virtio1: Error -2 initializing vqs [ 200.673094] virtio_console: probe of virtio1 failed with error -2 [ 200.689628] systemd-udevd[702]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. [ 200.718507] 8021q: 802.1Q VLAN Support v1.8 [ 228.193008] watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ip:847] [ 228.194492] Modules linked in: 8021q garp mrp stp llc virtio_console virtio_net virtio_blk serio_raw virtio_pci ata_generic pata_acpi qemu_fw_cfg sunrpc mcryptd sha256_ssse3 dm_crypt dm_round_robin dm_multipath linear raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 iscsi_ibft iscsi_boot_sysfs floppy iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi squashfs zstd_decompress xxhash cramfs edd virtio_rng virtio_ring virtio [ 228.203644] CPU: 0 PID: 847 Comm: ip Not tainted 4.16.3-301.fc28.x86_64 #1 [ 228.205113] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20180531_142017-buildhw-08.phx2.fedoraproject.org-1.fc28 04/01/2014 [ 228.207809] RIP: 0010:virtqueue_get_buf+0x0/0x10 [virtio_ring] [ 228.209081] RSP: 0018:ffffb2494050f4c8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff12 [ 228.210695] RAX: 0000000000000000 RBX: ffff911d7f9498c0 RCX: 0000000000000001 [ 228.212218] RDX: 0000000000000000 RSI: ffffb2494050f4d4 RDI: ffff911d7d741800 [ 228.213762] RBP: 0000000000000002 R08: ffffb2494050f4d4 R09: ffffffffc027ec09 [ 228.215290] R10: 000000003f949000 R11: 0000000000000089 R12: ffffb2494050f4f8 [ 228.216825] R13: 0000000000000000 R14: 0000000000000000 R15: ffff911d7f949000 [ 228.218369] FS: 00007f5164f2f800(0000) GS:ffff911d7bc00000(0000) knlGS:0000000000000000 [ 228.220107] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 228.221356] CR2: 00007f5164253ff0 CR3: 000000003d542000 CR4: 00000000000006f0 [ 228.222892] Call Trace: [ 228.223443] virtnet_send_command+0xfb/0x130 [virtio_net] [ 228.224604] virtnet_set_rx_mode+0xac/0x310 [virtio_net] [ 228.225754] ? __hw_addr_create_ex+0x85/0xc0 [ 228.226681] __dev_mc_add+0x67/0x70 [ 228.227478] igmp_group_added+0x153/0x160 [ 228.228379] ? kmem_cache_alloc_trace+0x111/0x1c0 [ 228.229401] ? igmpv3_del_delrec+0x6e/0x120 [ 228.230307] ip_mc_inc_group+0x114/0x1e0 [ 228.231158] ip_mc_up+0x33/0x90 [ 228.231849] inetdev_event+0x1a0/0x500 [ 228.232677] ? notifier_call_chain+0x47/0x70 [ 228.233611] ? inetdev_init+0x170/0x170 [ 228.234443] notifier_call_chain+0x47/0x70 [ 228.235342] __dev_notify_flags+0x5b/0xf0 [ 228.236223] dev_change_flags+0x48/0x60 [ 228.237063] do_setlink+0x660/0xdf0 [ 228.237842] ? __nla_reserve+0x38/0x50 [ 228.238665] ? __nla_put+0xc/0x20 [ 228.239389] ? nla_put+0x32/0x40 [ 228.240096] ? inet6_fill_ifla6_attrs+0x466/0x480 [ 228.241109] ? __kmalloc_node_track_caller+0x1ec/0x2a0 [ 228.242218] ? rtnl_newlink+0x816/0x990 [ 228.243048] ? netlink_attachskb+0x1b0/0x1e0 [ 228.243971] ? __netlink_sendskb+0x51/0x70 [ 228.244861] ? kmem_cache_alloc+0xc0/0x1b0 [ 228.245751] ? rtnetlink_rcv_msg+0x200/0x2f0 [ 228.246671] ? __wake_up_common_lock+0x89/0xc0 [ 228.247642] ? _cond_resched+0x15/0x30 [ 228.248452] ? rtnl_calcit.isra.30+0x100/0x100 [ 228.249406] ? netlink_rcv_skb+0x4c/0x120 [ 228.250277] ? netlink_unicast+0x19e/0x260 [ 228.251162] ? netlink_sendmsg+0x1fd/0x3c0 [ 228.252040] ? sock_sendmsg+0x36/0x40 [ 228.252840] ? ___sys_sendmsg+0x295/0x2f0 [ 228.253714] ? pick_next_task_fair+0x2f7/0x5a0 [ 228.254663] ? dev_get_by_name_rcu+0x73/0x90 [ 228.255583] ? skb_dequeue+0x52/0x60 [ 228.256360] ? __inode_wait_for_writeback+0x7f/0xf0 [ 228.257419] ? bit_waitqueue+0x30/0x30 [ 228.258221] ? fsnotify_grab_connector+0x3c/0x60 [ 228.259214] ? __sys_sendmsg+0x51/0x90 [ 228.260029] ? do_syscall_64+0x74/0x180 [ 228.260867] ? entry_SYSCALL_64_after_hwframe+0x3d/0xa2 Version-Release number of selected component (if applicable): f28 $ rpm -q kernel qemu libvirt-libs kernel-4.8.13-100.fc23.x86_64 kernel-4.18.9-200.fc28.x86_64 qemu-2.11.2-2.fc28.x86_64 libvirt-libs-4.1.0-5.fc28.x86_64 How reproducible: 90% Steps to Reproduce: sudo virt-install --connect qemu:///system \ --name=swanfedora28base \ --os-variant fedora28 \ --vcpus=1 \ --memory 1024 \ --nographics \ --disk size=8,cache=writeback,path=/home/build/pool/swanfedora28base.qcow2 \ --cpu host-passthrough \ --network=network:swandefault,model=virtio \ --rng type=random,device=/dev/random \ --location=/home/build/pool/Fedora-Server-dvd-x86_64-28-1.1.iso \ --initrd-inject=testing/libvirt/fedora28.ks \ --extra-args="swanname=swanfedora28base debug systemd.journald.forward_to_console=1 inst.text plymouth.enable=0 ks=file:/fedora28.ks console=ttyS0 net.ifnames=0 biosdevname=0" \ --noreboot Actual results: Expected results: Additional info: Machine was upgraded from F22. Have tried trimming the line to no avail. Have tried re-installing KVM stuff to no avail. Works fine on another machine (with a fresh fedora install).
I have same issue with kernel 4.18.11 qemu-2.7.1 eudev-3.2.6, and my system is LFS. "virtio_blk: probe of virtio1 failed with error -2" But another PC with same system but different hardware has no such issue.
Adding "--network none" seems to stop the kernel explosion (not surprising given the explosion looks like a timeout in the network code). However, things still eventually fail. Presumably because those pci devices were never properly initialized. Trying F27 guest gets similar results: [ 13.600501] dracut-initqueue[581]: Got message type=method_return sender=n/a destination=n/a object=n/a interface=n/a member=n/a cookie=329 reply_cookie=1 error=n/a [ 13.776116] dracut-initqueue[581]: calling: settle [ 73.320191] systemd-udevd[491]: seq 1364 '/devices/pci0000:00/0000:00:06.0/virtio3' is taking a long time [ 73.322786] systemd-udevd[491]: seq 1371 '/devices/pci0000:00/0000:00:04.0/virtio1/block/vda' is taking a long time [ 133.985318] systemd-journald[164]: Sent WATCHDOG=1 notification. [ 134.071825] dracut-initqueue[581]: calling: settle For what its worth, the host cpu is: vendor_id : GenuineIntel cpu family : 6 model : 15 model name : Intel(R) Core(TM)2 Quad CPU Q6700 @ 2.66GHz stepping : 11 microcode : 0xba cpu MHz : 2659.910 cache size : 4096 KB physical id : 0 siblings : 4 core id : 3 cpu cores : 4 apicid : 3 initial apicid : 3 fpu : yes fpu_exception : yes cpuid level : 10 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm pti tpr_shadow vnmi flexpriority dtherm bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf bogomips : 5319.82 clflush size : 64 cache_alignment : 64 address sizes : 36 bits physical, 48 bits virtual power management:
I guess unsurprisingly, '--virt-type qemu' gets around the problem :-/
Actually, --virt-type qemu doesn't really help. It gets further but then times out (oh, so slow) in (yes, inst.text was passed in): Anaconda 28.22.10-1.fc28 for Fedora 28 started. * installation log files are stored in /tmp during the installation * shell is available on TTY2 * if the graphical installation interface fails to start, try again with the inst.text bootoption to start text installation * when reporting a bug add logs from /tmp as separate text/plain attachments 16:58:45 Anaconda DBus modules failed to start on time. 16:58:45 Anaconda DBus modules failed to start on time. Pane is dead
This is a bug caused by upstream kernel commit kvm: x86: Set highest physical address bits in non-present/reserved SPTEs It's fixed in 4.19, and only affects certain older hardware.
https://patchwork.kernel.org/patch/10565905/
kernel 4.18.14 fixed this issue on my mathine. But I still have to do "modprobe virtio_pci" manually.
kernel 4.18.14 works for me
This message is a reminder that Fedora 28 is nearing its end of life. On 2019-May-28 Fedora will stop maintaining and issuing updates for Fedora 28. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as EOL if it remains open with a Fedora 'version' of '28'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version. Thank you for reporting this issue and we are sorry that we were not able to fix it before Fedora 28 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora, you are encouraged change the 'version' to a later Fedora version prior this bug is closed as described in the policy above. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete.
Sounds like this was fixed in the kernel