Created attachment 910470 [details] console.log Description of problem: Upon discovery of disks (where the xfs / partition is on LVM on top of RAID1 where one of the RAID members is marked as failed) in the anaconda rescue mode, the kernel panic occurs. Version-Release number of selected component (if applicable): kernel-3.16.0-0.rc1.git2.1.fc21.x86_64 anaconda-21.42-1.fc21.x86_64 How reproducible: always Steps to Reproduce: 1. Prepare VM with 2 VirtIO disks, instruct anaconda to put / fs on RAID1 2. In the installed system run mdadm -f /dev/md0 /dev/vdb1 3. Download rawhide boot.iso (tested on boot.iso from 20140619) 4. Boot the boot.iso, append inst.rescue on the kernel cmdline 5. On the anaconda rescue screen, select continue Actual results: kernel panic occurs Expected results: no kernel panic Additional info: please see the console.log attachment tested in F20
I'm seeing something similar with the current rawhide kernel, except no RAID1, just plain virtio: [ 153.089241] ------------[ cut here ]------------ [ 153.089269] kernel BUG at drivers/block/virtio_blk.c:172! [ 153.089292] invalid opcode: 0000 [#1] SMP [ 153.089314] Modules linked in: xfs libcrc32c fcoe libfcoe libfc scsi_transport_fc cfg80211 rfkill serio_raw virtio_balloon i2c_piix4 parport_pc parport pvpanic loop cirrus drm_kms_helper virtio_net virtio_blk ttm drm virtio_pci virtio_ring virtio ata_generic pata_acpi sunrpc dm_crypt dm_round_robin linear raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor xor async_tx raid6_pq raid1 raid0 iscsi_ibft iscsi_boot_sysfs floppy iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi squashfs cramfs edd dm_multipath [ 153.089599] CPU: 0 PID: 1401 Comm: mount Not tainted 3.18.0-0.rc2.git3.1.fc22.x86_64 #1 [ 153.089638] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 153.089664] task: ffff880073199a40 ti: ffff8800732fc000 task.ti: ffff8800732fc000 [ 153.089695] RIP: 0010:[<ffffffffa02067c0>] [<ffffffffa02067c0>] virtio_queue_rq+0x290/0x2a0 [virtio_blk] [ 153.089740] RSP: 0018:ffff8800732ff788 EFLAGS: 00010202 [ 153.089763] RAX: 0000000000000082 RBX: ffff8800b046ae00 RCX: dead000000200200 [ 153.089795] RDX: 0000000000000000 RSI: ffff8800b046ae00 RDI: ffff88007d9eb000 [ 153.089825] RBP: ffff8800732ff7c8 R08: ffff8800b046ae00 R09: 0000000000000001 [ 153.089855] R10: 0000000000000001 R11: 0000000000000004 R12: ffff8800732ff7f0 [ 153.089885] R13: ffff88007d9eb000 R14: ffff88007d89d800 R15: 0000000000000000 [ 153.090009] FS: 00007f9f6cd69840(0000) GS:ffff8800bca00000(0000) knlGS:0000000000000000 [ 153.090009] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 153.090009] CR2: 00007f9f6d8bf5e8 CR3: 000000006b10c000 CR4: 00000000000006f0 [ 153.090009] Stack: [ 153.090009] ffff880000000000 ffff8800b046af70 0000000000000000 ffff88007d9eb000 [ 153.090009] ffff8800732ff7f0 0000000000000000 ffff8800b046ae00 ffff88007d9eb048 [ 153.090009] ffff8800732ff838 ffffffff813f5958 ffff88007ef2e500 0000000000000001 [ 153.090009] Call Trace: [ 153.090009] [<ffffffff813f5958>] __blk_mq_run_hw_queue+0x1c8/0x330 [ 153.090009] [<ffffffff813f6480>] blk_mq_run_hw_queue+0x70/0x90 [ 153.090009] [<ffffffff813f73cd>] blk_mq_insert_requests+0xfd/0x2b0 [ 153.090009] [<ffffffff813f8384>] blk_mq_flush_plug_list+0x144/0x170 [ 153.090009] [<ffffffff813ebba1>] blk_flush_plug_list+0xc1/0x260 [ 153.090009] [<ffffffff813ec308>] blk_finish_plug+0x18/0x60 [ 153.090009] [<ffffffffa03a82f8>] _xfs_buf_ioapply+0x338/0x5f0 [xfs] [ 153.090009] [<ffffffffa03acd54>] ? xfs_bwrite+0x24/0x60 [xfs] [ 153.090009] [<ffffffffa03abe21>] xfs_buf_submit_wait+0x91/0x840 [xfs] [ 153.090009] [<ffffffffa03acd54>] xfs_bwrite+0x24/0x60 [xfs] [ 153.090009] [<ffffffffa03df777>] xlog_bwrite+0x87/0x110 [xfs] [ 153.090009] [<ffffffffa03e04cb>] xlog_write_log_records+0x1ab/0x240 [xfs] [ 153.090009] [<ffffffffa03e0617>] xlog_clear_stale_blocks+0xb7/0x1d0 [xfs] [ 153.090009] [<ffffffffa03dfec3>] ? xlog_bread+0x43/0x60 [xfs] [ 153.090009] [<ffffffffa03e21fb>] xlog_find_tail+0x32b/0x410 [xfs] [ 153.090009] [<ffffffffa03e712d>] xlog_recover+0x2d/0x120 [xfs] [ 153.090009] [<ffffffffa03d5c3d>] xfs_log_mount+0xdd/0x2c0 [xfs] [ 153.090009] [<ffffffffa03cb744>] xfs_mountfs+0x514/0x9c0 [xfs] [ 153.090009] [<ffffffffa03ccc8d>] ? xfs_mru_cache_create+0x18d/0x1f0 [xfs] [ 153.090009] [<ffffffffa03ceed0>] xfs_fs_fill_super+0x330/0x3b0 [xfs] [ 153.090009] [<ffffffff81276c7c>] mount_bdev+0x1bc/0x1f0 [ 153.090009] [<ffffffffa03ceba0>] ? xfs_parseargs+0xbe0/0xbe0 [xfs] [ 153.090009] [<ffffffffa03ccfd5>] xfs_fs_mount+0x15/0x20 [xfs] [ 153.090009] [<ffffffff81277628>] mount_fs+0x38/0x1c0 [ 153.090009] [<ffffffff8120c3e5>] ? __alloc_percpu+0x15/0x20 [ 153.090009] [<ffffffff8129a0c8>] vfs_kern_mount+0x68/0x160 [ 153.090009] [<ffffffff8129d53c>] do_mount+0x22c/0xc20 [ 153.090009] [<ffffffff812170fe>] ? might_fault+0x5e/0xc0 [ 153.090009] [<ffffffff812066eb>] ? memdup_user+0x4b/0x90 [ 153.090009] [<ffffffff8129e25e>] SyS_mount+0x9e/0x100 [ 153.090009] [<ffffffff81867429>] system_call_fastpath+0x12/0x17 [ 153.090009] Code: 00 00 c7 86 78 01 00 00 02 00 00 00 48 c7 86 80 01 00 00 00 00 00 00 89 86 7c 01 00 00 e9 02 fe ff ff 66 0f 1f 84 00 00 00 00 00 <0f> 0b 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 [ 153.090009] RIP [<ffffffffa02067c0>] virtio_queue_rq+0x290/0x2a0 [virtio_blk] [ 153.090009] RSP <ffff8800732ff788> [ 153.132413] ---[ end trace 81d3640f347f26b6 ]---
Looks like this is still outstanding: http://www.spinics.net/lists/linux-virtualization/msg23903.html
This bug appears to have been reported against 'rawhide' during the Fedora 22 development cycle. Changing version to '22'. More information and reason for this action is here: https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora22
I wonder what/why the discrepancy virtio_blk.c:167 vs. virtio_blk.c:172. I don't have access to the source file at the moment, but I have seen this bug numerous times on a RHEL7.2 guest VM on RHEL7.2 Host KVM, virtio_blk, luks disk encryption, LVM, XFS sysroot, ext4 /boot. It occurred a while ago when I installed a new kernel; however, rebooting into the previously-installed kernel did NOT exhibit this behavior! I removed the newer kernel and continued using it without problems until "now", where the problem is exhibited even on the previously error-free kernel. Console output follows: [ 11.294852] ------------[ cut here ]------------ [ 11.295007] kernel BUG at drivers/block/virtio_blk.c:172! [ 11.295007] invalid opcode: 0000 [#1] SMP [ 11.295007] Modules linked in: xfs libcrc32c dm_crypt drbg ata_generic crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel virtio_blk pata_acpi virtio_console virtio_net aesni_intel glue_helper qxl lrw gf128mul ablk_helper cryptd drm_kms_helper ttm ata_piix serio_raw 8139too libata drm 8139cp mii virtio_pci virtio_ring virtio i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod [ 11.295007] CPU: 0 PID: 562 Comm: dmcrypt_write Not tainted 3.10.0-306.0.1.el7.x86_64 #1 [ 11.295007] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 [ 11.295007] task: ffff8800361cdc00 ti: ffff880036230000 task.ti: ffff880036230000 [ 11.295007] RIP: 0010:[<ffffffffa004c825>] [<ffffffffa004c825>] virtio_queue_rq+0x285/0x290 [virtio_blk] [ 11.295007] RSP: 0018:ffff880036233bf8 EFLAGS: 00010202 [ 11.295007] RAX: 0000000000000082 RBX: ffff880036a39fc0 RCX: dead000000200200 [ 11.295007] RDX: ffff880036233c60 RSI: ffff880036233c80 RDI: ffff88003692e800 [ 11.295007] RBP: ffff880036233c38 R08: ffff880036a39fc0 R09: ffff880036233c60 [ 11.295007] R10: ffff880036d3ea00 R11: 0000000035ede000 R12: ffff88003692e800 [ 11.295007] R13: 0000000000000000 R14: ffff8800369ee0c0 R15: 0000000000000000 [ 11.295007] FS: 0000000000000000(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000 [ 11.295007] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 11.295007] CR2: 00007f3e12775740 CR3: 0000000036784000 CR4: 00000000000407f0 [ 11.295007] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 11.295007] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [ 11.295007] Stack: [ 11.295007] ffff880036233c80 ffff8800369ed210 ffff880036a3a140 ffff88003692e800 [ 11.295007] ffff880036233c60 0000000000000000 ffff880036a39fc0 0000000000000000 [ 11.295007] ffff880036233cc8 ffffffff812d23b2 ffff880036db4100 ffff88003692e808 [ 11.295007] Call Trace: [ 11.295007] [<ffffffff812d23b2>] __blk_mq_run_hw_queue+0x1e2/0x390 [ 11.295007] [<ffffffff812d2f35>] blk_mq_run_hw_queue+0xa5/0xd0 [ 11.295007] [<ffffffff812d3dc5>] blk_mq_insert_requests+0xc5/0x120 [ 11.295007] [<ffffffff812d4aeb>] blk_mq_flush_plug_list+0x13b/0x160 [ 11.295007] [<ffffffff812ca119>] blk_flush_plug_list+0xc9/0x230 [ 11.295007] [<ffffffff812ca634>] blk_finish_plug+0x14/0x40 [ 11.295007] [<ffffffffa01d1d5d>] dmcrypt_write+0x1ad/0x270 [dm_crypt] [ 11.295007] [<ffffffff810b7bb0>] ? wake_up_state+0x20/0x20 [ 11.295007] [<ffffffffa01d1bb0>] ? crypt_map+0x130/0x130 [dm_crypt] [ 11.295007] [<ffffffff810a4a8f>] kthread+0xcf/0xe0 [ 11.295007] [<ffffffff810a49c0>] ? kthread_create_on_node+0x140/0x140 [ 11.295007] [<ffffffff81643458>] ret_from_fork+0x58/0x90 [ 11.295007] [<ffffffff810a49c0>] ? kthread_create_on_node+0x140/0x140 [ 11.295007] Code: df 49 03 be b0 00 00 00 4c 89 fe 48 83 c7 08 e8 72 de 5e e1 49 8b 86 b0 00 00 00 48 8b 3c 18 e8 52 18 01 00 31 c0 e9 d0 fe ff ff <0f> 0b 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 41 [ 11.295007] RIP [<ffffffffa004c825>] virtio_queue_rq+0x285/0x290 [virtio_blk] [ 11.295007] RSP <ffff880036233bf8> [ 11.322446] ---[ end trace 9479ebc7ad80eb57 ]--- [ 11.322824] Kernel panic - not syncing: Fatal exception [ 11.323435] drm_kms_helper: panic occurred, switching back to text console
Follow-Up: It seems that the kernel had been upgraded again, which I hadn't noticed, and that caused the problem to reoccur. AFAIK, I haven't changed anything on VM-side that would cause a different kernel configuration to be built upon upgrades. Nevertheless, I don't know if this is specific to the kernel version being installed or the way it is being installed during the upgrade. Successful boot: 3.10.0-229.7.2.el7.x86_64 Error-prone boot: 3.10.0-306.0.1.el7.x86_64
Created attachment 1082075 [details] kernel build config diffs 3.10.0-229.7.2.el7 vs. 3.10.0-306.0.1.el7 kernel build config diffs 3.10.0-229.7.2.el7 vs. 3.10.0-306.0.1.el7 Maybe it'll help root out the problem if I include this diff before I remove the newer kernel again.
*********** MASS BUG UPDATE ************** We apologize for the inconvenience. There is a large number of bugs to go through and several of them have gone stale. Due to this, we are doing a mass bug update across all of the Fedora 22 kernel bugs. Fedora 22 has now been rebased to 4.2.3-200.fc22. Please test this kernel update (or newer) and let us know if you issue has been resolved or if it is still present with the newer kernel. If you have moved on to Fedora 23, and are still experiencing this issue, please change the version to Fedora 23. If you experience different issues, please open a new bug report for those.
*********** MASS BUG UPDATE ************** This bug is being closed with INSUFFICIENT_DATA as there has not been a response in over 4 weeks. If you are still experiencing this issue, please reopen and attach the relevant data from the latest kernel you are running and any data that might have been requested previously.