RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1676643 - system kernel panic on starting usb vdo volume
Summary: system kernel panic on starting usb vdo volume
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: kmod-kvdo
Version: 7.6
Hardware: All
OS: Linux
high
high
Target Milestone: rc
: ---
Assignee: Ken Raeburn
QA Contact: Filip Suba
URL:
Whiteboard:
Depends On:
Blocks: 1711360
TreeView+ depends on / blocked
 
Reported: 2019-02-12 18:15 UTC by Alex Wang
Modified: 2022-03-13 16:58 UTC (History)
4 users (show)

Fixed In Version: 6.1.3.2
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-03-31 20:00:30 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
dmesg from vmcore (90.41 KB, text/plain)
2019-02-12 18:15 UTC, Alex Wang 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:1105 0 None None None 2020-03-31 20:00:39 UTC

Description Alex Wang 2019-02-12 18:15:41 UTC 
Created attachment 1534192 [details]
dmesg from vmcore

Description of problem:

When attempting to start a vdo volume created on a USB device, system kernel panics

Version-Release number of selected component (if applicable):

kernel-3.10.0-933.el7.x86_64
vdo-6.1.1.125-3.el7.x86_64
kmod-kvdo-6.1.1.125-5.el7.x86_64

How reproducible:


Steps to Reproduce:
1. Create vdo volume on USB drive
2. Create xfs fs on vdo volume
3. Stop vdo volume
4. Detach USB device
5. Re-attach USB device
6. vdo start -a

Actual results:

System kernel panics

Expected results:

System should not kernel panic

Additional info:

On reboot after panic, if the USB drive is not attached, the service fails to start.  Once booted into the system, if you attempt to manually to start the service with the USB drive attached, it will also kernel panic the system.


[ 3370.320432] kvdo: modprobe: loaded version 6.1.1.125
[ 3388.561219] kvdo0:dmsetup: starting device 'vdo_usb'
[ 3388.561647] kvdo0:dmsetup: underlying device, REQ_FLUSH: not supported, REQ_FUA: not supported
[ 3388.561649] kvdo0:dmsetup: Using write policy sync automatically.
[ 3388.561662] kvdo0:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5
[ 3388.658454] kvdo0:journalQ: VDO commencing normal operation
[ 3388.658527] kvdo0:dmsetup: Setting UDS index target state to online
[ 3388.658533] kvdo0:dmsetup: device 'vdo_usb' started
[ 3388.658536] uds: kvdo0:dedupeQ: creating index: dev=/dev/disk/by-id/usb-SMI_USB_DISK-0:0 offset=4096 size=2781704192
[ 3388.658644] kvdo0:dmsetup: resuming device 'vdo_usb'
[ 3388.658646] kvdo0:dmsetup: device 'vdo_usb' resumed
[ 3388.693270] kvdo0:packerQ: compression is enabled
[ 3389.375896] uds: kvdo0:dedupeQ: Using 8 indexing zones for concurrency.
[ 3778.900522] XFS (dm-3): Mounting V5 Filesystem
[ 3779.145035] XFS (dm-3): Ending clean mount
[ 3802.592918] XFS (dm-3): Unmounting Filesystem
[ 3816.049190] kvdo0:dmsetup: suspending device 'vdo_usb'
[ 3816.049223] kvdo0:packerQ: compression is disabled
[ 3816.076424] kvdo0:packerQ: compression is enabled
[ 3816.076433] kvdo0:dmsetup: device 'vdo_usb' suspended
[ 3816.076452] kvdo0:dmsetup: stopping device 'vdo_usb'
[ 3817.102116] kvdo0:dmsetup: Setting UDS index target state to closed
[ 3817.102651] uds: kvdo0:dedupeQ: index_0: beginning save (vcn 4294967295)
[ 3825.963411] kvdo0:dmsetup: device 'vdo_usb' stopped
[ 3874.476169] usb 3-5.3: USB disconnect, device number 4
[ 3885.236482] usb 3-5.4: new high-speed USB device number 6 using xhci_hcd
[ 3885.324067] usb 3-5.4: New USB device found, idVendor=090c, idProduct=1000
[ 3885.324083] usb 3-5.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 3885.324085] usb 3-5.4: Product: USB DISK
[ 3885.324086] usb 3-5.4: Manufacturer: SMI Corporation
[ 3885.325044] usb-storage 3-5.4:1.0: USB Mass Storage device detected
[ 3885.325142] scsi host13: usb-storage 3-5.4:1.0
[ 3886.736713] scsi 13:0:0:0: Direct-Access     SMI      USB DISK         1100 PQ: 0 ANSI: 0 CCS
[ 3886.736857] scsi 13:0:0:0: alua: supports implicit and explicit TPGS
[ 3886.736860] scsi 13:0:0:0: alua: No target port descriptors found
[ 3886.736862] scsi 13:0:0:0: alua: not attached
[ 3886.737136] sd 13:0:0:0: Attached scsi generic sg2 type 0
[ 3886.737225] sd 13:0:0:0: [sdc] 15730688 512-byte logical blocks: (8.05 GB/7.50 GiB)
[ 3886.737846] sd 13:0:0:0: [sdc] Write Protect is off
[ 3886.737849] sd 13:0:0:0: [sdc] Mode Sense: 43 00 00 00
[ 3886.738444] sd 13:0:0:0: [sdc] No Caching mode page found
[ 3886.738452] sd 13:0:0:0: [sdc] Assuming drive cache: write through
[ 3886.743553] sd 13:0:0:0: [sdc] Attached SCSI removable disk
[ 3897.804351] kvdo1:dmsetup: starting device 'vdo_usb'
[ 3897.804471] kvdo1:dmsetup: underlying device, REQ_FLUSH: not supported, REQ_FUA: not supported
[ 3897.804474] kvdo1:dmsetup: Using write policy sync automatically.
[ 3897.804485] kvdo1:dmsetup: zones: 1 logical, 1 physical, 1 hash; base threads: 5
[ 3898.565129] ------------[ cut here ]------------
[ 3898.565162] kernel BUG at drivers/scsi/scsi_lib.c:1048!
[ 3898.565191] invalid opcode: 0000 [#1] SMP 
[ 3898.565222] Modules linked in: kvdo(O) uds(O) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc vfat fat sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass snd_hda_codec_hdmi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper snd_hda_codec_realtek ablk_helper snd_hda_codec_generic cryptd snd_hda_intel snd_hda_codec
[ 3898.565810]  uas snd_hda_core usb_storage mei_wdt iTCO_wdt iTCO_vendor_support intel_wmi_thunderbolt joydev snd_hwdep snd_seq i2c_i801 snd_seq_device snd_pcm sg snd_timer snd soundcore mei_me mei lpc_ich dm_multipath ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic rtsx_usb_sdmmc mmc_core rtsx_usb nouveau video mxm_wmi i2c_algo_bit drm_kms_helper qla2xxx syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul drm crct10dif_common crc32c_intel serio_raw ahci drm_panel_orientation_quirks nvme_fc nvme_fabrics libahci nvme scsi_transport_fc ata_generic e1000e pata_acpi scsi_tgt nvme_core libata ptp pps_core wmi dm_mirror dm_region_hash dm_log dm_mod
[ 3898.566329] CPU: 5 PID: 34 Comm: ksoftirqd/5 Kdump: loaded Tainted: G           O   ------------   3.10.0-957.5.1.el7.x86_64 #1
[ 3898.566388] Hardware name: LENOVO 30A6S0FS00/To be filled by O.E.M., BIOS A4KT73AUS 06/04/2015
[ 3898.566434] task: ffff88ce3989d140 ti: ffff88ce3990c000 task.ti: ffff88ce3990c000
[ 3898.566477] RIP: 0010:[<ffffffff88ed845c>]  [<ffffffff88ed845c>] scsi_init_io+0x1cc/0x200
[ 3898.566527] RSP: 0018:ffff88ce3990faf0  EFLAGS: 00010046
[ 3898.566555] RAX: ffff88cd384af800 RBX: ffff88d11fbccf00 RCX: 0000000000000000
[ 3898.566593] RDX: ffff88d11fbccf00 RSI: 0000000000000020 RDI: ffff88d12867ddc0
[ 3898.566634] RBP: ffff88ce3990fb28 R08: ffff88cd384af838 R09: ffffffff88ecfc9b
[ 3898.566673] R10: ffff88d13fd65d00 R11: ffffda9a51a19f00 R12: ffff88d12867ddc0
[ 3898.566711] R13: 0000000000000000 R14: ffff88d12867ddc0 R15: 0000000000000e00
[ 3898.566749] FS:  0000000000000000(0000) GS:ffff88d13fd40000(0000) knlGS:0000000000000000
[ 3898.566796] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 3898.566826] CR2: 00007eff486a25c0 CR3: 00000001dfc10000 CR4: 00000000001607e0
[ 3898.566864] Call Trace:
[ 3898.566885]  [<ffffffffc030039d>] sd_setup_read_write_cmnd+0x3d/0x950 [sd_mod]
[ 3898.566927]  [<ffffffff88ecfd3a>] ? __scsi_get_command+0x1a/0xf0
[ 3898.568640]  [<ffffffffc0300cdf>] sd_init_command+0x2f/0xc0 [sd_mod]
[ 3898.570377]  [<ffffffff88ed85a1>] scsi_setup_cmnd+0x111/0x1c0
[ 3898.572215]  [<ffffffff88ed872b>] scsi_prep_fn+0xdb/0x180
[ 3898.573933]  [<ffffffff88d46a8a>] blk_peek_request+0x17a/0x2a0
[ 3898.575637]  [<ffffffff88eda0f8>] scsi_request_fn+0x48/0x680
[ 3898.577265]  [<ffffffff88d435d9>] __blk_run_queue+0x39/0x50
[ 3898.578844]  [<ffffffff88d43656>] blk_run_queue+0x26/0x40
[ 3898.580400]  [<ffffffff88ed8b28>] scsi_run_queue+0x258/0x2f0
[ 3898.581948]  [<ffffffff88d3ebcd>] ? elv_requeue_request+0x5d/0xb0
[ 3898.583500]  [<ffffffff88edae3f>] scsi_io_completion+0x46f/0x6a0
[ 3898.585045]  [<ffffffff88ecffec>] scsi_finish_command+0xdc/0x140
[ 3898.586605]  [<ffffffff88eda082>] scsi_softirq_done+0x132/0x160
[ 3898.588076]  [<ffffffff88d4d2c6>] blk_done_softirq+0x96/0xc0
[ 3898.589543]  [<ffffffff88aa0f45>] __do_softirq+0xf5/0x280
[ 3898.591026]  [<ffffffff88aa1108>] run_ksoftirqd+0x38/0x50
[ 3898.592490]  [<ffffffff88aca604>] smpboot_thread_fn+0x144/0x1a0
[ 3898.593964]  [<ffffffff88aca4c0>] ? lg_double_unlock+0x40/0x40
[ 3898.595445]  [<ffffffff88ac1c71>] kthread+0xd1/0xe0
[ 3898.596928]  [<ffffffff88ac1ba0>] ? insert_kthread_work+0x40/0x40
[ 3898.598348]  [<ffffffff89174c37>] ret_from_fork_nospec_begin+0x21/0x21
[ 3898.599790]  [<ffffffff88ac1ba0>] ? insert_kthread_work+0x40/0x40
[ 3898.601200] Code: 8b 17 e8 d8 b1 e9 ff 39 45 cc 7c 43 48 8b 53 38 0f b7 92 18 06 00 00 39 d0 7f 32 4d 89 be f0 00 00 00 41 89 47 08 e9 24 ff ff ff <0f> 0b be 3a 04 00 00 48 c7 c7 ba 7e 4b 89 41 b7 01 e8 3e f3 bb 
[ 3898.604405] RIP  [<ffffffff88ed845c>] scsi_init_io+0x1cc/0x200
[ 3898.605969]  RSP <ffff88ce3990faf0>
Comment 2 Bryan Gurney 2019-02-12 18:38:10 UTC 
I was able to reproduce this with a USB drive from the same vendor / model as above:

[8:0:0:0]    disk    SMI      USB DISK         1100  /dev/sdf

...and these component versions:

vdo-6.1.1.125-3.el7.x86_64
kernel-3.10.0-957.el7.x86_64
kmod-kvdo-6.1.1.125-5.el7.x86_64
Comment 3 Alex Wang 2019-02-12 22:34:27 UTC 
Kernel package in question is actually 3.10.0-957.5.1.el7.x86_64 and not the pre-GA version as mentioned in the first comment.  vmcore dmesg version is from GA kernel.
Comment 5 Dennis Keefe 2019-02-12 23:16:14 UTC 
I'm not able to recreate this on RHEL8.  I believe the issue was fixed here https://patchwork.kernel.org/patch/10291407/
Setting scsi_init_io to warn.
Comment 8 Ken Raeburn 2019-07-11 21:27:36 UTC 
It appears that the VDO driver is not properly resetting some of the fields associated with the bio structures in between I/O operations, and in the very specific case of the last 4kB of a USB storage device, those fields will reliably get modified during the course of the I/O operation, leading to this problem. We're looking into possible fixes. At the moment I'm not aware of other storage devices where the VDO bug will cause problems, but the possibility exists...

A possible workaround may be to lay out the storage (via partition table or LVM) so that VDO does not use that last 4kB block from the USB drive.
Comment 13 Jakub Krysl 2019-10-15 14:38:54 UTC 
Mass migration to Filip.
Comment 14 Filip Suba 2020-02-11 14:48:09 UTC 
Verified with vdo 6.1.3.4.el7. Issue is no longer observable.
 266.889881] kvdo2:dmsetup: device 'vdo2' started
[  266.890020] kvdo2:dmsetup: resuming device 'vdo2'
[  266.890023] kvdo2:dmsetup: device 'vdo2' resumed
[  266.890311] uds: kvdo2:dedupeQ: loading or rebuilding index: dev=/dev/disk/by-id/usb-SMI_USB_DISK-0:0 offset=4096 size=2781704192
[  266.906806] uds: kvdo2:dedupeQ: Using 2 indexing zones for concurrency.
[  266.964579] kvdo2:packerQ: compression is enabled
[  267.880243] uds: kvdo2:dedupeQ: loaded index from chapter 0 through chapter 0
Comment 16 errata-xmlrpc 2020-03-31 20:00:30 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1105
Note You need to log in before you can comment on or make changes to this bug.