Bug 1669124
| Summary: | Attempting to remove a read-only VDO volume after failed growPhysical (due to read-only mode) encounters NULL pointer dereference | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Bryan Gurney <bgurney> | ||||
| Component: | kmod-kvdo | Assignee: | Sweet Tea Dorminy <sweettea> | ||||
| Status: | CLOSED ERRATA | QA Contact: | vdo-qe | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 8.0 | CC: | awalsh, sweettea | ||||
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
||||
| Target Release: | 8.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | 6.2.1.8 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2019-11-05 22:12:24 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | 1659247, 1682560 | ||||||
| Bug Blocks: | |||||||
| Attachments: |
|
||||||
|
Description
Bryan Gurney
2019-01-24 11:51:52 UTC
This is a bug in the VDO kernel/ directory not the base/ directory -- unit testing this case yields no issue. Created attachment 1523092 [details] vmcore-dmesg.txt from test system vdo-storage-02 running BZ1669124 test using kernel-4.18.0-63.el8.x86_64 I managed to reproduce this on storageqe-90 with 10TB SATA backing drive with kernels -60, -61 and -63. Default compose RHEL-8.0-20190116.1 is enough to reproduce it. I was not able to reproduce it on -60 kernel the first time, but it might have been caused by unclean system or me trying on LVM instead of partition the first time. This has the same root cause as https://bugzilla.redhat.com/show_bug.cgi?id=1659247 and can be used to test the fix for that. So I tried to NOT reproduce it on -60 kernel with LVM on clean install and I succeeded...to a degree.
Basically reproducing with same size LVM as the partition and trying to remove the vdo device fails with I/O error:
# date; time vdo remove --name=vdo --verbose; date
Thu Jan 24 14:51:27 CET 2019
Removing VDO vdo
Stopping VDO vdo
vdodumpconfig /dev/disk/by-id/dm-uuid-LVM-VN36nRj5fXGtDe2Lqw0u9gQSxsHnZfPSsufqVAv5Nd9szgxM0Bf4uAgb62nI8MJZ
dmsetup status --target vdo vdo
dmsetup resume vdo
vdo: ERROR - Can't resume VDO volume vdo; device-mapper: resume ioctl on vdo failed: Input/output error
vdo: ERROR - device-mapper: resume ioctl on vdo failed: Input/output error
real 0m0.183s
user 0m0.130s
sys 0m0.053s
Thu Jan 24 14:51:27 CET 2019
syslog:
Jan 24 14:51:27 storageqe-90 kernel: kvdo1:dmsetup: resuming device 'vdo'
Jan 24 14:51:27 storageqe-90 kernel: kvdo1:dmsetup: Resizing logical to 25117680
Jan 24 14:51:27 storageqe-90 kernel: kvdo1:dmsetup: grow logical operation failed, result = 2060
Jan 24 14:51:27 storageqe-90 kernel: kvdo1:dmsetup: Commit of modifications to device 'vdo' failed: kvdo: Parameters have conflicting values (2060)
Jan 24 14:51:27 storageqe-90 kernel: kvdo: dmsetup: mapToSystemError: mapping internal status code 2060 (kvdo: VDO_PARAMETER_MISMATCH: kvdo: Parameters have conflicting values) to EIO
Jan 24 14:51:27 storageqe-90 kernel: device-mapper: table: 253:1: vdo: preresume failed, error = -5
Jan 24 14:51:27 storageqe-90 vdo[17809]: ERROR - Can't resume VDO volume vdo; device-mapper: resume ioctl on vdo failed: Input/output error
Jan 24 14:51:27 storageqe-90 vdo[17809]: ERROR - device-mapper: resume ioctl on vdo failed: Input/output error
Than I partitioned another drive with intention to try the original reproducer at this point. Though I made a typo in the device at vdo creation (/dev/sdb1 should be /dev/sda1) and did not notice it at first. I corrected that and the new VDO device got created. At that point I got vdoStatus null pointer crash.
[root@storageqe-90 ~]# parted /dev/sda
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel MSDOS
(parted) mkpart
Partition type? primary/extended? pri
File system type? [ext2]?
Start? 1MB
End? 100GB
(parted) p
Model: HGST HUH721010AL4200 (scsi)
Disk /dev/sda: 10.0TB
Sector size (logical/physical): 4096B/4096B
Partition Table: msdos
Disk Flags:
Number Start End Size Type File system Flags
1 1049kB 100GB 100GB primary ext2 lba
(parted) q
Information: You may need to update /etc/fstab.
[root@storageqe-90 ~]# vdo create --name vdo1 --device /dev/sdb1
Creating VDO vdo1
^CTraceback (most recent call last):
File "/usr/bin/vdo", line 150, in <module>
main()
File "/usr/bin/vdo", line 128, in main
operation.run(arguments)
File "/usr/lib/python3.6/site-packages/vdo/vdomgmnt/VDOOperation.py", line 217, in run
self.execute(args)
File "/usr/lib/python3.6/site-packages/vdo/vdomgmnt/VDOOperation.py", line 68, in wrap
return lock(False, func, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdo/vdomgmnt/VDOOperation.py", line 63, in lock
return func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdo/utils/Transaction.py", line 223, in wrap
result = func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdo/vdomgmnt/VDOOperation.py", line 323, in execute
vdo.create(args.force)
File "/usr/lib/python3.6/site-packages/vdo/utils/Transaction.py", line 223, in wrap
result = func(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/vdo/vdomgmnt/VDOService.py", line 306, in create
self._createCheckCleanDevice()
File "/usr/lib/python3.6/site-packages/vdo/vdomgmnt/VDOService.py", line 1371, in _createCheckCleanDevice
runCommand(['pvcreate', '-qq', '--test', self.device])
File "/usr/lib/python3.6/site-packages/vdo/utils/Command.py", line 299, in runCommand
return Command(commandList, kwargs.pop('environment', None)).run(**kwargs)
File "/usr/lib/python3.6/site-packages/vdo/utils/Command.py", line 173, in run
output = self._execute(stdin)
File "/usr/lib/python3.6/site-packages/vdo/utils/Command.py", line 256, in _execute
stdoutdata, stderrdata = p.communicate(stdin)
File "/usr/lib64/python3.6/subprocess.py", line 863, in communicate
stdout, stderr = self._communicate(input, endtime, timeout)
File "/usr/lib64/python3.6/subprocess.py", line 1534, in _communicate
ready = selector.select(timeout)
File "/usr/lib64/python3.6/selectors.py", line 376, in select
fd_event_list = self._poll.poll(timeout)
KeyboardInterrupt
[root@storageqe-90 ~]# vdo create --name vdo1 --device /dev/sdb1 --force
Creating VDO vdo1
vdo: ERROR - vdoformat: /dev/sdb1 must be a block device
[root@storageqe-90 ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 9.1T 0 disk
└─sda1 8:1 0 93.1G 0 part
sdb 8:16 0 9.1T 0 disk
└─vg-lv 253:0 0 200G 0 lvm
└─vdo 253:1 0 95.8G 0 vdo
[root@storageqe-90 ~]# vdo create --name vdo1 --device /dev/sda1 --force
Creating VDO vdo1
Starting VDO vdo1
Starting compression on VDO vdo1
VDO instance 2 volume is ready at /dev/mapper/vdo1
CRASH:
[ 467.242685] BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
[ 467.250513] PGD 0 P4D 0
[ 467.253051] Oops: 0000 [#1] SMP NOPTI
[ 467.256718] CPU: 42 PID: 17728 Comm: dmeventd Kdump: loaded Tainted: G O --------- --- 4.18.0-60.el8.x86_64 #1
[ 467.268012] Hardware name: Supermicro AS -2023US-TR4/H11DSU-iN, BIOS 1.1a 04/26/2018
[ 467.275769] RIP: 0010:getKernelLayerBdev+0x10/0x20 [kvdo]
[ 467.281168] Code: e8 c5 4e ff ff 0f b6 c8 e9 38 fe ff ff 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 87 08 01 00 00 48 8b 40 08 <48> 8b 00 c3 66 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 85
[ 467.299910] RSP: 0018:ffffc06307fabbc8 EFLAGS: 00010246
[ 467.305135] RAX: 0000000000000000 RBX: 0000000000003ea0 RCX: 0000000001900000
[ 467.312263] RDX: ffffffffc0a249bd RSI: 0000000000000002 RDI: ffff9d2ee6fe3000
[ 467.319396] RBP: ffff9d2ee6fe3000 R08: ffff9d24e79a3338 R09: ffffffffc0a24772
[ 467.326527] R10: 0000000000000000 R11: ffff9d24e79a19e8 R12: ffff9d24d6d60160
[ 467.333657] R13: ffff9d2ee6fe3670 R14: ffffffffc0a24772 R15: ffffffffc0a24758
[ 467.340787] FS: 00007f9761192700(0000) GS:ffff9d24e7980000(0000) knlGS:0000000000000000
[ 467.348871] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 467.354614] CR2: 0000000000000000 CR3: 000000021b0b0000 CR4: 00000000003406e0
[ 467.361747] Call Trace:
[ 467.364217] vdoStatus+0x11d/0x170 [kvdo]
[ 467.368230] retrieve_status+0xa7/0x1f0 [dm_mod]
[ 467.372856] ? dm_get_live_or_inactive_table.isra.7+0x20/0x20 [dm_mod]
[ 467.379378] table_status+0x61/0xa0 [dm_mod]
[ 467.383650] ctl_ioctl+0x1af/0x3f0 [dm_mod]
[ 467.387841] ? selinux_file_ioctl+0xc0/0x200
[ 467.392112] dm_ctl_ioctl+0xa/0x10 [dm_mod]
[ 467.396296] do_vfs_ioctl+0xa4/0x630
[ 467.399873] ksys_ioctl+0x60/0x90
[ 467.403196] __x64_sys_ioctl+0x16/0x20
[ 467.406952] do_syscall_64+0x5b/0x1b0
[ 467.410616] entry_SYSCALL_64_after_hwframe+0x65/0xca
[ 467.415667] RIP: 0033:0x7f975f68445b
[ 467.419241] Code: 0f 1e fa 48 8b 05 2d aa 2c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 0f 1f 44 00 00 f3 0f 1e fa b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d fd a9 2c 00 f7 d8 64 89 01 48
[ 467.437979] RSP: 002b:00007f97611919d8 EFLAGS: 00000202 ORIG_RAX: 0000000000000010
[ 467.445548] RAX: ffffffffffffffda RBX: 0000563447e38a40 RCX: 00007f975f68445b
[ 467.452681] RDX: 00007f975803ca20 RSI: 00000000c138fd0c RDI: 0000000000000007
[ 467.459810] RBP: 00007f975fbb6173 R08: 0000000000000004 R09: 00007f975fbb6d00
[ 467.466934] R10: 000000000000001e R11: 0000000000000202 R12: 00007f975803ca20
[ 467.474059] R13: 0000000000000000 R14: 00007f975803cad0 R15: 00007f975803b280
[ 467.481191] Modules linked in: kvdo(O) uds(O) nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc vfat fat dm_multipath dm_mod amd64_edac_mod edac_mce_amd kvm_amd joydev ipmi_ssif kvm irqbypass crct10dif_pclmul crc32_pclmul sg ghash_clmulni_intel pcspkr sp5100_tco ipmi_si ccp i2c_piix4 k10temp ipmi_devintf ipmi_msghandler acpi_cpufreq xfs libcrc32c sd_mod ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm igb nvme megaraid_sas drm nvme_core crc32c_intel dca i2c_algo_bit pinctrl_amd
[ 467.528827] CR2: 0000000000000000
Hi Jakub: I have confirmed that the new crash is unrelated -- can you please open a new BZ for it? (In reply to Sweet Tea Dorminy from comment #7) > Hi Jakub: > > I have confirmed that the new crash is unrelated -- can you please open a > new BZ for it? Hi sweettea, Here is the new BZ 1669960. kmod-kvdo-6.2.1.138-57.el8.x86_64 vdo-6.2.1.134-11.el8.x86_64 It is not possible to grow the device now as VDO correctly complains about it being read-only: vdo: ERROR - Device vdo1 could not be changed; device-mapper: reload ioctl on vdo1 failed: Input/output error vdo: ERROR - device-mapper: reload ioctl on vdo1 failed: Input/output error syslog: [23332.263469] device-mapper: table: 253:0: vdo: Device prepareToGrowPhysical failed [23332.270959] device-mapper: ioctl: error adding target to table Attempting to remove the device after results in clean removal without issues. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:3548 |