RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1835450 - dmeventd segfault after attempting to derypt closed luks volume using OLD cryptsetup-reencrypt utility
Summary: dmeventd segfault after attempting to derypt closed luks volume using OLD cry...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: lvm2
Version: 8.2
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: rc
: 8.0
Assignee: David Teigland
QA Contact: cluster-qe@redhat.com
URL:
Whiteboard:
Depends On: 1809571
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-05-13 19:53 UTC by Corey Marthaler
Modified: 2021-09-07 11:49 UTC (History)
11 users (show)

Fixed In Version: lvm2-2.03.09-4.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-11-04 02:00:25 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:4546 0 None None None 2020-11-04 02:00:42 UTC

Description Corey Marthaler 2020-05-13 19:53:03 UTC 
Description of problem:
I was just attempting to see what the current state of bug 1743891 was, by using the cryptsetup-reencrypt utility to attempt to decrypt a closed luks volume, expecting it to fail with an error, but the decrypt actually appeared to work, presumably corrupting the volume. I'll attempt to reproduce and boil this test down.


SCENARIO - [fs_io_writes_to_origin_reads_from_thin_snaps]
Create snapshots of origin with fs data, and then verify that data on snapshots in between encryption operations
Making pool volume
lvcreate  --thinpool POOL -L 4G  --zero y --poolmetadatasize 4M snapper_thinp

Making origin volume
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n origin
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other1
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other2
lvcreate  -V 1G -T snapper_thinp/POOL -n other3
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other4
  WARNING: Sum of all thin volume sizes (5.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (4.00 GiB).
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other5
  WARNING: Sum of all thin volume sizes (6.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (4.00 GiB).
Placing an xfs filesystem on origin volume
Mounting origin volume

Writing files to /mnt/origin
Checking files on /mnt/origin

syncing before snap creation...
Making 1st snapshot of origin volume
lvcreate  -y -k n -s /dev/snapper_thinp/origin -n fs_snap1
Mounting 1st snap volume
Checking files on /mnt/fs_snap1
Writing files to /mnt/origin

syncing before snap creation...
Making 2nd snapshot of origin volume
lvcreate  -y -k n -s /dev/snapper_thinp/origin -n fs_snap2
Mounting 2nd snap volume
Checking files on /mnt/fs_snap2

Writing files to /mnt/origin

syncing before snap creation...
Encrypt existing filesystem using a detached header file (RFE 1566005)
cryptsetup reencrypt --encrypt /dev/snapper_thinp/origin --header /tmp/luks_detachedheader.8598
cryptsetup luksOpen --disable-keyring /dev/snapper_thinp/origin luks_origin --header /tmp/luks_detachedheader.8598

Checking files on /mnt/origin
Checking files on /mnt/origin
Checking files on /mnt/origin

Writing files to /mnt/origin
Checking files on /mnt/origin

Making 3rd snapshot of origin volume
lvcreate  -y -k n -s /dev/snapper_thinp/origin -n fs_snap3
cryptsetup luksOpen /dev/snapper_thinp/fs_snap3 fs_snap3 --header /tmp/luks_detachedheader_snap3.8598
Mounting 3rd snap volume
Checking files on /mnt/fs_snap1
Checking files on /mnt/fs_snap2
Checking files on /mnt/fs_snap2
Checking files on /mnt/fs_snap3
Checking files on /mnt/fs_snap3
Checking files on /mnt/fs_snap3
Checking files on /mnt/fs_snap3
umount /mnt/fs_snap1 /mnt/fs_snap2 /mnt/fs_snap3 /mnt/origin
cryptsetup luksClose fs_snap3
cryptsetup luksClose luks_origin

Decrypt and verify underlying LV before removal
First w/ old utility (bug 1743891): cryptsetup-reencrypt --decrypt /dev/snapper_thinp/origin --header /tmp/luks_detachedheader.8598
echo "Str0ngP455w0rd###" | cryptsetup-reencrypt --decrypt /dev/snapper_thinp/origin --header /tmp/luks_detachedheader.8598


May 13 14:07:55 hayes-01 lvm[162484]: Device open /dev/mapper/LUKS-03b0d064-e7a2-43bc-9147-78c849ac139e.org 253:12 failed errno 2
May 13 14:07:55 hayes-01 lvm[162484]: Device open /dev/mapper/LUKS-03b0d064-e7a2-43bc-9147-78c849ac139e.org 253:12 failed errno 2
May 13 14:07:55 hayes-01 lvm[162484]: WARNING: Scan ignoring device 253:12 with no paths.
May 13 14:07:55 hayes-01 kernel: dmeventd[169150]: segfault at fd0c ip 00007f58f8ed9655 sp 00007f58fa9db318 error 4 in libc-2.28.so[7f58f8d7c000+1b9000]
May 13 14:07:55 hayes-01 kernel: Code: 00 00 00 00 00 31 c0 c5 f8 77 c3 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 89 f9 48 89 fa c5 f9 ef c0 83 e1 3f 83 f9 20 77 2b <c5> fd 74 0f c5 fd d7 c1 85 c0 0f 85 eb 00 00 00 48 83 c7 20 83 e1
May 13 14:07:55 hayes-01 systemd[1]: Created slice system-systemd\x2dcoredump.slice.
May 13 14:07:55 hayes-01 systemd[1]: Started Process Core Dump (PID 169653/UID 0).
May 13 14:07:55 hayes-01 systemd[1]: dm-event.service: Main process exited, code=killed, status=11/SEGV
May 13 14:07:55 hayes-01 systemd[1]: dm-event.service: Failed with result 'signal'.
May 13 14:07:56 hayes-01 systemd-coredump[169654]: Process 162484 (dmeventd) of user 0 dumped core.#012#012Stack trace of thread 169150:#012#0  0x00007f58f8ed9655 __strlen_avx2 (libc.so.6)#012#1  0x00007f58f8dce43f vfprintf (libc.so.6)#012#2  0x00007f58f8e8719d __vsnprintf_chk (libc.so.6)#012#3  0x00007f58f7a3a22c _vprint_log (liblvm2cmd.so.2.03)#012#4  0x00007f58f7a3ae59 print_log (liblvm2cmd.so.2.03)#012#5  0x00007f58f7a1822b dev_get_size (liblvm2cmd.so.2.03)#012#6  0x00007f58f7a37076 _dev_in_hint_hash (liblvm2cmd.so.2.03)#012#7  0x00007f58f7a384b4 write_hint_file (liblvm2cmd.so.2.03)#012#8  0x00007f58f7a35788 label_scan (liblvm2cmd.so.2.03)#012#9  0x00007f58f79fe998 lvmcache_label_scan (liblvm2cmd.so.2.03)#012#10 0x00007f58f7af009a process_each_vg (liblvm2cmd.so.2.03)#012#11 0x00007f58f7ad61a5 lvresize (liblvm2cmd.so.2.03)#012#12 0x00007f58f7ad36ed lvm_run_command (liblvm2cmd.so.2.03)#012#13 0x00007f58f7b01bbe lvm2_run (liblvm2cmd.so.2.03)#012#14 0x00007f58f8086467 dmeventd_lvm2_run (libdevmapper-event-lvm2.so.2.03)#012#15 0x00007f58f8289964 _use_policy (libdevmapper-event-lvm2thin.so)#012#16 0x00007f58f8289c61 process_event (libdevmapper-event-lvm2thin.so)#012#17 0x000055ee9c8e3a7a _monitor_thread (dmeventd)#012#18 0x00007f58f95b82de start_thread (libpthread.so.0)#012#19 0x00007f58f8e77e83 __clone (libc.so.6)#012#012Stack trace of thread 162484:#012#0  0x00007f58f8e6f4ef __select (libc.so.6)#012#1  0x000055ee9c8e32f3 _client_read.isra.15 (dmeventd)#012#2  0x000055ee9c8e1e55 main (dmeventd)#012#3  0x00007f58f8d9f6a3 __libc_start_main (libc.so.6)#012#4  0x000055ee9c8e296e _start (dmeventd)#012#012Stack trace of thread 169151:#012#0  0x00007f58f95be7da pthread_cond_timedwait@@GLIBC_2.3.2 (libpthread.so.0)#012#1  0x000055ee9c8e5406 _timeout_thread (dmeventd)#012#2  0x00007f58f95b82de start_thread (libpthread.so.0)#012#3  0x00007f58f8e77e83 __clone (libc.so.6)



Version-Release number of selected component (if applicable):
kernel-4.18.0-193.2.1.el8_2    BUILT: Mon May  4 09:57:10 CDT 2020
lvm2-2.03.08-3.el8    BUILT: Wed Mar 18 08:37:57 CDT 2020
lvm2-libs-2.03.08-3.el8    BUILT: Wed Mar 18 08:37:57 CDT 2020
lvm2-dbusd-2.03.08-3.el8    BUILT: Wed Mar 18 08:40:15 CDT 2020
lvm2-lockd-2.03.08-3.el8    BUILT: Wed Mar 18 08:37:57 CDT 2020
boom-boot-1.0-1.el8    BUILT: Fri Nov 29 05:18:30 CST 2019
device-mapper-1.02.169-3.el8    BUILT: Wed Mar 18 08:37:57 CDT 2020
device-mapper-libs-1.02.169-3.el8    BUILT: Wed Mar 18 08:37:57 CDT 2020
device-mapper-event-1.02.169-3.el8    BUILT: Wed Mar 18 08:37:57 CDT 2020
device-mapper-event-libs-1.02.169-3.el8    BUILT: Wed Mar 18 08:37:57 CDT 2020
Comment 1 Zdenek Kabelac 2020-05-13 20:02:11 UTC 
Passing to Dave for exploration - since there were some 'hints' changes recently - f50e7ce76c0c6b2745d0ed2a766f46cea07f310c.
Comment 2 Zdenek Kabelac 2020-05-13 20:07:32 UTC 
Looking at bug 1809571 with QA failed - it likely might be duplicate.
Comment 3 David Teigland 2020-05-13 22:37:23 UTC 
It's unusual to end up with a device where all path names have been removed, so it has no name.  I don't know how to create/reproduce that, so I've not been able to verify this patch, but I expect it helps.  If we found a way to routinely test devs with no name, we'd probably find other places with problems, and it's likely that we will want to just drop a device entirely from dev-cache if all names have been removed.

https://sourceware.org/git/?p=lvm2.git;a=commit;h=2f29765e7fd1135d070310683cf486f07d041c81
Comment 8 Corey Marthaler 2020-08-17 22:29:13 UTC 
Marking verified in the rpms. 

kernel-4.18.0-232.el8    BUILT: Mon Aug 10 02:17:54 CDT 2020
lvm2-2.03.09-5.el8    BUILT: Wed Aug 12 15:51:50 CDT 2020
lvm2-libs-2.03.09-5.el8    BUILT: Wed Aug 12 15:51:50 CDT 2020
lvm2-lockd-2.03.09-5.el8    BUILT: Wed Aug 12 15:51:50 CDT 2020
device-mapper-1.02.171-5.el8    BUILT: Wed Aug 12 15:51:50 CDT 2020
device-mapper-libs-1.02.171-5.el8    BUILT: Wed Aug 12 15:51:50 CDT 2020
device-mapper-event-1.02.171-5.el8    BUILT: Wed Aug 12 15:51:50 CDT 2020
device-mapper-event-libs-1.02.171-5.el8    BUILT: Wed Aug 12 15:51:50 CDT 2020

I ran many iterations of the scenario that caused this (listed in comment #0) and saw no issues.

Create snapshots of origin with fs data, and then verify that data on snapshots in between encryption operations
Making pool volume
lvcreate  --thinpool POOL -L 4G  --zero y --poolmetadatasize 4M snapper_thinp

Sanity checking pool device (POOL) metadata
thin_check /dev/mapper/snapper_thinp-meta_swap.973
examining superblock
examining devices tree
examining mapping tree
checking space map counts

Making origin volume
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n origin
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other1
lvcreate  --virtualsize 1G -T snapper_thinp/POOL -n other2
lvcreate  -V 1G -T snapper_thinp/POOL -n other3
lvcreate  -V 1G -T snapper_thinp/POOL -n other4
  WARNING: Sum of all thin volume sizes (5.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (4.00 GiB).
lvcreate  -V 1G -T snapper_thinp/POOL -n other5
  WARNING: Sum of all thin volume sizes (6.00 GiB) exceeds the size of thin pool snapper_thinp/POOL (4.00 GiB).
Placing an xfs filesystem on origin volume
Mounting origin volume

Writing files to /mnt/origin
Checking files on /mnt/origin

syncing before snap creation...
Making 1st snapshot of origin volume
lvcreate  -y -k n -s /dev/snapper_thinp/origin -n fs_snap1
Mounting 1st snap volume
Checking files on /mnt/fs_snap1
Writing files to /mnt/origin

syncing before snap creation...
Making 2nd snapshot of origin volume
lvcreate  -y -k n -s /dev/snapper_thinp/origin -n fs_snap2
Mounting 2nd snap volume
Checking files on /mnt/fs_snap2

Writing files to /mnt/origin

syncing before snap creation...
Encrypt existing filesystem using a detached header file (RFE 1566005)
cryptsetup reencrypt --encrypt /dev/snapper_thinp/origin --header /tmp/luks_detachedheader.1766523
cryptsetup luksOpen --disable-keyring /dev/snapper_thinp/origin luks_origin --header /tmp/luks_detachedheader.1766523

Checking files on /mnt/origin
Checking files on /mnt/origin
Checking files on /mnt/origin

Writing files to /mnt/origin
Checking files on /mnt/origin

Making 3rd snapshot of origin volume
lvcreate  -y -k n -s /dev/snapper_thinp/origin -n fs_snap3
cryptsetup luksOpen /dev/snapper_thinp/fs_snap3 fs_snap3 --header /tmp/luks_detachedheader_snap3.1766523
Mounting 3rd snap volume
Checking files on /mnt/fs_snap1
Checking files on /mnt/fs_snap2
Checking files on /mnt/fs_snap2
Checking files on /mnt/fs_snap3
Checking files on /mnt/fs_snap3
Checking files on /mnt/fs_snap3
Checking files on /mnt/fs_snap3
umount /mnt/fs_snap1 /mnt/fs_snap2 /mnt/fs_snap3 /mnt/origin
cryptsetup luksClose fs_snap3
cryptsetup luksClose luks_origin

Decrypt and verify underlying LV before removal
First w/ old utility (bug 1743891): cryptsetup-reencrypt --decrypt /dev/snapper_thinp/origin --header /tmp/luks_detachedheader.1766523
Second (attenpt) w/ new utility: cryptsetup reencrypt --decrypt /dev/snapper_thinp/origin --header /tmp/luks_detachedheader.1766523
mount /dev/snapper_thinp/origin /mnt/origin
Checking files on /mnt/origin
Checking files on /mnt/origin
Checking files on /mnt/origin
Checking files on /mnt/origin


Removing snap volume snapper_thinp/fs_snap1
lvremove -f /dev/snapper_thinp/fs_snap1
Removing snap volume snapper_thinp/fs_snap2
lvremove -f /dev/snapper_thinp/fs_snap2
Removing snap volume snapper_thinp/fs_snap3
lvremove -f /dev/snapper_thinp/fs_snap3
Removing thin origin and other virtual thin volumes
Removing pool snapper_thinp/POOL
Comment 11 errata-xmlrpc 2020-11-04 02:00:25 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (lvm2 bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4546
Note You need to log in before you can comment on or make changes to this bug.