Bug 2162305

The steps up through fsadm work for me, but are you saying that the busctl steps are running more lvm commands where some failure appears?  Can we replace those busctl steps with direct lvm commands?  When I try the second busctl step it gives me an error:

"Call failed: No such interface “org.freedesktop.UDisks2.Encrypted” on object at path /org/freedesktop/UDisks2/block_devices/dm_2d0"

what is "dm_2d0"?

Ah, the reproducer is for a VM with no other LVM devices; you can check `busctl tree org.freedesktop.UDisks2` or `udisksctl dump` to get the actual object path.

I suppose libblockdev somehow tries to feed the password in to lvresize/fsadm (over stdin?), as it's provided through the API, and cannot be typed into a tty.

$ busctl call org.freedesktop.UDisks2 /org/freedesktop/UDisks2/lvm/test/lv org.freedesktop.UDisks2.LogicalVolume Resize 'ta{sv}' 170000000 0
Call failed: No such interface “org.freedesktop.UDisks2.LogicalVolume” on object at path /org/freedesktop/UDisks2/lvm/test/lv

busctl tree org.freedesktop.UDisks2 is not displaying any info about VGs or LVs.

Is this issue specifically related to calling lvm via udisks, i.e. do the equivalent lvm commands work fine when called directly?

Yes, the direct commands as in bug 2157591 work. That is asking for the passphrase on TTY. udisks/libblockdev somehow call this in a different way, although [1] does not make it obvious how.

Did you do the EnableModules call, and have udisks2-lvm2 package installed?

[1] https://github.com/storaged-project/libblockdev/blob/master/src/plugins/lvm.c#L1891

Thanks, I didn't have udisks2-lvm2 installed.  I'm now running all of those busctl commands, and it's not producing the error for me:

$ lvcreate -L 200M -n lv test
  Logical volume "lv" created.
$ echo 93R4P4pIqAH8 | cryptsetup luksFormat -i1 --type luks1 /dev/test/lv; echo 93R4P4pIqAH8 | cryptsetup luksOpen /dev/test/lv lvcr
$ mkfs.ext4 /dev/mapper/lvcr
mke2fs 1.46.5 (30-Dec-2021)
Creating filesystem with 202752 1k blocks and 50600 inodes
Filesystem UUID: daab05e2-c346-4437-bd82-b539ca5140ef
Superblock backups stored on blocks: 
        8193, 24577, 40961, 57345, 73729

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done 
$ fsadm -y resize /dev/mapper/lvcr 130M
resize2fs 1.46.5 (30-Dec-2021)
Resizing the filesystem on /dev/mapper/lvcr to 133120 (1k) blocks.
The filesystem on /dev/mapper/lvcr is now 133120 (1k) blocks long.
$ busctl call org.freedesktop.UDisks2 /org/freedesktop/UDisks2/Manager org.freedesktop.UDisks2.Manager EnableModules b true
$ busctl call org.freedesktop.UDisks2  /org/freedesktop/UDisks2/block_devices/dm_2d2 org.freedesktop.UDisks2.Encrypted Resize 'ta{sv}' 160000000 1 passphrase s 93R4P4pIqAH8
$ busctl call org.freedesktop.UDisks2 /org/freedesktop/UDisks2/lvm/test/lv org.freedesktop.UDisks2.LogicalVolume Resize 'ta{sv}' 170000000 0


I notice that the lvresize command generated from busctl is exercising a condition that I just fixed today in bug 2162144.  It's possible that this is the same issue.  However, when I revert the fix for that bug, I'm still not seeing the error, so I'm not sure it's the same issue.  In that bug, lvresize_fs_helper is unnecessarily calling cryptsetup resize (i.e. the crypt device is already reduced, so the unnecessary cryptsetup resize is a no op.)

After trying the busctl steps again *without* the fix for bug 2162144, I did manage to reproduce an error, but with a slightly different message:

$ busctl call org.freedesktop.UDisks2 /org/freedesktop/UDisks2/lvm/test/lv org.freedesktop.UDisks2.LogicalVolume Resize 'ta{sv}' 170000000 0
Call failed: Error resizing logical volume: Process reported exit code 5:   No size change.

With the fix for bug 2162144, I'm not seeing it, so I'm going to make this bz a duplicate.

*** This bug has been marked as a duplicate of bug 2162144 ***

I don't think it's a duplicate -- The numbers in the reproducer are specifically set so that the size *does* change. Perhaps you can the Resize call several times in a row, and it previously succeeded? That would explain the "No size change" error, but that's different than "Nothing to read on input.".

Reproducer in a completely predictable environment:

curl -L -O http://download.devel.redhat.com/rhel-9/nightly/RHEL-9/latest-RHEL-9/compose/BaseOS/x86_64/images/rhel-guest-image-9.2-20230119.4.x86_64.qcow2
# nothing fancy, just admin:foobar and root:foobar
curl -L -O https://github.com/cockpit-project/bots/raw/main/machine/cloud-init.iso
qemu-system-x86_64 -cpu host -enable-kvm -nographic -m 2048 -drive file=rhel-guest-image-9.2-20230119.4.x86_64.qcow2,if=virtio -snapshot -cdrom cloud-init.iso -net nic,model=virtio -net user,hostfwd=tcp::2201-:22

You can log into the VT as admin or root, but it's more comfortable with SSH:

ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o CheckHostIP=no -p 2201 admin@localhost

Password "foobar". Then "sudo -i".

Unfortunately the RHEL compose images don't have valid dnf repos, so you either need to subscribe, or set up the nightly one:

cat <<EOF > /etc/yum.repos.d/nightly.repo
[RHEL-NIGHTLY-BaseOS]    
name=baseos    
baseurl=http://download.devel.redhat.com/rhel-9/nightly/RHEL-9/latest-RHEL-9.2/compose/BaseOS/x86_64/os/    
enabled=1    
gpgcheck=0    
    
[RHEL-NIGHTLY-AppStream]    
name=appstream    
baseurl=http://download.devel.redhat.com/rhel-9/nightly/RHEL-9/latest-RHEL-9.2/compose/AppStream/x86_64/os/    
enabled=1    
gpgcheck=0    

EOF

dnf install -y udisks2-lvm2 cryptsetup

Then you can run the original reproducer, except for modifying the "foobar" password; it fails the default pwquality rules (we disable that in our test images):

modprobe scsi_debug dev_size_mb=300
sleep 1
#### WARNING!!!! Make sure that /dev/sda is the scsi_debug one, not a real drive!
vgcreate TEST /dev/sda
lvcreate TEST -n vol -L 200m
echo foobarfoo | cryptsetup luksFormat -q /dev/mapper/TEST-vol
echo foobarfoo | cryptsetup luksOpen /dev/mapper/TEST-vol l1
mkfs -t ext4 /dev/mapper/l1
fsadm -y resize /dev/mapper/l1 130M

busctl call org.freedesktop.UDisks2 /org/freedesktop/UDisks2/Manager org.freedesktop.UDisks2.Manager EnableModules b true
busctl call org.freedesktop.UDisks2 /org/freedesktop/UDisks2/block_devices/dm_2d0 org.freedesktop.UDisks2.Encrypted Resize 'ta{sv}' 160000000 1 passphrase s foobarfoo
busctl call org.freedesktop.UDisks2 /org/freedesktop/UDisks2/lvm/TEST/vol org.freedesktop.UDisks2.LogicalVolume Resize 'ta{sv}' 170000000 0

Created attachment 1939335 [details]
strace of udisks for the failing command

This is the strace -fvvs1024 of udisks for the last command. In particular:

15150 execve("/usr/sbin/lvm", ["lvm", "lvresize", "--force", "-L", "166015K", "TEST/vol"], ...
15151 execve("/usr/libexec/lvresize_fs_helper", ["/usr/libexec/lvresize_fs_helper", "--cryptresize", "--cryptpath", "/dev/dm-1", "--newsizebytes", "155189248"], ...
15022 sendmsg(6, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\3\1\1\222\0\0\0 \0\0\0P\0\0\0\4\1s\0$\0\0\0org.freedesktop.UDisks2.Error.Failed\0\0\0\0\6\1s\0\5\0\0\0:1.29\0\0\0\10\1g\0\1s\0\0\5\1u\0\2\0\0\0\215\0\0\0Error resizing logical volume: Process reported exi
t code 5: Nothing to read on input.\n  Failed to resize crypt dev with lvresize_fs_helper.\n\0", iov_len=242}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 242

This doesn't contain any trace of "foobarfoo" (i.e. the password), so I don't know how exactly this is being fed into the LVM tools.

(In reply to David Teigland from comment #1)
> 
> "Call failed: No such interface “org.freedesktop.UDisks2.Encrypted” on
> object at path /org/freedesktop/UDisks2/block_devices/dm_2d0"
> 
> what is "dm_2d0"?


just info comment: 

dm_2d0   is dm-0  with '-' encoded as _2d due to some dbus naming convention here.

(udiskctl dump     shows all the info for all the knowledge udisk has)

I've reproduced the same error, and it's an issue with password input.  (I'd been using luks1 in my testing which apparently has different password input requirements than your luks2 example.)

The failing busctl command is running the lvm command:
$ lvresize --force -L 166015K test/lv

If you run that command directly, and press ctrl-c at the password prompt:

$ lvresize --force -L 166015K test/lv
  Rounding size to boundary between physical extents: 164.00 MiB.
  File system found on crypt device /dev/dm-3 on LV test/lv.
  File system size 155189248b is adjusted for crypt data offset 16777216b.
  File system ext4+crypto_LUKS found on test/lv.
  File system size (130.00 MiB) is smaller than the requested size (148.00 MiB).
  File system reduce is not needed, skipping.
  Forcing cryptsetup resize (--resizefs should be used to enable crypt resize)
cryptsetup resize 303104 sectors /dev/dm-3
Enter passphrase for /dev/mapper/test-lv: Error reading passphrase from terminal.
cryptsetup failed
  Failed to resize crypt dev with lvresize_fs_helper.

So the problem is that lvresize->lvresize_fs_helper->cryptsetup requires a password input which is not being provided.  When run through busctl/udisks:

$ busctl call org.freedesktop.UDisks2 /org/freedesktop/UDisks2/lvm/test/lv org.freedesktop.UDisks2.LogicalVolume Resize 'ta{sv}' 170000000 0
Call failed: Error resizing logical volume: Process reported exit code 5: Nothing to read on input.
  Failed to resize crypt dev with lvresize_fs_helper.

And "nothing to read on input" is I believe the cryptsetup error when no password is provided at stdin.

The following works to provide the password:

$ echo 93R4P4pIqAH8 | lvresize --force -L 166015K test/lv
  Rounding size to boundary between physical extents: 164.00 MiB.
  File system found on crypt device /dev/dm-3 on LV test/lv.
  File system size 155189248b is adjusted for crypt data offset 16777216b.
  File system ext4+crypto_LUKS found on test/lv.
  File system size (130.00 MiB) is smaller than the requested size (148.00 MiB).
  File system reduce is not needed, skipping.
  Forcing cryptsetup resize (--resizefs should be used to enable crypt resize)
cryptsetup resize 303104 sectors /dev/dm-3
cryptsetup done
  Size of logical volume test/lv changed from 200.00 MiB (50 extents) to 164.00 MiB (41 extents).
  Logical volume test/lv successfully resized.

But I don't know how the busctl/udisks Resize command accepts and uses a password.


[Note: there is a separate issue, which is that the proper way to enable crypt resizing from lvresize is by setting --resizefs, even if the fs doesn't require reducing.  --resizefs is what enables the lvm command to look above the lv to handle any crypt or fs resizing.  It looks like --force previously enabled this, so for now we'll continue to accept --force to enable crypt resizing even though it's not advisable.]

It's not clear to me how the crypt password is supposed to be provided when Resize is initiated through busctl.

I have no idea, I'm afraid, I'm just the messenger. CC'ing the three main contributors of libblockdev. This worked fine before, maybe they can explain how it's supposed to work? I'm afraid just looking at the libblockdev code keeps it rather unclear how the password is passed on.

(In reply to David Teigland from comment #11)
>  the problem is that lvresize->lvresize_fs_helper->cryptsetup requires a password input which is not being provided.  When run through busctl/udisks:

(In reply to David Teigland from comment #12)
> It's not clear to me how the crypt password is supposed to be provided when
> Resize is initiated through busctl.

There is no way to provide LUKS password when resizing an LV, because we expect users to do all the resize steps manually, meaning for an encrypted device, users need to resize the filesystem, the LUKS device/format and the backing block device (regardless whether it is a partition or LV) separately. We are not calling lvresize with --resizefs and we don't expect LVM to try to resize anything else than the LV.

> We are not calling lvresize with --resizefs and we don't expect LVM to try to resize anything else than the LV.

Good, I think that's the correct expectation, and that's what the recent fix in bug 2162144 is enforcing: lvresize will not try to resize either crypt or fs on the LV unless --resizefs is set.

So, given that behavior (and including that fix), the busctl Resize command reported above now fails with this error:

$ busctl call org.freedesktop.UDisks2 /org/freedesktop/UDisks2/lvm/test/lv org.freedesktop.UDisks2.LogicalVolume Resize 'ta{sv}' 170000000 0
Call failed: Error resizing logical volume: Process reported exit code 5:   crypt reduce is required (see --resizefs or cryptsetup resize.)

That's because the lvm command was "lvresize --force -L 166015K test/lv", which does not include --resizefs.  The lvresize error message tells us the reason the lvresize failed was because the crypt device on top of the LV has not been correctly reduced, and the lvresize command itself is not allowed to resize the crypt layer unless --resizefs is provided.

So, why is the busctl Resize expected to succeed?  Resizing the LV by itself would produce a mismatch between the LV and crypt device sizes, which isn't a nice result.  Is that what you want?  If so, is there a practical reason for that?  Are you expecting that --force will override this check and force an inconsistent LV/crypt combination?  If so, should we change lvresize so that --force will produce inconsistent LV/crypt layers, or should we change udisks/libblockdev to not expect lvresize to produce the inconsistency?

> Resizing the LV by itself would produce a mismatch between the LV and crypt device sizes, which isn't a nice result.

FTR, that's not what cockpit does. It first calls `org.freedesktop.UDisks2.Encrypted Resize`, and my reproducer here in the description does as well. Of course cockpit will use the same size, the reproducer just generously rounded the numbers and ensured that the LV is bigger than the LUKS container, to make sure that the commands don't fail on some "not enough margin for metadata" issues.

Perhaps this should just be one D-Bus call instead of two in the future?

(In reply to Martin Pitt from comment #16)
> > Resizing the LV by itself would produce a mismatch between the LV and crypt device sizes, which isn't a nice result.
> FTR, that's not what cockpit does. It first calls `org.freedesktop.UDisks2.Encrypted Resize`, and my reproducer here in the description does as well. 

The test is fairly unusual because we have the LV, the crypt device, and the file system, and the test is trying to resize each of them to different sizes.

1. fs is resized to:    155189248 bytes using fsadm
2. crypt is resized to: 160000000 bytes using busctl/udisks command
3. lv is resized to:    171966464 bytes using busctl/udisks command

In step 3, lvresize wants to reduce the crypt device to match the fs size, which means the crypt device still needs to be reduced, but this fails because lvresize was not given --resizefs.  However, the test seems to expect that lvresize will be happy to leave the crypt device alone since it is less than the requested lv size and greater than the existing fs size.  Usually, lvresize is trying to make the sizes of all three (lv/crypt/fs) consistent with each other, so I'm not entirely sure how I want to deal with this case.  It doesn't seem very realistic, or something that we really need to permit.  Without more careful analysis, I'm concerned that allowing this sort of inconsistency among all the sizes could set up some problematic cases.  I will think about this further, but I'm not sure this is an urgent case to support.


Here's exactly what I'm running and explaining in comment 15:

$ lvcreate -L 200M -n lv -y test
  Wiping crypto_LUKS signature on /dev/test/lv.
  Wiping crypto_LUKS signature on /dev/test/lv.
  Logical volume "lv" created.
$ echo 93R4P4pIqAH8 | cryptsetup luksFormat -q /dev/test/lv; echo 93R4P4pIqAH8 | cryptsetup luksOpen /dev/test/lv lvcr
$ mkfs.ext4 /dev/mapper/lvcr
mke2fs 1.46.5 (30-Dec-2021)
Creating filesystem with 188416 1k blocks and 47104 inodes
Filesystem UUID: 9c32765d-b29a-4955-9ee1-b1cf37a82002
Superblock backups stored on blocks: 
        8193, 24577, 40961, 57345, 73729

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done 

$ fsadm -y resize /dev/mapper/lvcr 130M
resize2fs 1.46.5 (30-Dec-2021)
Resizing the filesystem on /dev/mapper/lvcr to 133120 (1k) blocks.
The filesystem on /dev/mapper/lvcr is now 133120 (1k) blocks long.

$ busctl call org.freedesktop.UDisks2 /org/freedesktop/UDisks2/Manager org.freedesktop.UDisks2.Manager EnableModules b true
$ busctl call org.freedesktop.UDisks2  /org/freedesktop/UDisks2/block_devices/dm_2d2 org.freedesktop.UDisks2.Encrypted Resize 'ta{sv}' 160000000 1 passphrase s 93R4P4pIqAH8
$ busctl call org.freedesktop.UDisks2 /org/freedesktop/UDisks2/lvm/test/lv org.freedesktop.UDisks2.LogicalVolume Resize 'ta{sv}' 170000000 0
Call failed: Error resizing logical volume: Process reported exit code 5:   crypt reduce is required (see --resizefs or cryptsetup resize.)

(In reply to David Teigland from comment #15)
> So, why is the busctl Resize expected to succeed?  Resizing the LV by itself
> would produce a mismatch between the LV and crypt device sizes, which isn't
> a nice result.  

We expect that user will resize the LUKS device first by calling the Resize function on the org.freedesktop.UDisks2.Encrypted interface. We know it isn't very user friendly and we have an RFE[1] to provide a function to resize both device and the filesystem (which probably should include the LUKS container too), but that's not implemented yet. AFAICT all users of UDisks that support resizing are dealing with it manually right now.

> Are you expecting that --force will override this check and force
> an inconsistent LV/crypt combination?

No, we use the --force (sometimes twice and usually with --yes too) basically as a default option for all LVM calls to ignore all questions LVM might want to ask. I am not sure if there is a specific reason for adding it to lvresize.


[1] https://github.com/storaged-project/udisks/issues/549

Btw. we had a different issue with the new lvresize behaviour in our tests where we tried to resize an inactive lv, see https://github.com/storaged-project/libblockdev/issues/841 and we will probably add the "--fs ignore" option by default to all lvresize calls which will also "solve" this issue with LUKS.

(In reply to Vojtech Trefny from comment #18)
> (In reply to David Teigland from comment #15)
> > So, why is the busctl Resize expected to succeed?  Resizing the LV by itself
> > would produce a mismatch between the LV and crypt device sizes, which isn't
> > a nice result.  
> 
> We expect that user will resize the LUKS device first by calling the Resize
> function on the org.freedesktop.UDisks2.Encrypted interface.

Yes, that's fine, the question is what should happen if this user manually resized the LUKS device to the "wrong" size?  And how strictly do we define the "wrong" vs "right" size?  In the test above, the user calls the crypt Resize interface, but they use a size that does not match the file system.  Then, they use lvresize on the LV, and lvresize tells them that the crypt resize wasn't good enough (it doesn't match the fs).  So, lvresize returns an error and wants one of the following:

a) include --resizefs in lvresize so that lvresize can make the sizes match if it sees an inconsistency

b) include --fs ignore in lvresize to ignore fs and crypt layers because they know what they're doing

c) manually resize crypt again to make it match the fs and then use the original lvresize (with no --resizefs and no --fs ignore)


If none of those options sound good, then we could loosen the lvresize definition of "right size", and allow the crypt device size to float anywhere between the fs size on the lower end, and the LV size on the upper end.  That looser behavior may be ok, but needs more careful analysis and testing.

Are there any remaining lvm issues here?

FTR, we have not seen this issue on RHEL 9 any more since Feb 23, according to our automatic tracker in https://github.com/cockpit-project/bots/issues/4292

Not sure what closed to state to use, there were changes in lvm and changes in libblockdev, and maybe those changes finally lined up to avoid the problems.