Created attachment 1760395 [details]
Journal with debug enabled

cryptsetup does not touch mounted fs. Let's see if systemd team has an idea... (dm device resize definitely generates change event)

One more hint to the journal:

kernel: dm-1: detected capacity change from 610304 to 405504


This is the event that caused the unmounting of the /run/foo.
This is pressumably wrong, we actually tried to grow the LUKS container, not shrink it. The numbers should be opposite.

I did some more debugging, and the thing that causes systemd to do the unmount is that the SYSTEMD_READY udev property for dm-1 is temporarily set to 0.  As far as systemd is concerned, dm-1 has completed disappeared at that point and it cleans up accordingly.

KERNEL[57.422018] change   /devices/virtual/block/dm-1 (block)
ACTION=change
DEVPATH=/devices/virtual/block/dm-1
SUBSYSTEM=block
RESIZE=1
DEVNAME=/dev/dm-1
DEVTYPE=disk
SEQNUM=2295
MAJOR=253
MINOR=1

UDEV  [57.424746] change   /devices/virtual/block/dm-1 (block)
ACTION=change
DEVPATH=/devices/virtual/block/dm-1
SUBSYSTEM=block
RESIZE=1
DEVNAME=/dev/dm-1
DEVTYPE=disk
SEQNUM=2295
USEC_INITIALIZED=29949362
DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1
DM_UDEV_PRIMARY_SOURCE_FLAG=1
DM_UDEV_RULES_VSN=2
DM_NAME=luks-c9037f79-68b5-4814-b7b9-8cb7f6f916d9
DM_UUID=CRYPT-LUKS1-c9037f7968b54814b7b98cb7f6f916d9-luks-c9037f79-68b5-4814-b7b9-8cb7f6f916d9
DM_SUSPENDED=1
DM_UDEV_DISABLE_OTHER_RULES_FLAG=1
SYSTEMD_READY=0
MAJOR=253
MINOR=1
DEVLINKS=/dev/mapper/luks-c9037f79-68b5-4814-b7b9-8cb7f6f916d9 /dev/disk/by-id/dm-uuid-CRYPT-LUKS1-c9037f7968b54814b7b98cb7f6f916d9-luks-c9037f79-68b5-4814-b7b9-8cb7f6f916d9 /dev/disk/by-id/dm-name-luks-c9037f79-68b5-4814-b7b9-8cb7f6f916d9
TAGS=:systemd:
CURRENT_TAGS=:systemd:

KERNEL[57.427809] change   /devices/virtual/block/dm-1 (block)
ACTION=change
DEVPATH=/devices/virtual/block/dm-1
SUBSYSTEM=block
DM_COOKIE=6342948
DEVNAME=/dev/dm-1
DEVTYPE=disk
SEQNUM=2296
MAJOR=253
MINOR=1

UDEV  [57.505807] change   /devices/virtual/block/dm-1 (block)
ACTION=change
DEVPATH=/devices/virtual/block/dm-1
SUBSYSTEM=block
DM_COOKIE=6342948
DEVNAME=/dev/dm-1
DEVTYPE=disk
SEQNUM=2296
USEC_INITIALIZED=29949362
DM_UDEV_DISABLE_LIBRARY_FALLBACK_FLAG=1
DM_UDEV_PRIMARY_SOURCE_FLAG=1
DM_ACTIVATION=1
DM_NAME=luks-c9037f79-68b5-4814-b7b9-8cb7f6f916d9
DM_UUID=CRYPT-LUKS1-c9037f7968b54814b7b98cb7f6f916d9-luks-c9037f79-68b5-4814-b7b9-8cb7f6f916d9
DM_SUSPENDED=0
DM_UDEV_RULES_VSN=2
ID_FS_LABEL=FSYS
ID_FS_LABEL_ENC=FSYS
ID_FS_UUID=4962827d-782a-4b8a-88c3-80bb5ace0e51
ID_FS_UUID_ENC=4962827d-782a-4b8a-88c3-80bb5ace0e51
ID_FS_VERSION=1.0
ID_FS_TYPE=ext4
ID_FS_USAGE=filesystem
.ID_FS_TYPE_NEW=ext4
MAJOR=253
MINOR=1
DEVLINKS=/dev/disk/by-uuid/4962827d-782a-4b8a-88c3-80bb5ace0e51 /dev/mapper/luks-c9037f79-68b5-4814-b7b9-8cb7f6f916d9 /dev/disk/by-id/dm-name-luks-c9037f79-68b5-4814-b7b9-8cb7f6f916d9 /dev/disk/by-id/dm-uuid-CRYPT-LUKS1-c9037f7968b54814b7b98cb7f6f916d9-luks-c9037f79-68b5-4814-b7b9-8cb7f6f916d9 /dev/disk/by-label/FSYS
TAGS=:systemd:
CURRENT_TAGS=:systemd:

Looking further, I think the initial trigger here is DM_SUSPENDED=1.  Something temporarily suspends dm-1 and sends an event about it.  On Fedora 33, there is no event, and maybe dm-1 doesn't even get temporarily suspended.

From DM_SUSPENDED=1 we get DM_UDEV_DISABLE_OTHER_RULES_FLAG=1 which (I think) skips running blkid, which leaves ID_FS_TYPE empty, which results in SYSTEMD_READY=0 for a DM_UUID that starts with CRYPT-.

In any case, suspending a device mapper device does not warrant unmounting a filesystem on it, imo.  This rule in 99-systemd.rules is probably triggered unintentionally here:

    SUBSYSTEM=="block", ENV{DM_UUID}=="CRYPT-*", ENV{ID_PART_TABLE_TYPE}=="", ENV{ID_FS_USAGE}=="", \
    ENV{SYSTEMD_READY}="0"

(In reply to Marius Vollmer from comment #5)
> Looking further, I think the initial trigger here is DM_SUSPENDED=1. 
> Something temporarily suspends dm-1 and sends an event about it.  On Fedora
> 33, there is no event, and maybe dm-1 doesn't even get temporarily suspended.

Just a remark from DM perspective:

dm-crypt device gets suspended during resize (and always was). It's the usual cycle for changing active DM device table.

1) new table gets loaded into inactive slot (with new size)
2) device gets suspended
3) device gets resumed with new table in active (effective) slot.

If SYSTEMD_READY=0 is set in reaction to device suspend event... it's broken.

(In reply to Ondrej Kozina from comment #6)

> If SYSTEMD_READY=0 is set in reaction to device suspend event... it's broken.

Yeah, I think this was always broken, but we got lucky on Fedora 33 and earlier that no uevent was generated.  Just running `dmsetup suspend ...` does not generate an event on any platform that I have tested, but arguably it should.

(In reply to Ondrej Kozina from comment #6)

> If SYSTEMD_READY=0 is set in reaction to device suspend event... it's broken.

I'd say removing ID_FS_TYPE in reaction to device suspend is broken as well, and is the thing that needs fixing.

(In reply to Marius Vollmer from comment #7)
> Just running `dmsetup suspend ...`
> does not generate an event on any platform that I have tested, but arguably
> it should.

I take that back... Is there any point in running udev rules on a suspended device?  Without being an expert here, what seems to make sense to me is to skip all rule processing for suspended devices (and leave their udev properties unchanged), and run the rules when the device is resumed.  This should remove a lot of complexity, no?

Anyway, you guys figure it out. :-)

This turns out to be hard to reproduce.  I made a reproducer script for bug 1985288, so let me attach it here as well.  The bug is triggered only accasionally, but the script has a commented out section that triggers it reliably (using dmsetup suspend instead of cryptsetup).

Created attachment 1859039 [details]
Reproducer

This still affects every Fedora version since F34. However, as F36 is around the corner, and thus F34 EOL at some point, moving to F35.