Hello,

Could you please provide us installation logs? You can find them in /tmp/*.log in the installation environment.

as machine reboots after install (but fails to boot), these files aren't left behind on the installed system, right ?

Created attachment 1822059 [details]
anaconda log files on installed system

Anaconda log files from installed system

(In reply to farrotin from comment #2)
> as machine reboots after install (but fails to boot), these files aren't
> left behind on the installed system, right ?

I forgot that on each installed system, anaconda copies files to /var/log/anaconda, so here there are.

Worth knowing that to be able to boot, one has to go to emergency mode, remove /home from /etc/fstab and reboot.

Here is the layout : 

 lsblk 
NAME             MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda                8:0    0 186.3G  0 disk  
├─sda1             8:1    0     1G  0 part  
│ └─md127          9:127  0  1022M  0 raid1 /boot
└─sda2             8:2    0 185.3G  0 part  
  └─md126          9:126  0 185.2G  0 raid1 
    ├─cs_n8-root 253:0    0    70G  0 lvm   /
    └─cs_n8-swap 253:1    0  31.5G  0 lvm   [SWAP]
sdb                8:16   0 186.3G  0 disk  
├─sdb1             8:17   0     1G  0 part  
│ └─md127          9:127  0  1022M  0 raid1 /boot
└─sdb2             8:18   0 185.3G  0 part  
  └─md126          9:126  0 185.2G  0 raid1 
    ├─cs_n8-root 253:0    0    70G  0 lvm   /
    └─cs_n8-swap 253:1    0  31.5G  0 lvm   [SWAP]

cs_n8-home should be there but it's not automatically enabled at the LVM side :

  ACTIVE            '/dev/cs_n8/root' [70.00 GiB] inherit
  inactive          '/dev/cs_n8/home' [2.00 GiB] inherit
  ACTIVE            '/dev/cs_n8/swap' [<31.46 GiB] inherit

So something doesn't correctly set it as required to be activated, and on reboot, systemd doesn't do it but tries to mount ext4 FS (as listed in /etc/fstab) but underlying lv isn't activated

Let's try to activate it

[root@n8 anaconda]# lvchange -a y /dev/cs_n8/home
[root@n8 anaconda]# lvscan 
  ACTIVE            '/dev/cs_n8/root' [70.00 GiB] inherit
  ACTIVE            '/dev/cs_n8/home' [2.00 GiB] inherit
  ACTIVE            '/dev/cs_n8/swap' [<31.46 GiB] inherit

and now lsblk

NAME             MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
sda                8:0    0 186.3G  0 disk  
├─sda1             8:1    0     1G  0 part  
│ └─md127          9:127  0  1022M  0 raid1 /boot
└─sda2             8:2    0 185.3G  0 part  
  └─md126          9:126  0 185.2G  0 raid1 
    ├─cs_n8-root 253:0    0    70G  0 lvm   /
    ├─cs_n8-swap 253:1    0  31.5G  0 lvm   [SWAP]
    └─cs_n8-home 253:2    0     2G  0 lvm   
sdb                8:16   0 186.3G  0 disk  
├─sdb1             8:17   0     1G  0 part  
│ └─md127          9:127  0  1022M  0 raid1 /boot
└─sdb2             8:18   0 185.3G  0 part  
  └─md126          9:126  0 185.2G  0 raid1 
    ├─cs_n8-root 253:0    0    70G  0 lvm   /
    ├─cs_n8-swap 253:1    0  31.5G  0 lvm   [SWAP]
    └─cs_n8-home 253:2    0     2G  0 lvm   

Something missing ? 

Strangely it happens only if home is on lv on top of vg on md device. (and no issue for / being on same vg/md device)
I tried to install it directly on one device (/dev/sda) and so with vg (and root/home LVs) and it then works fine

Yes, those files are copied but also you could use `inst.nokill` to avoid auto-reboot.

Thanks for the logs.

This seems to be a storage related issue. Switching to our storage library for further investigation.

Moving this once more, this time to LVM. I am not sure whether this is LVM or systemd fault but the /home LV is not activated during boot.

Well, /home lv is activated when it's just on a *non* md device (already tested)
so it fails only when there is a md => pv/vg and /home being lv

I'd like to see the journal log from the failing boot.  We may also need to see the journal log when booting with rd.udev.debug.

Created attachment 1823045 [details]
journal logs

including one with rd.udev.debug

Worth knowing that when dropped to emergency, I created log files for journal, and then just 
lvchange -a y /dev/vg_humpty-n12/home && exit

So it quits emergency mode and then continue to boot

It doesn't look like the udev coldplug is generating an event for the md device, so the lvm udev rule is never triggered to activate the VG on the md device.  I wonder if udev knows anything about the md device (udevadm info --export-db).  Need info from systemd udev experts about why we're not seeing the expected event.

Created attachment 1823195 [details]
udevadm info --report-db output

Peter had a number of insights into the issue:

- This looks like a problem with md udev rules in the initrd not handing over udev db state to the root fs.  Are there updated md udev rules for rhel9?

- SYSTEMD_READY=0 in the udev db, but it should be 1 since the md device is ready, and has already been used in the initrd.  That state may not have been transferred from initrd to root fs udev db.

- The md udev rule needs OPTIONS+="db_persist" to hand over the md state.  See https://github.com/systemd/systemd/issues/1551

- To debug this further and observe this problem, we'd want to install and enable the udev monitor service from here:
https://bugzilla.redhat.com/show_bug.cgi?id=1306539#c13

Passing this bz to the md group to help figure out if the md udev rules are missing something.

Reproduced with oVirt Node default layout as well (ISO: https://resources.ovirt.org/pub/ovirt-master-snapshot-static/iso/ovirt-node-ng-installer/4.5.0-2021091610/el9/ovirt-node-ng-installer-4.5.0-2021091610.el9.iso )

Workaround at comment #11 worked for me.

I don't know where to look for the latest version of lvm in rhel9 any more (distgit, centos, brew, compose?)

But, as far as I can tell, a quite old version of lvm (based on release 2.03.12 from May) has been included in all rhel9 installations to date.  To get useful test results from lvm we need to be using something based on 2.03.13.  I don't know when/where/how that version might become part of installations.

That said, I don't think the old version of lvm is necessarily related to the missing activation.

That's a question for the maintainer I guess, but clearly it was pushed to gitlab and built :

https://gitlab.com/redhat/centos-stream/rpms/lvm2
https://kojihub.stream.rdu2.redhat.com/koji/packageinfo?packageID=1200

But all latest builts are gated and so not included in compose, so the one in latest compose for 9-stream seems to be :
lvm2-2.03.12-4.el9.x86_64 (when writing this) :

https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/os/Packages/lvm2-2.03.12-4.el9.x86_64.rpm

(subject to change as a symlink so latest being today https://composes.stream.centos.org/production/CentOS-Stream-9-20210917.0/

very likely this bug is related to:
Bug 1977994 - RHEL 9 FC hosts often boot into maintenance due to issues in udev

Michal, Is there extra we can do to confirm this?


farrotin, that bug is currently in MODIFIED - would be great to test this with the latest package there to be sure.

(In reply to Jonathan Earl Brassow from comment #24)
> 
> 
> farrotin, that bug is currently in MODIFIED - would be great to test this
> with the latest package there to be sure.

I'll ping Brian Stinson to see if we have a Stream9 compose that can be tested : just to ensure, which specific package and NVR are we looking at to be sure ? 
if we're talking about systemd (that provides systemd-udev pkg) the last one is from August : https://kojihub.stream.rdu2.redhat.com/koji/buildinfo?buildID=13694

(In reply to Jonathan Earl Brassow from comment #24)
> very likely this bug is related to:
> Bug 1977994 - RHEL 9 FC hosts often boot into maintenance due to issues in
> udev
> 
> Michal, Is there extra we can do to confirm this?
> 

Netapp tested the packages provided in #1977994 and they've reported that the issue is fixed. Feel free to grab those packages and test if they fix this issue as well.

https://msekleta.fedorapeople.org/bz1977994-systemd-test-rpms/

as it needs a real compose in real CentOS Stream, I'll ask the maintainer to build it for Stream 9 / RHEL 9 and so be available for a composed tree.
Also, can't read (even if Red Hat employee) the other bug that is private, so I'll ask someone with enough rights to let me know what's in that report :)

appears my reply from yesterday didn't go through... reposting here in case it is still useful:

(In reply to farrotin from comment #25)
> (In reply to Jonathan Earl Brassow from comment #24)
> > 
> > 
> > farrotin, that bug is currently in MODIFIED - would be great to test this
> > with the latest package there to be sure.
> 
> I'll ping Brian Stinson to see if we have a Stream9 compose that can be
> tested : just to ensure, which specific package and NVR are we looking at to
> be sure ? 
> if we're talking about systemd (that provides systemd-udev pkg) the last one
> is from August :
> https://kojihub.stream.rdu2.redhat.com/koji/buildinfo?buildID=13694

It's a good question.  The 'fixed in version' is not yet set in bug 1977994 .

For LVM2, we'd like greater than 2.03.13.

Digging around, it seems like systemd >= 249.5

(In reply to farrotin from comment #27)
> as it needs a real compose in real CentOS Stream, I'll ask the maintainer to
> build it for Stream 9 / RHEL 9 and so be available for a composed tree.
> Also, can't read (even if Red Hat employee) the other bug that is private,
> so I'll ask someone with enough rights to let me know what's in that report
> :)

FYI, We consider this to be an urgent bug, but we are waiting to pipeline it for a release for when we know more.  If you have a chance to test - plz let us know if it is still an issue.

CentOS Stream 9 still has systemd-249-4.el9, if the fix is in systemd >= 249-5 can't really test it yet.

(In reply to Sandro Bonazzola from comment #30)
> CentOS Stream 9 still has systemd-249-4.el9, if the fix is in systemd >=
> 249-5 can't really test it yet.

Michal, FYI.

I see that newer systemd was built last week but not yet included in official compose, as currently in c9s-gate tag but there is progress : https://kojihub.stream.centos.org/koji/buildinfo?buildID=14710

(In reply to farrotin from comment #32)
> I see that newer systemd was built last week but not yet included in
> official compose, as currently in c9s-gate tag but there is progress :
> https://kojihub.stream.centos.org/koji/buildinfo?buildID=14710

Checking the link, I see that systemd-249-8.el9 now has the c9s-candidate tag - can you check it now?  We are pretty sure this bug is resolved by this package, tempted to close this bug, but want to hear from reporter.

Back from PTO but no, the official approved package in Stream 9 (today that is) is still older systemd : 

http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/systemd-249-4.el9.x86_64.rpm

Who can consider it stable/good to have it included in stream 9 ? I see based on what was pushed downstream to git.centos.org that a higher systemd version was included in rhel9beta : 
https://git.centos.org/rpms/systemd/commits/c9-beta

@jwboyer can you help getting the fix shipped?

If the fix is in systemd, and you are waiting on the systemd team to include that fix, it would probably be good to have the bug assigned to the systemd component...

Just a quick update : I see that systemd-249-9.el9 version landed in stream 9 and normally was included in compose so it's on mirror.stream.centos.org

But I gave it a quick try and it's failing again : seems installed, anaconda ends and when machine reboots, still no luck : dropped to emergency shell as before

Just another small update : it doesn't seem related to md device as even just having /home on a lv seems to trigger the same issue

How to reproduce : 
- download http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/iso/CentOS-Stream-9-20211207.2-x86_64-dvd1.iso
- setup a VM and disk layout that will use / on lv and /home on other lv (ext4)
reboot

- dropped to emergency mode shell

(In reply to farrotin from comment #42)
> Just another small update : it doesn't seem related to md device as even
> just having /home on a lv seems to trigger the same issue
> 
> How to reproduce : 
> - download
> http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/iso/CentOS-Stream-9-
> 20211207.2-x86_64-dvd1.iso
> - setup a VM and disk layout that will use / on lv and /home on other lv
> (ext4)
> reboot
> 
> - dropped to emergency mode shell

I've done this and it boots fine for me, except that networking fails to be set up.

Could you attach the journalctl output after the failure?

It's worth knowing that it's not in a VM but on two different bare-metal nodes.
I wanted to reboot and interrupted grub to see what was the cmdline that grub was using and I manually added an option and it booted .

Before :

cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-25.el9.x86_64 root=/dev/mapper/cs_host-root ro resume=/dev/mapper/cs_host-swap rd.lvm.lv=cs_host/root rd.lvm.lv=cs_host/swap biosdevname=0 net.ifnames=0


After my edit in grub :

cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-25.el9.x86_64 root=/dev/mapper/cs_host-root ro resume=/dev/mapper/cs_host-swap rd.lvm.lv=cs_host/root rd.lvm.lv=cs_host/home rd.lvm.lv=cs_host/swap biosdevname=0 net.ifnames=0

So now to find what generates the grub config and that doesn't add the `rd.lvm.lv= ` argument to force the lv to be activated on boot

That's a workaround to the problem.  The initrd is not meant to activate all the LVs the system needs to boot.  The home LV should be autoactivated and then mounted after switching to root.  I'm hoping the journal will have some messages that tell us why that failed, but we may also need to enable udev debugging (in /etc/udev/udev.conf) to get more info if the problem precedes lvm.

So, just another comment : testing in a VM is useless, for unknow reason : using same tree, and having /home on dedicated lv in same VG works but not on bare-metal 
Let me try to find some free time to just be able to gather information from the two systems (old private thinkpad at home, and supermicro node in RH DC) where it fails to boot

Created attachment 1845549 [details]
journal log from thinkpad (not using md device)

This is journal log from the thinkpad, not using any md device but still not activating /home LV automatically after a fresh install

Is there anything else you'd need from me ? 
I had a quick look and see in lvm2 changelog this commit : https://gitlab.com/redhat/centos-stream/rpms/lvm2/-/commit/dc2ed04bad5936e8f9fe7e0d8383468ec7c66b50

From journal log, I also see
lvm[726]: /dev/sda2 excluded by filters: device is not in devices file

So wondering if that's where we have to dig

> From journal log, I also see
> lvm[726]: /dev/sda2 excluded by filters: device is not in devices file

Could you paste the content of /etc/lvm/devices/system.devices here?  The lvm devices file is enabled by default in RHEL9, and there are a handful of cases like this not being handled correctly.  There are a couple possible issues:

1. lvm is mishandling the wwid or serial number of sda2.  There's a fix from the past week for some of these cases.  system.devices should show that, but it would also be useful to see:

# cat /sys/block/sda/device/wwid 
# cat /sys/block/sda/device/serial

2. sda2 is missing from system.devices.  Was the machine upgraded to rhel9 or freshly installed?  On a fresh install, anaconda runs vgimportdevices -a which should put all your PVs into system.devices.  On an upgrade from a previous version of RHEL there shouldn't be a system.devices file, which will effectively disable the feature.  So, I'm interested in figuring out what steps led to this system having a system.devices file without sda2 in it.


To fix the problem, you can just disable the devices file with rm /etc/lvm/devices/system.devices.

Or, if the problem is that sda2 is missing from system.devices, you can add it with lvmdevices --adddev /dev/sda2

I just had to remove temporary /home from /etc/fstab to be able to boot and get access.
Here we go :

 cat /etc/lvm/devices/system.devices 
# LVM uses devices listed in this file.
# Created by LVM command vgimportdevices pid 33637 at Thu Dec  9 22:34:34 2021
VERSION=1.1.1
IDTYPE=sys_wwid IDNAME=t10.ATA     BT58SSD05S                              BS27040500004        DEVNAME=/dev/sda2 PVID=2F1B7ntMAyZw6wyBIZ0LUaecEnDuxa7U PART=2

cat /sys/block/sda/device/wwid
t10.ATA     BT58SSD05S                              BS27040500004       

cat /sys/block/sda/device/serial
cat: /sys/block/sda/device/serial: No such file or directory

So no serial for that device it seems but what is strange is that if I try pvs it doesn't see the disk either :
 pvscan 
  Devices file sys_wwid t10.ATA PVID 2F1B7ntMAyZw6wyBIZ0LUaecEnDuxa7U last seen on /dev/sda2 not found.
  No matching physical volumes found

Worth knowing that it's a fresh install, so nothing normally to import

I then tried your suggestion of deleting /etc/lvm/devices/system.devices and reboot and  ... it worked :

 vgs ; pvs ; df -h |grep home
  VG      #PV #LV #SN Attr   VSize  VFree
  cs_host   1   3   0 wz--n- 54.89g    0 
  PV         VG      Fmt  Attr PSize  PFree
  /dev/sda2  cs_host lvm2 a--  54.89g    0 
/dev/mapper/cs_host-home   17G  152M   17G   1% /home

So is that this new option in lvm2 that is causing the issue here ? (https://gitlab.com/redhat/centos-stream/rpms/lvm2/-/commit/dc2ed04bad5936e8f9fe7e0d8383468ec7c66b50#0cd4b04a6aee0f12ee79a4b687a2df8b2640d3e4_176_176)

(In reply to farrotin from comment #50)
>  cat /etc/lvm/devices/system.devices 
> # LVM uses devices listed in this file.
> # Created by LVM command vgimportdevices pid 33637 at Thu Dec  9 22:34:34
> 2021
> VERSION=1.1.1
> IDTYPE=sys_wwid IDNAME=t10.ATA     BT58SSD05S                             
> BS27040500004        DEVNAME=/dev/sda2 PVID=2F1B7ntMAyZw6wyBIZ0LUaecEnDuxa7U
> PART=2
> 
> cat /sys/block/sda/device/wwid
> t10.ATA     BT58SSD05S                              BS27040500004       

Thanks, this is the same issue as bug 2024100 which is fixed here:
https://sourceware.org/git/?p=lvm2.git;a=commit;h=ae54e75176d787de2d447ec40142f85f4dcc47c4

lvm was not expecting spaces in the wwid.  Even though the correct wwid was saved in system.devices, the later comparisons were failing due to the spaces, and lvm thought sda2 was not in the devices file, so it was ignored.

> So is that this new option in lvm2 that is causing the issue here ?
> (https://gitlab.com/redhat/centos-stream/rpms/lvm2/-/commit/
> dc2ed04bad5936e8f9fe7e0d8383468ec7c66b50#0cd4b04a6aee0f12ee79a4b687a2df8b2640
> d3e4_176_176)

Related to some build problems that temporarily disabled the devices file by mistake.

Thanks ..

Does it work differently for now a md device ? the original bug report was about a supermicro node and with 2 ssd disks in raid1 :

pvs
  PV         VG            Fmt  Attr PSize    PFree  
  /dev/md126 vg_humpty-n12 lvm2 a--  <185.18g 171.41g


cat /etc/lvm/devices/system.devices 
# LVM uses devices listed in this file.
# Created by LVM command lvs pid 1394 at Fri Dec 10 08:56:56 2021
VERSION=1.1.2
IDTYPE=md_uuid IDNAME=e5b5940d-7299-3234-860d-134a7baa7459 DEVNAME=/dev/md126 PVID=uDKqeOfmqxtR98ykUAkAF4IeV8vM6B3O

cat /sys/block/md126/md/uuid 
e5b5940d-7299-3234-860d-134a7baa7459

on that server I tried the previous "fix" : "/bin/rm /etc/lvm/devices/system.devices  && systemctl reboot" but it doesn't boot properly and it's dropped to emergency shell as before, so it's a different issue here now
Wondering if you prefer having another one or track all in this BZ ? (it was initially for this supermicro device with /home on md device and same symptom as before : lv not activated)

(In reply to David Teigland from comment #51)
> 
> 
> Thanks, this is the same issue as bug 2024100 which is fixed here:
> https://sourceware.org/git/?p=lvm2.git;a=commit;
> h=ae54e75176d787de2d447ec40142f85f4dcc47c4
> 

Also, that bug is private so impossible to have a look at it, nor if we have to expect a newer lvm2 package bump anytime soon in Stream 9

(In reply to farrotin from comment #52)
> Thanks ..
> 
> Does it work differently for now a md device ? the original bug report was
> about a supermicro node and with 2 ssd disks in raid1 :
> 
> pvs
>   PV         VG            Fmt  Attr PSize    PFree  
>   /dev/md126 vg_humpty-n12 lvm2 a--  <185.18g 171.41g
> 
> 
> cat /etc/lvm/devices/system.devices 
> # LVM uses devices listed in this file.
> # Created by LVM command lvs pid 1394 at Fri Dec 10 08:56:56 2021
> VERSION=1.1.2
> IDTYPE=md_uuid IDNAME=e5b5940d-7299-3234-860d-134a7baa7459
> DEVNAME=/dev/md126 PVID=uDKqeOfmqxtR98ykUAkAF4IeV8vM6B3O
> 
> cat /sys/block/md126/md/uuid 
> e5b5940d-7299-3234-860d-134a7baa7459

That lvm devices config looks correct for the md device, so the bug in comment 51 would not apply here.

> on that server I tried the previous "fix" : "/bin/rm
> /etc/lvm/devices/system.devices  && systemctl reboot" but it doesn't boot
> properly and it's dropped to emergency shell as before, so it's a different
> issue here now
> Wondering if you prefer having another one or track all in this BZ ? (it was
> initially for this supermicro device with /home on md device and same
> symptom as before : lv not activated)

The original subject of this bug still seems to be a problem.  For a time there was a suspicion that it was caused by systemd, but it seems like the systemd update has not fixed the problem.  This gets us back to my original analysis about incomplete udev state for the md device.  The incomplete or incorrect udev processing of the md device means that lvm is not asked to activate the device, which means the home LV is never activated, leading to the timeout.  I wonder if there is more info or debugging that the md developers could use since this is very reproducable?

There are three separate issues contained in this bz:
1. the original failure to activate a VG from an md device.
2. the systemd bug.
3. the lvm wwid mishandling.

Let's ignore issue 3, there's another bz for it.

This bz has been used to process issue 2, even though it seems to be unrelated to issue 2.  I don't know if we can separate this bz from issue 2 at this point.

So, perhaps we should create new bug for the original topic of this bug.  File it against md for now since md seems to be the lowest level at which we currently see problems.

well, I'd say that as original issue in this BZ is still there, let's keep it open and probably no need to open another one for the lvm "spaces in wwid" as it seems there is already a patch and BZ for it (can't open it but seems related as you pointed to it)

So should this BZ be moved to another category (like 'md') ?

I think the next stage in debugging this is to follow Peter's recommendation in https://bugzilla.redhat.com/show_bug.cgi?id=1306539#c13, so I'm copying the key part of that comment here:

- take the special systemd unit to monitor udev during boot, available here: http://people.redhat.com/~prajnoha/lvm2/rpms/RHEL7/bz1306539/systemd-udev-monitor.service.service

- place systemd-udev-monitor.service file in /etc/systemd/system

- call "systemctl daemon-reload"

- call "systemctl enable systemd-udev-monitor.service"

- reboot

- append "systemd.log_level=debug systemd.log_target=kmsg udev.log-priority=debug log_buf_len=8M" to kernel command line

- wait for the failure and timeout to happen

- take the journalctl output from the boot

- take the /run/udev_monitor.log file that got generated by that systemd-udev-monitor.service

- call "systemctl disable systemd-udev-monitor.service" and "rm /etc/systemd/system/systemd-udev-monitor.service" for cleanup

(In reply to David Teigland from comment #56)
> I think the next stage in debugging this is to follow Peter's recommendation
> in https://bugzilla.redhat.com/show_bug.cgi?id=1306539#c13, so I'm copying
> the key part of that comment here:

These steps don't seem to work correctly any more, they have probably become a little outdated.  I'm working on updating them, and in the process I might have reproduced the original md activation issue.

I'm reproducing the same or similar boot failure with root on thin LVs and root VG on md raid1.  The udev monitor service includes the same info that we collected previously, which shows SYSTEMD_READY=0.  I'm not really sure what we're looking for from the udev-monitor service.

UDEV  [14.358770] add      /devices/virtual/block/md1 (block)
ACTION=add
DEVLINKS=/dev/md/1 /dev/disk/by-id/md-uuid-1f1ae97e:91e31d11:b51d73a1:71ef886e /dev/disk/by-id/md-name-localhost.localdomain:1 /dev/disk/by-id/lvm-pv-uuid-8pUxt1-MsNo-dhLa-DouQ-FfDo-FbEL-vanOrn
DEVNAME=/dev/md1
DEVPATH=/devices/virtual/block/md1
DEVTYPE=disk
ID_FS_TYPE=LVM2_member
ID_FS_USAGE=raid
ID_FS_UUID=8pUxt1-MsNo-dhLa-DouQ-FfDo-FbEL-vanOrn
ID_FS_UUID_ENC=8pUxt1-MsNo-dhLa-DouQ-FfDo-FbEL-vanOrn
ID_FS_VERSION=LVM2 001
MAJOR=9
MD_DEVICES=2
MD_DEVNAME=1
MD_LEVEL=raid1
MD_METADATA=1.2
MD_NAME=localhost.localdomain:1
MD_UUID=1f1ae97e:91e31d11:b51d73a1:71ef886e
MINOR=1
SEQNUM=1989
SUBSYSTEM=block
SYNTH_UUID=0
SYSTEMD_READY=0
SYSTEMD_WANTS=mdmonitor.service
TAGS=:systemd:
USEC_INITIALIZED=3649545


It looks like the systemd-udev-trigger service run in the initrd is triggering uevents, but after switching to root, systemd-udev-trigger seems to do nothing.  I would expect to see udev-related log messages after udev-trigger in the root fs.  Here's the section of the logs where nothing shows up:

Dec 13 21:22:42 vm3 systemd[978]: Operating on architecture: x86-64
Dec 13 21:22:42 vm3 systemd[978]: systemd-journald.service: Executing: /usr/lib/systemd/systemd-journald
Dec 13 21:22:42 vm3 systemd[982]: systemd-sysctl.service: Executing: /usr/lib/systemd/systemd-sysctl
Dec 13 21:22:42 vm3 systemd[983]: kmod-static-nodes.service: Executing: /usr/bin/kmod static-nodes --format=tmpfiles --output=/run/tmpfiles.d/kmod.conf
Dec 13 21:22:42 vm3 systemd[984]: dev-hugepages.mount: Executing: /usr/bin/mount hugetlbfs /dev/hugepages -t hugetlbfs
Dec 13 21:22:42 vm3 systemd[985]: systemd-remount-fs.service: Executing: /usr/lib/systemd/systemd-remount-fs
Dec 13 21:22:42 vm3 systemd[986]: dev-mqueue.mount: Executing: /usr/bin/mount mqueue /dev/mqueue -t mqueue
Dec 13 21:22:42 vm3 systemd[987]: systemd-udev-monitor.service: Executing: /usr/bin/sh -c '/usr/sbin/udevadm monitor --udev --env > /run/udev/udev_monitor.log'
Dec 13 21:22:42 vm3 systemd[988]: systemd-udev-trigger.service: Executing: /usr/bin/udevadm trigger --type=subsystems --action=add
Dec 13 21:22:42 vm3 systemd[989]: lvm2-monitor.service: Executing: /usr/sbin/lvm vgchange --monitor y
Dec 13 21:22:42 vm3 udevadm[988]:   device-enumerator: scanning /sys/bus
Dec 13 21:22:42 vm3 systemd[990]: systemd-udev-trigger.service: Executing: /usr/bin/udevadm trigger --type=devices --action=add
Dec 13 21:22:42 vm3 systemd-journald[978]: Journal started
Dec 13 21:22:42 vm3 systemd-journald[978]: Runtime journal (/run/log/journal/302e92ae52b3421b9506bdbe0d9f2a49) is 8.0M, max 90.8M, 82.8M free.
Dec 13 21:22:42 vm3 udevadm[990]: device-enumerator: scan all dirs
Dec 13 21:22:42 vm3 udevadm[990]:   device-enumerator: scanning /sys/bus
Dec 13 21:22:42 vm3 udevadm[990]:   device-enumerator: scanning /sys/class
Dec 13 21:22:42 vm3 systemd-remount-fs[985]: Remounting /
Dec 13 21:22:42 vm3 systemd-remount-fs[985]: Successfully forked off '(remount)' as PID 993.
Dec 13 21:22:42 vm3 systemd-remount-fs[985]: Remounting /usr
Dec 13 21:22:42 vm3 systemd-remount-fs[985]: Successfully forked off '(remount)' as PID 994.

Following up on the suggestion that the udev state may not be transferred from the initrd to the root fs, I think that the proper setting for this is being used:

# grep db_persist /lib/dracut/modules.d/90mdraid/*
/lib/dracut/modules.d/90mdraid/59-persistent-storage-md.rules:OPTIONS+="db_persist"

And I think that this might be the udev db state that is saved from the initrd that is passed to the root fs:

# cat /run/udev/data/b9:1
S:md/1
S:disk/by-id/lvm-pv-uuid-8pUxt1-MsNo-dhLa-DouQ-FfDo-FbEL-vanOrn
S:disk/by-id/md-name-localhost.localdomain:1
S:disk/by-id/md-uuid-1f1ae97e:91e31d11:b51d73a1:71ef886e
L:100
W:7
I:6936788
E:MD_LEVEL=raid1
E:MD_DEVICES=2
E:MD_METADATA=1.2
E:MD_UUID=1f1ae97e:91e31d11:b51d73a1:71ef886e
E:MD_DEVNAME=1
E:MD_NAME=localhost.localdomain:1
E:ID_FS_UUID=8pUxt1-MsNo-dhLa-DouQ-FfDo-FbEL-vanOrn
E:ID_FS_UUID_ENC=8pUxt1-MsNo-dhLa-DouQ-FfDo-FbEL-vanOrn
E:ID_FS_VERSION=LVM2 001
E:ID_FS_TYPE=LVM2_member
E:ID_FS_USAGE=raid
E:SYSTEMD_WANTS=mdmonitor.service
E:SYSTEMD_READY=0
G:systemd

This shows that SYSTEMD_READY=0 even in the initrd.  I don't know how SYSTEMD_READY is supposed to become 1, but perhaps lvm using md1 in the initrd is interfering with udev processing md1.  This mdmonitor message appears along with other md1 messages, and is the only thing that stands out to me as potentially looking unusual:

systemd[1]: mdmonitor.service: Failed to load configuration: No such file or directory

I'm hoping that Peter might spot something out of place in these messages.

(In reply to David Teigland from comment #59)
> # cat /run/udev/data/b9:1
> S:md/1
> S:disk/by-id/lvm-pv-uuid-8pUxt1-MsNo-dhLa-DouQ-FfDo-FbEL-vanOrn
> S:disk/by-id/md-name-localhost.localdomain:1
> S:disk/by-id/md-uuid-1f1ae97e:91e31d11:b51d73a1:71ef886e
> L:100
> W:7
> I:6936788
> E:MD_LEVEL=raid1
> E:MD_DEVICES=2
> E:MD_METADATA=1.2
> E:MD_UUID=1f1ae97e:91e31d11:b51d73a1:71ef886e
> E:MD_DEVNAME=1
> E:MD_NAME=localhost.localdomain:1
> E:ID_FS_UUID=8pUxt1-MsNo-dhLa-DouQ-FfDo-FbEL-vanOrn
> E:ID_FS_UUID_ENC=8pUxt1-MsNo-dhLa-DouQ-FfDo-FbEL-vanOrn
> E:ID_FS_VERSION=LVM2 001
> E:ID_FS_TYPE=LVM2_member
> E:ID_FS_USAGE=raid
> E:SYSTEMD_WANTS=mdmonitor.service
> E:SYSTEMD_READY=0
> G:systemd
> 

(So this is the situation where we have MD activated in initrd, the we switch-root, then we have the udev-trigger, right?)


Actually, I'm missing one important variable here - the LVM_MD_PV_ACTIVATED=1.

Just like we have DM and its activation steps "create (add event) + table load + resume (change event)", we have similar situation with MD. That also has "create" and "change" events in sequence for activation and only after that "change" the device is ready to use.

The LVM_MD_PV_ACTIVATED is a helper variable for us to know that the full activation for the MD (on which there's a PV label) has already been done (so already passing the activation's "add + change" sequence) and we can consider any other subsequent "add" events as triggers for a refresh where we should act (not ignore the "add" because we still expect the device's initialization to finish). And such "add" event we need to be able to recognize is exactly the one coming from the systemd-udev-trigger after switch-root.

So far, it looks like the LVM_MD_PV_ACTIVATED=1 has not been set in initrd at all, which is wrong. Now, looking at the exact 69-dm-lvm.rules, we have:

  ACTION=="change", ENV{LVM_MD_PV_ACTIVATED}!="1", TEST=="md/array_state", ENV{LVM_MD_PV_ACTIVATED}="1", GOTO="lvm_scan"

That means, for the LVM_MD_PV_ACTIVATED=1 to be set in initrd, the /sys/<dev>/md/array_state must be present there (which it usually is after full MD activatation). So that is one thing to inspect - why that LVM_MD_PV_ACTIVATED=1 is not set. I'm not quite sure right now why it failed to do so. I don't expect the md/array_state to be missing, unless there's a race I'm not aware of right now...

===

The SYSTEMD_READY=0 looks like a simple consequece of the previous issue, the 69-dm-lvm.rules has:

  ENV{LVM_MD_PV_ACTIVATED}!="1", ENV{SYSTEMD_READY}="0"

The reason we're setting the SYSTEMD_READY there is because we needed the switch from SYSTEMD_READY=0 to SYSTEMD_READY=1 and setting SYSTEMD_WANTS="lvm2-pvscan@<major>:<minor>.service" to happen at once, otherwise systemd would not instantiate the service based on the SYSTEMD_WANTS directive. And MD rules did not handle SYSTEMD_READY at all (at least not at the time when 69-dm-...rules were written). However, it's true we don't need to care about SYSTEMD_READY now when we removed the SYSTEMD_WANTS line and replaced by direct "pvscan" call and "systemd-run lvm vgchange -aay" call. Instead, we should probably just let MD rules set the SYSTEMD_READY when it considers its own devices as being ready and we shouldn't be setting that for them in LVM rules. But I think switching from SYSTEMD_READY=0 to SYSTEMD_READY=1 at correct time is still important for other things like mount point dependencies and similar (...we don't want others to consider the device as ready if it hasn't finished its initialization yet - just like we don't want others to access DM device when the tables has not been loaded and activated yet - we surely do this for DM, the same should MD do).

Interesting read .. let me know if you want me to also have a look at the supermicro server in that situation but it seems that you at least found where to look :)

Well, wait. I've just realized we don't have 69-dm-lvm.rules in initrd, so if the MD device is activated in initrd, the LVM_MD_PV_ACTIVATED=1 won't be set. Then when we switch-root and udev-trigger (with "add" event), then the LVM_MD_PV_ACTIVATED=1 is not set from initrd of course and hence this line is always hit:

 ENV{LVM_MD_PV_ACTIVATED}!="1", ENV{SYSTEMD_READY}="0"

So it seems the rules are a bit wrong in the end... Not counting with switch-root from initrd where the 69-dm-lvm.rules rules are not installed. A fix would be to have this kind of rule in initrd:

 KERNEL=="md[0-9]*", ACTION=="change", ENV{ID_FS_TYPE}=="LVM2_member", ENV{LVM_MD_PV_ACTIVATED}!="1", TEST=="md/array_state", ENV{LVM_MD_PV_ACTIVATED}="1"

(Or simply have the 69-dm-lvm.rules installed in initrd but without the PV scanning and LVM activation part, depending on whether we've already moved to LVM's autoactivation in initrd instead of dracut's custom hooks...)

(In reply to Peter Rajnoha from comment #64)
> fix would be to have this kind of rule in initrd:
> 
>  KERNEL=="md[0-9]*", ACTION=="change", ENV{ID_FS_TYPE}=="LVM2_member",
> ENV{LVM_MD_PV_ACTIVATED}!="1", TEST=="md/array_state",
> ENV{LVM_MD_PV_ACTIVATED}="1"

Thanks! adding that line to the lvm dracut udev rule made it work for me.  The edited rule looks like:

SUBSYSTEM!="block", GOTO="lvm_end"
ACTION!="add|change", GOTO="lvm_end"
KERNEL=="md[0-9]*", ACTION=="change", ENV{ID_FS_TYPE}=="LVM2_member", ENV{LVM_MD_PV_ACTIVATED}!="1", TEST=="md/array_state", ENV{LVM_MD_PV_ACTIVATED}="1"
# Also don't process disks that are slated to be a multipath device
ENV{DM_MULTIPATH_DEVICE_PATH}=="1", GOTO="lvm_end"
KERNEL=="dm-[0-9]*", ACTION=="add", GOTO="lvm_end"
ENV{ID_FS_TYPE}!="LVM?_member", GOTO="lvm_end"

PROGRAM=="/bin/sh -c 'for i in $sys/$devpath/holders/dm-[0-9]*; do [ -e $$i ] && exit 0; done; exit 1;' ", \
    GOTO="lvm_end"

RUN+="/sbin/initqueue --settled --onetime --unique /sbin/lvm_scan"
RUN+="/sbin/initqueue --timeout --name 51-lvm_scan --onetime --unique /sbin/lvm_scan --partial"
RUN+="/bin/sh -c '>/tmp/.lvm_scan-%k;'"

LABEL="lvm_end"


Fabian, could you edit /lib/dracut/modules.d/90lvm/64-lvm.rules on your system to match that?  I expect you only need to add the one line from Peter, and rebuild your initrd (e.g. dracut --force --kver ...)

I confirm it now works with the following change : 

diff -Naur /lib/dracut/modules.d/90lvm/64-lvm.rules.orig /lib/dracut/modules.d/90lvm/64-lvm.rules
--- /lib/dracut/modules.d/90lvm/64-lvm.rules.orig	2021-12-15 10:33:08.487111849 +0000
+++ /lib/dracut/modules.d/90lvm/64-lvm.rules	2021-12-15 10:34:00.676152009 +0000
@@ -6,6 +6,9 @@
 
 SUBSYSTEM!="block", GOTO="lvm_end"
 ACTION!="add|change", GOTO="lvm_end"
+KERNEL=="md[0-9]*", ACTION=="change", ENV{ID_FS_TYPE}=="LVM2_member",
+ENV{LVM_MD_PV_ACTIVATED}!="1", TEST=="md/array_state",
+ENV{LVM_MD_PV_ACTIVATED}="1"
 # Also don't process disks that are slated to be a multipath device
 ENV{DM_MULTIPATH_DEVICE_PATH}=="1", GOTO="lvm_end"
 KERNEL=="dm-[0-9]*", ACTION=="add", GOTO="lvm_end"

Just followed by a "dracut --force && systemctl reboot" and it boots fine now

That file is coming from 'dracut' package :
rpm -qf /lib/dracut/modules.d/90lvm/64-lvm.rules
dracut-055-10.git20210824.el9.x86_64

So worth reassigning this bug to dracut instead of systemd ?

> So worth reassigning this bug to dracut instead of systemd ?

Thanks for testing, I created a new bug 2032993 against dracut and referenced this bug.

FWIW, as there was no momentum on this ticket for some time, I gave it another try in the CentOS infra, and confirm that bug is still there, meaning no way to deploy CentOS Stream 9 with just two disks on md device and also /home as dedicated lv on top

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (new packages: systemd), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:3979