Bug 740575

FYI,
In previous lvm2-2.02.87-1.el6.x86_64 I got

[root@white-vdsd ~]# lvs -o +devices 094a369c-6cf2-4edd-aab5-2f50a04d0ce2
LV  VG  Attr   LSize   Origin Snap%  Move Log Copy%  Convert Devices                                          
 6701a412-f2e5-4c74-bd18-ce3b60af2f00 094a369c-6cf2-4edd-aab5-2f50a04d0ce2 -wi---   1.00g                                       /dev/mapper/3600144f076de060000004b122a6c0001(31)
 ids                                  094a369c-6cf2-4edd-aab5-2f50a04d0ce2 -wi-a- 128.00m                                       /dev/mapper/3600144f076de060000004b122a6c0001(20)
 inbox                                094a369c-6cf2-4edd-aab5-2f50a04d0ce2 -wi-a- 128.00m                                       /dev/mapper/3600144f076de060000004b122a6c0001(21)
 leases                               094a369c-6cf2-4edd-aab5-2f50a04d0ce2 -wi-a-   2.00g                                       /dev/mapper/3600144f076de060000004b122a6c0001(4) 
  master                               094a369c-6cf2-4edd-aab5-2f50a04d0ce2 -wi-ao   1.00g                                       /dev/mapper/3600144f076de060000004b122a6c0001(23)
 metadata                             094a369c-6cf2-4edd-aab5-2f50a04d0ce2 -wi-a- 512.00m                                       /dev/mapper/3600144f076de060000004b122a6c0001(0) 
 outbox                               094a369c-6cf2-4edd-aab5-2f50a04d0ce2 -wi-a- 128.00m                                       /dev/mapper/3600144f076de060000004b122a6c0001(22)

*** Bug 740289 has been marked as a duplicate of this bug. ***

Please attach -vvvv traces from both old and new.
Also attach lvm.conf.

Run 'vgscan' before starting each test (so we can eliminate any cache effect).

(Actually, add in the output of those vgscans too i.e. vgscan -vvvv.)

Created attachment 524428 [details]
lvm.conf from rhev-h host (lvm2-2.02.87-2.el6)

Created attachment 524429 [details]
lvm.conf (lvm2-2.02.87-1.el6)

Created attachment 524432 [details]
lvcreate -l1 -vvvv vg_name (lvm2-2.02.87-1.el6)

Looking at the logs, there's been a fallback in .87-2 version when compared to .87-1 and this could result in libudev not containing the information about the symlink (node) under /dev/mapper. Since we read the list of block devices from libudev now (avoiding full /dev scan), this ends up with using "next best" alias to that device, which is exactly what the bug report says.

However, there shouldn't be any change around this code betwen .87-1 and .87-2. 

As for the lvm.conf files - there's no "verify_udev_operations" in 87-1 also I don't see the "obtain_device_list_from_udev" setting there (so a default value of "1" is used for both instead). Was an older lvm.conf from previous version < 87 used?

The udev_sync code is executed properly in both cases and in both versions the "Clearing start of logical volume" is executed just after that which means the node/symlink must be in place at that time. So now why we get the message:

  "/dev/mapper/06a3a92c--c40a--40bc--a213--cb7038145974-lvol1 not set up by udev: Falling back to direct node creation."

..in the .87-2.

Could you please try adding the "verify_udev_operations=1" setting for the .87.1 version (in the "activation" section of lvm.conf). I'm curious if we get the same result now.

(In reply to comment #10)
> As for the lvm.conf files - there's no "verify_udev_operations" in 87-1 also I
> don't see the "obtain_device_list_from_udev" setting there (so a default value
> of "1" is used for both instead). 

Sorry, a default value for "verify_udev_operations" setting is "0". Which makes the problem even more puzzling (because udev must be working fine then - nobody else would create the node in 87-1 and we're clearing it after that so it must exist there).

Even if there was a race and the operation not properly synced with udev in the .87-2 case, the udev must be running, otherwise the synchronization semaphore wouldn't be unlocked and the process would block. And if udev is running and the fallback code done, udev would overwrite existing nodes/symlinks after that and so the record would be in libudev (also, there would be a symlink under /dev/mapper, not a node from the fallback).

Can you also post the /lib/udev/rules.d/*dm-*.rules from the .87-2 install?

(In reply to comment #12)
> Even if there was a race and the operation not properly synced with udev in the
> .87-2 case, the udev must be running, otherwise the synchronization semaphore
> wouldn't be unlocked and the process would block. And if udev is running and
> the fallback code done, udev would overwrite existing nodes/symlinks after that
> and so the record would be in libudev (also, there would be a symlink under
> /dev/mapper, not a node from the fallback).
> 
> Can you also post the /lib/udev/rules.d/*dm-*.rules from the .87-2 install?

Peter, please set needinfo next time to get our attention.
Igor, Mike, please reply with above info

Created attachment 525115 [details]
dm rules files

Note:  This is with lvm2 87-3.  

lvm.conf file is identical to attachment 524428 [details]

Also, in RHEV-H we do have a couple edits in the dm rules (See bug 736486) due to open lvm bugs.

Attachment includes all files matching /lib/udev/rules.d/*dm-*.rules on a running rhev-h system.

(In reply to comment #14)
> Attachment includes all files matching /lib/udev/rules.d/*dm-*.rules on a
> running rhev-h system.

Please, attach also /lib/udev/rules.d/10-dm.rules. Thanks.

Created attachment 525141 [details]
10-dm.rules

(In reply to comment #16)
> Created attachment 525141 [details]
> 10-dm.rules

So it's clear now - there's this line removed from 10-dm.rules:

  ENV{DM_UDEV_DISABLE_DM_RULES_FLAG}!="1", ENV{DM_NAME}=="?*", SYMLINK+="mapper/$env{DM_NAME}"

So we end up with with no symlink created by udev, we do a fallback to direct node creation...

However, if "obtain_device_list_from_udev=1" is used as well, we read the list of block devices (together with all aliases) from udev database. But the /dev/mapper/* content is not in the database now, of course, because that symlink (node) is not created by udev anymore with that line removed.

Was that line removed just because of bug #736486? Or is there any other reason?

(the pvs output just shows another best alias it can use)

(In reply to comment #17)
> Was that line removed just because of bug #736486?

Yes, actually original issue was bug 633222 - duplicate entries under /dev/mapper/
see also bug 736486 comment 10

That 'SYMLINK="mapper/$env{DM_NAME}' line must be there in the 10-dm.rules, we can't misuse the "fallback to direct node creation" in LVM this way. Let's try to solve the "space" problem urgently then (the bug #736486).

As already stated in comment #17 and comment #20, the line creating /dev/mapper symlinks can't be removed without any consequences. If you do so, please also set the "obtain_device_list_from_udev=0" - this way the list of devices (together with all their aliases) will not be read from udev database, but LVM will scan the /dev itself instead.

As for the original problem, see bug #736486 comment #35 for more info.

(In reply to comment #26)
> As already stated in comment #17 and comment #20, the line creating /dev/mapper
> symlinks can't be removed without any consequences.

This old workaround was reverted in RHEV-H but we also had to change default multipath.conf:

defaults {
    getuid_callout "/lib/udev/scsi_id --replace-whitespace --whitelisted --device=/dev/%n"
}

Without --replace-whitespace device-mapper-multipath would create DM devices with spaces in names, if wwwid contained spaces.

So I think we can close this bug then since the problem reported here had its origin in removing the line from 10-dm.rules. If we have this line back now (as stated in bug #736486 comment #37), we should not hit this problem anymore (this problem was a consequence of that change).

As for the original request to support device names with special characters and their encoding, there's still bug #736486 scheduled for 6.3 timeline.