Bug 740575 - Wrong device's path name from 'lvs -o +devices'
Summary: Wrong device's path name from 'lvs -o +devices'
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: lvm2
Version: 6.2
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: rc
: ---
Assignee: Peter Rajnoha
QA Contact: Cluster QE
URL:
Whiteboard:
Depends On: 736486
Blocks: 682015 731472
TreeView+ depends on / blocked
 
Reported: 2011-09-22 14:19 UTC by Igor Lvovsky
Modified: 2013-03-01 04:54 UTC (History)
20 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2011-10-19 09:11:08 UTC


Attachments (Terms of Use)
lvcreate -l1 -vvvv vg_name (rhev-h lvm2-2.02.87-2.el6) (102.88 KB, text/plain)
2011-09-22 14:19 UTC, Igor Lvovsky
no flags Details
lvm.conf from rhev-h host (lvm2-2.02.87-2.el6) (28.01 KB, text/plain)
2011-09-22 15:11 UTC, Igor Lvovsky
no flags Details
lvm.conf (lvm2-2.02.87-1.el6) (21.26 KB, text/plain)
2011-09-22 15:12 UTC, Igor Lvovsky
no flags Details
lvcreate -l1 -vvvv vg_name (lvm2-2.02.87-1.el6) (113.46 KB, text/plain)
2011-09-22 15:21 UTC, Igor Lvovsky
no flags Details
dm rules files (1.30 KB, application/gzip)
2011-09-27 12:48 UTC, Mike Burns
no flags Details
10-dm.rules (6.29 KB, text/plain)
2011-09-27 14:20 UTC, Mike Burns
no flags Details

Description Igor Lvovsky 2011-09-22 14:19:47 UTC
Created attachment 524419 [details]
lvcreate -l1 -vvvv vg_name (rhev-h  lvm2-2.02.87-2.el6)

Description of problem:
The problem occurred on rhev-h host  with lvm2-2.02.87-2.el6.
'lvs -o devices' returned devices through  /dev/disk/by-id.

[root@virtlab103 ~]# lvs -o lv_name,devices
  LV       Devices                                                     
  lvol0    /dev/disk/by-id/dm-name-3600c0ff000d795576663ff4901000000(4)
  lvol1    /dev/disk/by-id/dm-name-3600c0ff000d795576663ff4901000000(5)
  metadata /dev/disk/by-id/dm-name-3600c0ff000d795576663ff4901000000(0)
  metadata /dev/dm-1(0)                                                
  Config   /dev/dm-7(4295)                                             
  Data     /dev/dm-7(4809)                                             
  Logging  /dev/dm-7(4297)                                             
  Swap     /dev/dm-7(0)                                                
  metadata /dev/disk/by-id/dm-name-3600c0ff000d795576663ff4902000000(0)



Version-Release number of selected component (if applicable):
lvm2-2.02.87-2.el6

How reproducible:


Steps to Reproduce:
1.
2.
3.
  
Actual results:


Expected results:


Additional info:

Comment 1 Igor Lvovsky 2011-09-22 14:24:52 UTC
FYI,
In previous lvm2-2.02.87-1.el6.x86_64 I got

[root@white-vdsd ~]# lvs -o +devices 094a369c-6cf2-4edd-aab5-2f50a04d0ce2
LV  VG  Attr   LSize   Origin Snap%  Move Log Copy%  Convert Devices                                          
 6701a412-f2e5-4c74-bd18-ce3b60af2f00 094a369c-6cf2-4edd-aab5-2f50a04d0ce2 -wi---   1.00g                                       /dev/mapper/3600144f076de060000004b122a6c0001(31)
 ids                                  094a369c-6cf2-4edd-aab5-2f50a04d0ce2 -wi-a- 128.00m                                       /dev/mapper/3600144f076de060000004b122a6c0001(20)
 inbox                                094a369c-6cf2-4edd-aab5-2f50a04d0ce2 -wi-a- 128.00m                                       /dev/mapper/3600144f076de060000004b122a6c0001(21)
 leases                               094a369c-6cf2-4edd-aab5-2f50a04d0ce2 -wi-a-   2.00g                                       /dev/mapper/3600144f076de060000004b122a6c0001(4) 
  master                               094a369c-6cf2-4edd-aab5-2f50a04d0ce2 -wi-ao   1.00g                                       /dev/mapper/3600144f076de060000004b122a6c0001(23)
 metadata                             094a369c-6cf2-4edd-aab5-2f50a04d0ce2 -wi-a- 512.00m                                       /dev/mapper/3600144f076de060000004b122a6c0001(0) 
 outbox                               094a369c-6cf2-4edd-aab5-2f50a04d0ce2 -wi-a- 128.00m                                       /dev/mapper/3600144f076de060000004b122a6c0001(22)

Comment 2 Igor Lvovsky 2011-09-22 14:29:32 UTC
*** Bug 740289 has been marked as a duplicate of this bug. ***

Comment 3 Alasdair Kergon 2011-09-22 14:30:18 UTC
Please attach -vvvv traces from both old and new.
Also attach lvm.conf.

Comment 4 Alasdair Kergon 2011-09-22 14:31:43 UTC
Run 'vgscan' before starting each test (so we can eliminate any cache effect).

Comment 5 Alasdair Kergon 2011-09-22 14:32:34 UTC
(Actually, add in the output of those vgscans too i.e. vgscan -vvvv.)

Comment 7 Igor Lvovsky 2011-09-22 15:11:50 UTC
Created attachment 524428 [details]
lvm.conf from rhev-h host (lvm2-2.02.87-2.el6)

Comment 8 Igor Lvovsky 2011-09-22 15:12:54 UTC
Created attachment 524429 [details]
lvm.conf (lvm2-2.02.87-1.el6)

Comment 9 Igor Lvovsky 2011-09-22 15:21:27 UTC
Created attachment 524432 [details]
lvcreate -l1 -vvvv vg_name (lvm2-2.02.87-1.el6)

Comment 10 Peter Rajnoha 2011-09-26 09:33:55 UTC
Looking at the logs, there's been a fallback in .87-2 version when compared to .87-1 and this could result in libudev not containing the information about the symlink (node) under /dev/mapper. Since we read the list of block devices from libudev now (avoiding full /dev scan), this ends up with using "next best" alias to that device, which is exactly what the bug report says.

However, there shouldn't be any change around this code betwen .87-1 and .87-2. 

As for the lvm.conf files - there's no "verify_udev_operations" in 87-1 also I don't see the "obtain_device_list_from_udev" setting there (so a default value of "1" is used for both instead). Was an older lvm.conf from previous version < 87 used?

The udev_sync code is executed properly in both cases and in both versions the "Clearing start of logical volume" is executed just after that which means the node/symlink must be in place at that time. So now why we get the message:

  "/dev/mapper/06a3a92c--c40a--40bc--a213--cb7038145974-lvol1 not set up by udev: Falling back to direct node creation."

..in the .87-2.

Could you please try adding the "verify_udev_operations=1" setting for the .87.1 version (in the "activation" section of lvm.conf). I'm curious if we get the same result now.

Comment 11 Peter Rajnoha 2011-09-26 09:41:19 UTC
(In reply to comment #10)
> As for the lvm.conf files - there's no "verify_udev_operations" in 87-1 also I
> don't see the "obtain_device_list_from_udev" setting there (so a default value
> of "1" is used for both instead). 

Sorry, a default value for "verify_udev_operations" setting is "0". Which makes the problem even more puzzling (because udev must be working fine then - nobody else would create the node in 87-1 and we're clearing it after that so it must exist there).

Comment 12 Peter Rajnoha 2011-09-26 09:52:48 UTC
Even if there was a race and the operation not properly synced with udev in the .87-2 case, the udev must be running, otherwise the synchronization semaphore wouldn't be unlocked and the process would block. And if udev is running and the fallback code done, udev would overwrite existing nodes/symlinks after that and so the record would be in libudev (also, there would be a symlink under /dev/mapper, not a node from the fallback).

Can you also post the /lib/udev/rules.d/*dm-*.rules from the .87-2 install?

Comment 13 Ayal Baron 2011-09-26 22:05:02 UTC
(In reply to comment #12)
> Even if there was a race and the operation not properly synced with udev in the
> .87-2 case, the udev must be running, otherwise the synchronization semaphore
> wouldn't be unlocked and the process would block. And if udev is running and
> the fallback code done, udev would overwrite existing nodes/symlinks after that
> and so the record would be in libudev (also, there would be a symlink under
> /dev/mapper, not a node from the fallback).
> 
> Can you also post the /lib/udev/rules.d/*dm-*.rules from the .87-2 install?

Peter, please set needinfo next time to get our attention.
Igor, Mike, please reply with above info

Comment 14 Mike Burns 2011-09-27 12:48:59 UTC
Created attachment 525115 [details]
dm rules files

Note:  This is with lvm2 87-3.  

lvm.conf file is identical to attachment 524428 [details]

Also, in RHEV-H we do have a couple edits in the dm rules (See bug 736486) due to open lvm bugs.

Attachment includes all files matching /lib/udev/rules.d/*dm-*.rules on a running rhev-h system.

Comment 15 Peter Rajnoha 2011-09-27 14:07:46 UTC
(In reply to comment #14)
> Attachment includes all files matching /lib/udev/rules.d/*dm-*.rules on a
> running rhev-h system.

Please, attach also /lib/udev/rules.d/10-dm.rules. Thanks.

Comment 16 Mike Burns 2011-09-27 14:20:25 UTC
Created attachment 525141 [details]
10-dm.rules

Comment 17 Peter Rajnoha 2011-09-27 18:11:04 UTC
(In reply to comment #16)
> Created attachment 525141 [details]
> 10-dm.rules

So it's clear now - there's this line removed from 10-dm.rules:

  ENV{DM_UDEV_DISABLE_DM_RULES_FLAG}!="1", ENV{DM_NAME}=="?*", SYMLINK+="mapper/$env{DM_NAME}"

So we end up with with no symlink created by udev, we do a fallback to direct node creation...

However, if "obtain_device_list_from_udev=1" is used as well, we read the list of block devices (together with all aliases) from udev database. But the /dev/mapper/* content is not in the database now, of course, because that symlink (node) is not created by udev anymore with that line removed.

Was that line removed just because of bug #736486? Or is there any other reason?

Comment 18 Peter Rajnoha 2011-09-27 18:11:45 UTC
(the pvs output just shows another best alias it can use)

Comment 19 Alan Pevec 2011-09-27 18:27:45 UTC
(In reply to comment #17)
> Was that line removed just because of bug #736486?

Yes, actually original issue was bug 633222 - duplicate entries under /dev/mapper/
see also bug 736486 comment 10

Comment 20 Peter Rajnoha 2011-09-28 02:42:14 UTC
That 'SYMLINK="mapper/$env{DM_NAME}' line must be there in the 10-dm.rules, we can't misuse the "fallback to direct node creation" in LVM this way. Let's try to solve the "space" problem urgently then (the bug #736486).

Comment 26 Peter Rajnoha 2011-10-18 11:04:57 UTC
As already stated in comment #17 and comment #20, the line creating /dev/mapper symlinks can't be removed without any consequences. If you do so, please also set the "obtain_device_list_from_udev=0" - this way the list of devices (together with all their aliases) will not be read from udev database, but LVM will scan the /dev itself instead.

As for the original problem, see bug #736486 comment #35 for more info.

Comment 27 Alan Pevec 2011-10-18 12:57:25 UTC
(In reply to comment #26)
> As already stated in comment #17 and comment #20, the line creating /dev/mapper
> symlinks can't be removed without any consequences.

This old workaround was reverted in RHEV-H but we also had to change default multipath.conf:

defaults {
    getuid_callout "/lib/udev/scsi_id --replace-whitespace --whitelisted --device=/dev/%n"
}

Without --replace-whitespace device-mapper-multipath would create DM devices with spaces in names, if wwwid contained spaces.

Comment 28 Peter Rajnoha 2011-10-19 09:11:08 UTC
So I think we can close this bug then since the problem reported here had its origin in removing the line from 10-dm.rules. If we have this line back now (as stated in bug #736486 comment #37), we should not hit this problem anymore (this problem was a consequence of that change).

As for the original request to support device names with special characters and their encoding, there's still bug #736486 scheduled for 6.3 timeline.


Note You need to log in before you can comment on or make changes to this bug.