Bug 690308

Summary: error: "should had been created by udev but it was not found"
Product: [Fedora] Fedora Reporter: G. Michael Carter <mikey>
Component: lvm2Assignee: LVM and device-mapper development team <lvm-team>
Status: CLOSED CURRENTRELEASE QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: high Docs Contact:
Priority: unspecified    
Version: 15CC: agk, bmarzins, bmr, dwysocha, harald, heinzm, jonathan, lvm-team, mbroz, moli, msnitzer, prajnoha, prockai
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-04-11 15:21:13 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Verbose lvcreate command none

Description G. Michael Carter 2011-03-23 20:40:12 UTC 
I think I've found why none of my lvm mounts are working on boot up.   

First the vgchange in fedora-storage-init never seems to get hit.   And udev seems to have a problem setting up the drivers as well.

[root@liandra systemd]# /sbin/lvm vgchange -a y --sysinit
  10 logical volume(s) in volume group "LiandraStore" now active
  semid 32768: semop failed for cookie 0xd4d83af: incorrect semaphore state
  Failed to set a proper state for notification semaphore identified by cookie value 223183791 (0xd4d83af) to initialize waiting for incoming notifications.
  /dev/mapper/LiandraStore-VirtualMachines not set up by udev: Falling back to direct node creation.
  /dev/mapper/LiandraStore-Old--Liandra not set up by udev: Falling back to direct node creation.
  /dev/mapper/LiandraStore-Backups not set up by udev: Falling back to direct node creation.
  /dev/mapper/LiandraStore-LiveCD not set up by udev: Falling back to direct node creation.
  /dev/mapper/LiandraStore-iTunes_Media not set up by udev: Falling back to direct node creation.
  /dev/mapper/LiandraStore-DVD--Working not set up by udev: Falling back to direct node creation.
  The link /dev/LiandraStore/VirtualMachines should had been created by udev but it was not found. Falling back to direct link creation.
  The link /dev/LiandraStore/Old-Liandra should had been created by udev but it was not found. Falling back to direct link creation.
  The link /dev/LiandraStore/Backups should had been created by udev but it was not found. Falling back to direct link creation.
  The link /dev/LiandraStore/LiveCD should had been created by udev but it was not found. Falling back to direct link creation.
  The link /dev/LiandraStore/iTunes_Media should had been created by udev but it was not found. Falling back to direct link creation.
  The link /dev/LiandraStore/DVD-Working should had been created by udev but it was not found. Falling back to direct link creation.


Any ideas how I debug this further?
Comment 1 G. Michael Carter 2011-03-23 20:41:52 UTC 
This may be related to by dbus not starting as well in bug: 689861.   Not sure.
Comment 2 Harald Hoyer 2011-03-24 08:57:04 UTC 
are you sure udevd is running?
Comment 3 G. Michael Carter 2011-03-24 13:13:11 UTC 
I'm assuming it is:


[root@liandra ~]# ps -ef | grep udev
root       488     1  0 Mar23 ?        00:00:00 /sbin/udevd
root     18778 18319  0 09:12 pts/1    00:00:00 grep --color=auto udev

[root@liandra ~]# lvcreate -L 10G -n Test LiandraStore 
  /dev/mapper/LiandraStore-Test not set up by udev: Falling back to direct node creation.
  The link /dev/LiandraStore/Test should had been created by udev but it was not found. Falling back to direct link creation.
  Logical volume "Test" created

[root@liandra ~]# lvremove LiandraStore/Test
Do you really want to remove active logical volume Test? [y/n]: y
  Logical volume "Test" successfully removed
  semid 98304: semop failed for cookie 0xd4d613a: incorrect semaphore state
  Failed to set a proper state for notification semaphore identified by cookie value 223174970 (0xd4d613a) to initialize waiting for incoming notifications.


Any further tests I ran run to verify it is running correctly?
Comment 4 Harald Hoyer 2011-03-24 13:44:12 UTC 
any ideas from the lvm team, what's going wrong here?
Comment 5 Milan Broz 2011-03-24 14:21:54 UTC 
This usually means, that you are using udev-enabled lvm (default in Fedora) but there are missing lvm udev rules (explicitly rule with "dmsetup udevcomplete" call, 95-dm-notify.rules).

lvm will fallback to manual node removal (I think that the warning is about semaphore state is some minor bug in lvm code but it is just consequence - if udev rules missing, your system is simply misconfigured.)

Verify, that device-mapper is properly installed.

To diagnose it, please run the lvm command with -vvvv and add output here.
Comment 6 G. Michael Carter 2011-03-29 13:23:30 UTC 
Created attachment 488442 [details]
Verbose lvcreate command
Comment 7 G. Michael Carter 2011-03-29 13:24:05 UTC 
Also verified the /lib/udev/rules.d/95-dm-notify.rules is the same in F15 as it is in other releases.
Comment 8 Peter Rajnoha 2011-03-29 13:55:17 UTC 
What's the kernel version used? Is this a stock Fedora kernel?
Comment 9 G. Michael Carter 2011-03-29 13:58:25 UTC 
Yes it's stock.   2.6.38-1.fc15.x86_64
Comment 10 Peter Rajnoha 2011-03-31 14:30:19 UTC 
Well, based on the log, uevent was not generated correctly:

  " Uevent not generated! Calling udev_complete internally to avoid process lock-up."

This decrements the value of the semaphore used for synchronization. However, if the event (for some reason) *is* generated despite the fact that the kernel function responsible for sending the uevent reports a failure and then we decrement the value once again from within the udev rules based on the generated event (/lib/udev/rules.d/95-dm-notify.rules), we can end up with that:

  "semid 32768: semop failed for cookie 0xd4d83af: incorrect semaphore state..."

The other messages about "... not set up by udev. Falling back to direct node creation" is just a consequence of failed synchronisation with udev (since we decremented semaphore value prematurely).

However I'm not quite sure now why this happens. It could be a bug in kernel code as well... (but I can't reproduce that)
Comment 11 Harald Hoyer 2011-03-31 15:31:04 UTC 
you might also try the new udev version in F15
Comment 12 G. Michael Carter 2011-04-11 15:05:54 UTC 
New version in F15 seems to have solved the problems.