Yes, running plain "udevadm trigger" (with default action=add) is dangerous with respect to device-mapper devices and should never be run without properly filtering out dm devices.

The reason is that we need to create any nodes and symlinks on CHANGE event only. ADD event is generated just after the device is created (it is generated automatically and this process is not managed by device-mapper itself, but it's in "outer" parts of the kernel). Such device does not have its table loaded nor is it yet properly activated/resumed and ready to use. Just when the device is resumed, we get a CHANGE event that we need to react on (this one *is* generated directly by device-mapper in the kernel).

Unluckily, there's no way in udev to ignore an event. DM/LVM2 udev rules are written in a way to ignore ADD events because of the reason given above. We're still trying to persuade and discuss this issue with udev team. I believe we'll get to a solution once (..although I still haven't managed to persuade them. But bug reports like this one help me to gather more and more evidence.. so thanks a lot! :)).

Anyway, the udevadm trigger with (default) add events should not be called for device-mapper devices in anaconda anymore. The last time I looked at anaconda, I saw CHANGE events generated only for block devices. But it seems I need to have a look once again if anything has changed there..

(In reply to comment #1)
> Yes, running plain "udevadm trigger" (with default action=add) is dangerous
> with respect to device-mapper devices and should never be run without properly
> filtering out dm devices.

The complaint here is that a basic action ("udevadm trigger") causes device-mapper devices to *disappear*. It does not only occur in anaconda. This is seriously broken and should be fixed. Hence the bug.

That's an impasse we have with the upstream udev developers I'm afraid, not something we can fix.  Please do not run udevadm trigger on these devices - it is not meant to be used in this way.

To be clear, it is simply wrong for anything other than the kernel to issue an ADD event on a dm device.  For dm, 'add' means "a new major+minor has been assigned for a new dm device".  That can only happen once, and only the kernel can do that - userspace cannot perform that assignment.  And we require that parameters are attached to *all* dm uevents.  So even if you wanted to trigger a 'change' event from userspace, you would still have to set all the parameters just like the kernel did.

What is going wrong such that you are finding you need to run 'trigger'?
We should track down the underlying problem and get that fixed instead.

If the problem is that /dev has got unexplained incorrect or missing entries in it, 'dmsetup mknodes' or 'vgmknodes' (for LVM) are correct ways to fix that.  (Although they are not completely udev-ified yet.)

Originally the dm/lvm udev rules coped with 'trigger' by doing nothing in response.  But that useful feature got removed from udev and the workaround now would be to reimplement that within our own rules - basically to stat the node and store the result such that udev would reproduce the status quo - and implementing that no-op would be a waste of time.

At coldplug, or at module package update we always run "udevadm trigger". Users must be able to run "udevadm trigger" at any time. It must produce the same results in /dev in all cases. The logic in the dm rules should be changed to allow trigger to run at any time, just like all other udev rules works.

We can't do that without the ability to 'ignore' events, as explained.  That ability got removed.  There should be no need to trigger false events.

To be clear, after a kernel-generated 'add' event on a dm device, the node should normally not be present in /dev - it has no table and is not yet usable.  Without the 'ignore' capability, the rules make sure the node does not exist.  Previously they could "do nothing".  It's quite wrong for anything to generate such an event routinely from userspace.

(In reply to comment #7)
> To be clear, after a kernel-generated 'add' event on a dm device, the node
> should normally not be present in /dev - it has no table and is not yet usable.
>  Without the 'ignore' capability, the rules make sure the node does not exist. 
> Previously they could "do nothing".  It's quite wrong for anything to generate
> such an event routinely from userspace.    

Fedora uses devtmpfs too, all kernel devices will automatically have a corresponding /dev entry created by the kernel, and not touched by userspace at that time.

The rules you declare here can not be applied to current systems. I'm still sure the dm rules need to be fixed here, not to rely on all the magic assumptions they make.

Possible fixes:

(1) Don't perform trigger on any dm devices routinely - use one of the match options to exclude them?

(2) Reinstate the 'ignore' functionality which we used to rely upon to avoid this problem.

(3) ? Change the functionality of 'trigger' somehow so that when no variables are supplied (which would normally be set by the kernel), they are pre-populated from the database - so that the input to our rules already tells us what the device state is meant to be (and it can by verified by the rules - or modified if a package update happened to change them)?

(In reply to comment #6)
> We can't do that without the ability to 'ignore' events, as explained.  That
> ability got removed.  There should be no need to trigger false events.    

I don't think we can make such rules. Trigger does run at any time, it always
did, and we can probably not make such assumptions now. It must produce the
same results with every event, like it does for all other rules.

Events could not be suppressed by rules, the removed functionality only removed
the execution of later rules, but the events were still sent out. This was
completely inconsistent and not a useful behavior, and therefore got removed.

The fundamental point here is that there are two incompatible definitions of what an add event is for.  The kernel ties an add event to the creation of a device kobject.  Userspace treats that as if that is the same as the creation of a device that userspace may access.  The two are not the same thing.  So far we've been working around this on the userspace side - ignore 'add' and use 'change' instead.  The alternative would be to try to fix it kernel-side, and not tie 'add' directly to the kobject, but generate it instead when there is a device that is ready for userspace to access i.e. to push the userspace definition into the kernel.

(In reply to comment #9)
> Possible fixes:
> 
> (1) Don't perform trigger on any dm devices routinely - use one of the match
> options to exclude them?

It happens at coldplug two times. Many external programs do this, like kernel module updates and all such things.
 
> (2) Reinstate the 'ignore' functionality which we used to rely upon to avoid
> this problem.

So far, no subsystem is allowed to "own" events. We can never know which rules need to run. And it was never possible to suppress device events, the events have always been sent out, only rules did not run, but there are many subscribers to events which run their own stuff, and this is nothing we wanted to support in that fashion.

> (3) ? Change the functionality of 'trigger' somehow so that when no variables
> are supplied (which would normally be set by the kernel), they are
> pre-populated from the database - so that the input to our rules already tells
> us what the device state is meant to be (and it can by verified by the rules -
> or modified if a package update happened to change them)?    

So far, udev needs to prevent stale information, and repopulates the database at any new event. This is required and ecxpected behavior for current users. We would need to add something to make this possible without breaking current users.

Anyway, on current systems, all primary device nodes are created by the kernel itself, even before the event is sent out. Udev can not do anything here. The assumptions made by the dm tools need to be updated, I guess.

I suspect one thing the 'trigger' with no args is trying to do is to 'fix up' /dev to match (a possibly new) reality.  For dm, that should be triggered off a change event not an add, and even then (in a few cases) some information is missing, which the still-running lvm process knows (and supplied via variables attached to the event).

"all primary device nodes are created by the kernel itself, even before the event is sent out"

Device-mapper pre-dates udev and this has never applied to dm device-nodes (and there is no easy way to re-sequence this).  So could 'udevadm trigger' treat non-primary device nodes differently?

I doubt there is going to be any 'clean' solution to this problem - it's a matter finding some least-bad workaround(s) that actually have a chance of working.

If you go back to first principles, it's the in-kernel 'add' event generation that should be moved for 'non-primary' devices.  Then at least we'll get closer to consistent userspace handling.

Otherwise we need to come up with some further hacks/workarounds that allow for no-ops, and as you point out, still won't help other folk listening into the event stream and not having the same 'special case logic'.

(In reply to comment #8)
> The rules you declare here can not be applied to current systems. I'm still
> sure the dm rules need to be fixed here, not to rely on all the magic
> assumptions they make.

That depends on the view that is taken. For us, it's not an assumption - this is how DM devices are activated and it's not a single-step process. Instead, it consists of three distinct steps as described above. From this viewpoint, we could consider that udev makes an assumption that devices are prepared for use after an ADD event only (and apply all the rules/create the nodes at that time) and totally forgets that it's not true in each situation.

It's a little pity that we're probably the only ones who have more complex activation scheme than just "sheer plugging". This is not how DM works. But that's not a reason to ignore the problem and pretend we're not here...

Anyway, we can't go in circles forever. I think we have something in common and that is we would like to provide users functionality and do it the best way we can. We tried and tried to solve this, but it seems it won't be possible without working together on the final solution. As Alasdair pointed out, the major problem lies in the fact that there are different views on what the meaning of the ADD event is first.

We can't do more here unless the kernel is changed or udev provides some option or workaround for such situations (or we do more and more ugly hacks). There's no clean fix we could do here from our side only at this point.

The meaaning of ADD event is just one side of the problem. The other one is generating any events artificially in userspace through calling that udevadm triggers. Again, as Alasdair pointed out, we also set several variables that go with the events. These variables originate in userspace program based on the decisions made in runtime and direct the application of the rules, e.g. which devices should be created in which dev directory (like LVM2 does) - the visibility of the nodes we started to debate about once in email correspondence but faded away somehow... The notion of "private devices" or basic set of "permissions" if you prefer.

Also, there is the "watch" rule that we find really hard to support. First of all, we can't synchronize with artificial events based on this rule at all. There's no way to do it, at least I can't see any, but I'm open to any ideas. All this effort is to provide users a way to have at least some degree of certainty that all dm/lvm functionality is synchronized with udev and does not interfere with each other.

Now, bugs appear where we can't even remove a device properly since it is still being opened as a result of the spurious events still processed and we can't synchronize with them (for example, see bug #509908).

Again, these spurious events don't have all needed variables set...

So two basic problems we need to tackle - the notion of ADD event, its significance and artificial events being generated, either based on "udevadm trigger" logic or based on the "watch" rule.

Neither of us like ugly hacks but only clean solutions, I presume... We tried hard, but I'm afraid we're on the edge of what we can do from our side.

Just for the record, the removed functionality of "ignore_device" and "last_rule" was not 100% sufficient anyway - it removed any existing symlinks/nodes when used, just like when the rules are skipped. This is also not what we expect. Why this behaviour? There's a legitimate REMOVE event to do that... Or am I wrong?

(In reply to comment #15)
> Just for the record, the removed functionality of "ignore_device" and
> "last_rule" was not 100% sufficient anyway - it removed any existing
> symlinks/nodes when used, just like when the rules are skipped. This is also
> not what we expect. Why this behaviour? There's a legitimate REMOVE event to do
> that... Or am I wrong?    

Symlinks are custom properties of the device. New events clear the database to remove all stale information. Therefore all properties have to be set with every event again. If the symlink is not set again, it will be removed. Otherwise, if, for example, the filesystem label changes, we have no way to remove the old and now invalid symlink. The remove event just removes everything, add/change updates things, which might also need to remove stuff.

Now we should be able to support even artificial events using the new IMPORT{db} udev rule that helps us to reuse any existing information stored in udev database before.

I'm just awaiting one change in udev's start_udev script that will also enable the use of udev database information from initrd (dracut) after the root fs is mounted and so the udev is reinitialised (rhbz #603724).

The patch for device-mapper udev rules included in lvm2 2.02.68 release (device-mapper 1.02.50). I'm not quite sure if this change will end up as an F-13 update or not (it also needs that change in udev init script mentioned above). It will surely be included in upcoming F-14. Setting status to nextrelease.