Bug 1730055

This looks like a docker problem to me. Does devicemapper.dmUdevWaitFct() recover after the udev sockets are back up?

Well, this is an interesting one!

I've just discussed this with Michal Sekletar from systemd and it looks we need systemd-udevd to support handing over the open uevent netlink socket from old to new instance of the udev daemon, hence not closing it and not loosing the uevents this way. He's going to investigate this possibility. Actually we do something very similar in dmeventd already...

Otherwise, from LVM/DM point of view (as well as docker's DM wrapper), I don't think we can do anything else here than getting the udevd's uevent processing timeout somehow (e.g. via libudev) and then set that timeout for the semaphores we use for the DM uevent processing notification. But there are two issues with this still:

  - getting the timeout this way is race-prone (but at least "some" timeout would be there preventing complete hang, even though the timeout could be set incorrectly if we restart udevd with different timeout value after we get the timeout value);

  - and then there's the fact that the uevents are still lost in between udevd stop and start (the "restart" during upgrade is still "stop" and "start" inside).

As demonstrated by this bug, the assumption that the event is not going to disappear is not safe and makes the code deadlock, which shouldn't ever happen. IMHO it's better to time out and continue running with some data possibly lost (and log about that, of course) than to deadlock because of a false assumption.

(In reply to Peter Rajnoha from comment #6)
>   - getting the timeout this way is race-prone (but at least "some" timeout
> would be there preventing complete hang, even though the timeout could be
> set incorrectly if we restart udevd with different timeout value after we
> get the timeout value);

I don't know how lvm handles this, but I'm not sure where the race would be. If there's no data on the socket within let's say 1 sec, time out don't read from the socket anymore. Where's the race, exactly? 

>   - and then there's the fact that the uevents are still lost in between
> udevd stop and start (the "restart" during upgrade is still "stop" and
> "start" inside).

Well, that's the problem that we're trying to solve... Even with the implementation of the socket handover, it's always a good defensive practise to assume that socket data gets lost.

So, ideally, the fix should be on both sides, correct?

(In reply to Jan Synacek from comment #7)
> As demonstrated by this bug, the assumption that the event is not going to
> disappear is not safe and makes the code deadlock, which shouldn't ever
> happen. IMHO it's better to time out and continue running with some data
> possibly lost (and log about that, of course) than to deadlock because of a
> false assumption.

Timeouts are always wrong. You never know what's the best timeout value. You have different environments with different workloads and different timing - one value can work perfectly in one environment, but fails horribly in another.

But as I said, if we were going to use SOME timeout value for the sempahores in libdm, it would be the timeout that udevd is currently set to (...because it doesn't make sense to wait longer than the time udevd has to process the uevent because the notification to unlock the libdm semaphore is part of that uevent processing in udevd). But...

> 
> (In reply to Peter Rajnoha from comment #6)
> >   - getting the timeout this way is race-prone (but at least "some" timeout
> > would be there preventing complete hang, even though the timeout could be
> > set incorrectly if we restart udevd with different timeout value after we
> > get the timeout value);
> 
> I don't know how lvm handles this, but I'm not sure where the race would be.
> If there's no data on the socket within let's say 1 sec, time out don't read
> from the socket anymore. Where's the race, exactly? 

1) the external tool (like LVM) would get the udevd event processing timeout value (say via libudev and it's 180s)
2) the udevd is stopped and started with a different timeout value, processing the uevent (say the timeout is now higher, e.g. 360s)
3) the external tool uses the outdated timeout value (so here it would timeout sooner than it should - after 180s)

> 
> >   - and then there's the fact that the uevents are still lost in between
> > udevd stop and start (the "restart" during upgrade is still "stop" and
> > "start" inside).
> 
> Well, that's the problem that we're trying to solve... Even with the
> implementation of the socket handover, it's always a good defensive practise
> to assume that socket data gets lost.
> 
> So, ideally, the fix should be on both sides, correct?

Udevd is a core system service, it can't be losing uevents! It MUST do the very best it can do to not lose them as its the main consumer of them and various other system components depend on it.

Of course, the socket buffer is not infinite so if it gets full within the time we're restarting udevd, well, that's a problem. But I think we can detect buffer overrun(?). In that case, udevd can at least execute "udevadm trigger" to reexecute udev rules. However, if the buffer is set appropriately, we should minimize this condition to a very minimum. Otherwise, all the socket activation for services in systemd would be useless and spoiled - it depends on socket buffers and then services reading that buffer after systemd instantiates the service itself.

But there are theoretical and practical issues here.

There's also the fact that after all those years, I think this is the very first time I see this problem with udevd not handing over the open netlink socket during its restart to manifest somehow. So for a long time, this was just a theoretical issue. It's because the situation where there are udevd restart and at the same time there are devices reconfigurations in parallel aren't (weren't) that common and now they are, it seems.

If the LVM/libdm didn't deadlock, we wouldn't find out so quickly that there's the problem with udevd not handing over the socket properly during restarts - we're not hiding the problems this way and they at least get the proper urgency to resolve. When timeouts just happen, the issues simply get less urgency - iow, who cares about some timeout somewhere if the system keeps going. We've been having a few reported hangs sporadically with these semaphores over 10 years we use them in libdm, but it was always pointing to some problem with urgency then and it got resolved (or at least it pointed to a problematic component).

This should be fixed on udevd side - udevd needs to hand over the open netlink socket from old to new instance on restart. Otherwise, udevd can lose important uevents...

Red Hat Enterprise Linux 7 shipped it's final minor release on September 29th, 2020. 7.9 was the last minor releases scheduled for RHEL 7.
From intial triage it does not appear the remaining Bugzillas meet the inclusion criteria for Maintenance Phase 2 and will now be closed. 

From the RHEL life cycle page:
https://access.redhat.com/support/policy/updates/errata#Maintenance_Support_2_Phase
"During Maintenance Support 2 Phase for Red Hat Enterprise Linux version 7,Red Hat defined Critical and Important impact Security Advisories (RHSAs) and selected (at Red Hat discretion) Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available."

If this BZ was closed in error and meets the above criteria please re-open it flag for 7.9.z, provide suitable business and technical justifications, and follow the process for Accelerated Fixes:
https://source.redhat.com/groups/public/pnt-cxno/pnt_customer_experience_and_operations_wiki/support_delivery_accelerated_fix_release_handbook  

Feature Requests can re-opened and moved to RHEL 8 if the desired functionality is not already present in the product. 

Please reach out to the applicable Product Experience Engineer[0] if you have any questions or concerns.  

[0] https://bugzilla.redhat.com/page.cgi?id=agile_component_mapping.html&product=Red+Hat+Enterprise+Linux+7