While using lvmetad socket activation, the lvmetad initialization consisting of two steps could cause the information to be missing if any client queries the lvmetad too early. The lvmetad intialization consists of two steps: 1 - starting the lvmetad daemon itself 2 - running the pvscan --cache to fill it up with information The problem that arise is probably rare, but still existing, with older SysV init script mechanism as this one does not support on-demand socket-based activation and it just starts the service on boot sequentially with other services. However, with systemd where the lvmetad daemon is started on-demand on the first access to the socket, we could hit the obvious problem: 1 - lvmetad daemon is not running yet, waiting on the access to the socket (lvm2-lvmetad.socket) to be autoactivated 2 - once the socket is accessed and a query from a client on the socket buffered waiting to be processed, systemd starts lvm2-lvmetad.service unit and executes the ExecStart=/usr/sbin/lvmetad and then the additional ExecStartPost=/usr/sbin/lvm pvscan --cache command 3 - while already finished with the ExecStart, the ExecStartPost command is still being executed, *however, the query is processed*, but the lvmetad cache is still empty (as pvscan --cache has not finished yet) A simple reproducer might be: [0] rawhide/~ # systemctl stop lvm2-lvmetad.service Warning: Stopping lvm2-lvmetad.service, but it can still be activated by: lvm2-lvmetad.socket [0] rawhide/~ # pvs -----> no data here as this opens the socket and fires the on-demand lvmetad service activation!!! [0] rawhide/~ # pvs ----> data available here as pvscan --cache has finished already!!! PV VG Fmt Attr PSize PFree /dev/sda vg1 lvm2 a-- 124.00m 108.00m Suggestions for a solution: A - when starting lvmetad, automatically do the scan and fill the cache with info directly within lvmetad code and only consider lvmetad daemon started if the initial scan is done as well (but this is against the original lvmetad design I guess) B - block lvmetad answer (and so the client) for a client query until the initial scan is done via pvscan --cache call C - ...any other idea is welcome
A) Yes, this goes against design, and undermines the intended security model (lvmetad does not need, and should not have, any privileges to access block devices or really anything at all but its own socket). B) This is workable, although somewhat double-edged. If the pvscan --cache fails, you could end up with a stuck system, because lvmetad is running and accepting connections but it just sits there blocking all clients. Clients could time out but it's still very inconvenient. C) The best route would be a systemd extension to allow running an init command *before* unblocking the FIFO for the rest of the system (i.e. forwarding traffic only for commands in the init block, while blocking the rest, when that's done, allow everyone through). D) Failing C, I would propose to keep a "complete" flag in the lvmetad, which would reflect whether lvmetad believes it has data on the complete system. Commands would, besides the normal response, also give the value of this flag. The other commands have three choices then: 1) Re-try a couple times, dozing off for a moment inbetween. If this keeps failing, just pretend lvmetad is not there, scan normally. 2) Pretend lvmetad is not there right away. Scan normally. 3) Either 1 or 2, but instead of "scan normally", we both scan and while scanning, stuff the data into lvmetad. The variant 3) would also come in useful if we ever wanted to do autoactivation without systemd (there was a plan to do that, but I am not entirely convinced this is necessary; we do have the code to do that for dmeventd). As for the "complete" flag, it would be also useful to implement pvscan --cache during normal lifetime of lvmetad, by setting complete 0, erasing all data in lvmetad, filling it back in from scan results and re-setting complete to 1.
(In reply to comment #1) > A) Yes, this goes against design, and undermines the intended security model > (lvmetad does not need, and should not have, any privileges to access block > devices or really anything at all but its own socket). > OK. > B) This is workable, although somewhat double-edged. If the pvscan --cache > fails, you could end up with a stuck system, because lvmetad is running and > accepting connections but it just sits there blocking all clients. Clients > could time out but it's still very inconvenient. > Yes, it has some inconveniences, like the risk of blocking and bricking the LVM if something goes wrong. > C) The best route would be a systemd extension to allow running an init > command *before* unblocking the FIFO for the rest of the system (i.e. > forwarding traffic only for commands in the init block, while blocking the > rest, when that's done, allow everyone through). > Try to file a systemd RFC. Yes, that would solve that, BUT let' make sure it's not only us who need this and it's widely usable by others as well (...do others have this tiny problem as we do? If yes, how do they cope with it? Do they even care?). But yes, I agree with you :) However, I can imagine the very first counterargument that would come and that would be: "Why don't you initialize your daemon properly during the daemon start itself???". > D) Failing C, I would propose to keep a "complete" flag in the lvmetad, > which would reflect whether lvmetad believes it has data on the complete > system. Commands would, besides the normal response, also give the value of > this flag. The other commands have three choices then: > > 1) Re-try a couple times, dozing off for a moment inbetween. If this keeps > failing, just pretend lvmetad is not there, scan normally. > 2) Pretend lvmetad is not there right away. Scan normally. > 3) Either 1 or 2, but instead of "scan normally", we both scan and while > scanning, stuff the data into lvmetad. > I don't like "2" at all, "1" is ugly, but if we don't have anything better, I'd go with that one. > The variant 3) would also come in useful if we ever wanted to do > autoactivation without systemd (there was a plan to do that, but I am not > entirely convinced this is necessary; we do have the code to do that for > dmeventd). > If you mean forking the daemon off of an lvm binary - we should really avoid that! Deprecated. > As for the "complete" flag, it would be also useful to implement pvscan > --cache during normal lifetime of lvmetad, by setting complete 0, erasing > all data in lvmetad, filling it back in from scan results and re-setting > complete to 1. Yup, I agree with "complete erase" and subsequent "refill" support. I can imagine it might come handy in some problematic situations or for debugging at least.
The solution for #814782 will also fix this bug. When lvmetad starts, its "validity token" will be empty, but each command that tries to use lvmetad will use a non-empty token; first such command will thus trigger an initial scan. No further scans will be triggered until either the validity token actually changes (currently only happens when lvmetad/filter changes) or when lvmetad is killed/restarted.
...and as a consequence, we can remove the "ExecStartPost=pvscan --cache" command from lvmetad systemd unit altogether as lvmetad will be intialized lazily by first LVM command/lvm2app use. So in case LVM is not necessary for system use, lvmetad will be instantiated only if really needed (which will save some resources). The first command will take a little bit more to execute, but that's fine. There's no place for races this way (I hope).
Fixed upstream. The systemd unit can be updated to reflect that.