Bug 839941 - Missing information for lvmetad query while using socket-based systemd service autoactivation
Missing information for lvmetad query while using socket-based systemd servic...
Status: CLOSED NEXTRELEASE
Product: Fedora
Classification: Fedora
Component: lvm2 (Show other bugs)
rawhide
All Linux
unspecified Severity high
: ---
: ---
Assigned To: Petr Rockai
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-07-13 05:13 EDT by Peter Rajnoha
Modified: 2012-09-26 15:14 EDT (History)
11 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-09-26 15:14:06 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Peter Rajnoha 2012-07-13 05:13:26 EDT
While using lvmetad socket activation, the lvmetad initialization consisting of two steps could cause the information to be missing if any client queries the lvmetad too early.

The lvmetad intialization consists of two steps:

  1 - starting the lvmetad daemon itself

  2 - running the pvscan --cache to fill it up with information

The problem that arise is probably rare, but still existing, with older SysV init script mechanism as this one does not support on-demand socket-based activation and it just starts the service on boot sequentially with other services.

However, with systemd where the lvmetad daemon is started on-demand on the first access to the socket, we could hit the obvious problem:

  1 - lvmetad daemon is not running yet, waiting on the access to the socket (lvm2-lvmetad.socket) to be autoactivated

  2 - once the socket is accessed and a query from a client on the socket buffered waiting to be processed, systemd starts lvm2-lvmetad.service unit and executes the ExecStart=/usr/sbin/lvmetad and then the additional ExecStartPost=/usr/sbin/lvm pvscan --cache command

  3 - while already finished with the ExecStart, the ExecStartPost command is still being executed, *however, the query is processed*, but the lvmetad cache is still empty (as pvscan --cache has not finished yet)

A simple reproducer might be:
[0] rawhide/~ # systemctl stop lvm2-lvmetad.service
Warning: Stopping lvm2-lvmetad.service, but it can still be activated by:
  lvm2-lvmetad.socket

[0] rawhide/~ # pvs
-----> no data here as this opens the socket and fires the on-demand lvmetad service activation!!!

[0] rawhide/~ # pvs
----> data available here as pvscan --cache has finished already!!!
  PV         VG   Fmt  Attr PSize   PFree  
  /dev/sda   vg1  lvm2 a--  124.00m 108.00m


Suggestions for a solution:

  A - when starting lvmetad, automatically do the scan and fill the cache with info directly within lvmetad code and only consider lvmetad daemon started if the initial scan is done as well (but this is against the original lvmetad design I guess)

  B - block lvmetad answer (and so the client) for a client query until the initial scan is done via pvscan --cache call

  C - ...any other idea is welcome
Comment 1 Petr Rockai 2012-07-20 11:19:23 EDT
A) Yes, this goes against design, and undermines the intended security model (lvmetad does not need, and should not have, any privileges to access block devices or really anything at all but its own socket).

B) This is workable, although somewhat double-edged. If the pvscan --cache fails, you could end up with a stuck system, because lvmetad is running and accepting connections but it just sits there blocking all clients. Clients could time out but it's still very inconvenient.

C) The best route would be a systemd extension to allow running an init command *before* unblocking the FIFO for the rest of the system (i.e. forwarding traffic only for commands in the init block, while blocking the rest, when that's done, allow everyone through).

D) Failing C, I would propose to keep a "complete" flag in the lvmetad, which would reflect whether lvmetad believes it has data on the complete system. Commands would, besides the normal response, also give the value of this flag. The other commands have three choices then:

  1) Re-try a couple times, dozing off for a moment inbetween. If this keeps failing, just pretend lvmetad is not there, scan normally.
  2) Pretend lvmetad is not there right away. Scan normally.
  3) Either 1 or 2, but instead of "scan normally", we both scan and while scanning, stuff the data into lvmetad.

The variant 3) would also come in useful if we ever wanted to do autoactivation without systemd (there was a plan to do that, but I am not entirely convinced this is necessary; we do have the code to do that for dmeventd).

As for the "complete" flag, it would be also useful to implement pvscan --cache during normal lifetime of lvmetad, by setting complete 0, erasing all data in lvmetad, filling it back in from scan results and re-setting complete to 1.
Comment 2 Peter Rajnoha 2012-07-23 06:28:56 EDT
(In reply to comment #1)
> A) Yes, this goes against design, and undermines the intended security model
> (lvmetad does not need, and should not have, any privileges to access block
> devices or really anything at all but its own socket).
> 

OK.

> B) This is workable, although somewhat double-edged. If the pvscan --cache
> fails, you could end up with a stuck system, because lvmetad is running and
> accepting connections but it just sits there blocking all clients. Clients
> could time out but it's still very inconvenient.
> 

Yes, it has some inconveniences, like the risk of blocking and bricking the LVM if something goes wrong.

> C) The best route would be a systemd extension to allow running an init
> command *before* unblocking the FIFO for the rest of the system (i.e.
> forwarding traffic only for commands in the init block, while blocking the
> rest, when that's done, allow everyone through).
> 

Try to file a systemd RFC. Yes, that would solve that, BUT let' make sure it's not only us who need this and it's widely usable by others as well (...do others have this tiny problem as we do? If yes, how do they cope with it? Do they even care?). But yes, I agree with you :) However, I can imagine the very first counterargument that would come and that would be: "Why don't you initialize your daemon properly during the daemon start itself???".

> D) Failing C, I would propose to keep a "complete" flag in the lvmetad,
> which would reflect whether lvmetad believes it has data on the complete
> system. Commands would, besides the normal response, also give the value of
> this flag. The other commands have three choices then:
> 
>   1) Re-try a couple times, dozing off for a moment inbetween. If this keeps
> failing, just pretend lvmetad is not there, scan normally.
>   2) Pretend lvmetad is not there right away. Scan normally.
>   3) Either 1 or 2, but instead of "scan normally", we both scan and while
> scanning, stuff the data into lvmetad.
> 

I don't like "2" at all, "1" is ugly, but if we don't have anything better, I'd go with that one.

> The variant 3) would also come in useful if we ever wanted to do
> autoactivation without systemd (there was a plan to do that, but I am not
> entirely convinced this is necessary; we do have the code to do that for
> dmeventd).
> 

If you mean forking the daemon off of an lvm binary - we should really avoid that! Deprecated.

> As for the "complete" flag, it would be also useful to implement pvscan
> --cache during normal lifetime of lvmetad, by setting complete 0, erasing
> all data in lvmetad, filling it back in from scan results and re-setting
> complete to 1.

Yup, I agree with "complete erase" and subsequent "refill" support. I can imagine it might come handy in some problematic situations or for debugging at least.
Comment 3 Petr Rockai 2012-08-07 16:32:29 EDT
The solution for #814782 will also fix this bug. When lvmetad starts, its "validity token" will be empty, but each command that tries to use lvmetad will use a non-empty token; first such command will thus trigger an initial scan. No further scans will be triggered until either the validity token actually changes (currently only happens when lvmetad/filter changes) or when lvmetad is killed/restarted.
Comment 4 Peter Rajnoha 2012-08-08 04:31:07 EDT
...and as a consequence, we can remove the "ExecStartPost=pvscan --cache" command from lvmetad systemd unit altogether as lvmetad will be intialized lazily by first LVM command/lvm2app use. So in case LVM is not necessary for system use, lvmetad will be instantiated only if really needed (which will save some resources). The first command will take a little bit more to execute, but that's fine. There's no place for races this way (I hope).
Comment 5 Petr Rockai 2012-09-26 15:14:06 EDT
Fixed upstream. The systemd unit can be updated to reflect that.

Note You need to log in before you can comment on or make changes to this bug.