Can you please escalate? This is urgent for RHV 4.1 release and should be fixed for 7.3.z.

(In reply to Liron Aravot from comment #0)
> (...)we use use_lvmetad=0 on the command (so cache should have no
> effect).

What do you have set for lvmetad in /etc/lvm/lvm.conf?

> 
> Additional info:
> 
> When attempting to execute with -vvvv, I've got the following error -
> LVM command executed by lvmpolld failed.

Long running operations are managed by lvmpolld daemon since RHEL7.2 IIRC. The pvmove monitoring executed on daemon side failed for some reason.

Could you have a look at the daemon's log for more info?
(journalctl -u lvm2-lvmpolld)

If it's not of much help please try to run it in debug mode by:
systemctl stop lvm2-lvmpolld
systemctl stop lvm2-lvmpolld.socket
lvmpolld -f -l all

and capture the output when you issue your pvmove cmd in another terminal.

I think that what happened is in fact this:

you started the pvmove w/ lvmetad dissabled but lvmpolld daemon runs commands in default config therefore probably with use_lvmetad=1 and lvmetad was not aware of changes in mda made by pvmove cmd with use_lvmetad=0 on cmdline (according to your report)

Thanks Ondrej,
prior to you comment I made a run with lvm2-lvmetad.service lvm2-lvmetad.socket disabled, pvmove/lvmpolld worked as expected from the first execution - so the scenario you specified makes sense.

I'll report back with the additional info you requested.

Created attachment 1231806 [details]
lvmpolld test - log

Ondrej, please find the requested log attached.

Thanks, Liron.

Hi Liron,

it's as I suspected earlier. The command initiated on lvmpolld side runs with use_lvmetad=1 set and therefore doesn't see pvmove0 mirror lv (missing in lvmetad cache). My recommendation is to turn off lvmetad for good if you have bad experience with it and most importantly don't mix commands modifyting lvm metadata (with use_lvmetad=0) and commands with use_lvmetad=1 set while lvmetad is running because that's what's causing the mda corruption on lvmetad side.

The only currently supported setup is to stop lvmetad manually after command modifying mda w/ use_lvmetad=0 was run (also clearly stated in logs). There's plan to stop lvmetad automatically by such harmful command, but that's future feature.

Hi Ondrej,
Thanks for your reply.
Question - when the lvmpolld command is initiated to monitor the pvmove, why doesn't the use_lvmetad=0 flag passed to it? regardless of the option of disabling lvmetad, passing the flag makes sense to me (if pvmove doesn't use the metadata, so should the lvmpolld command monitoring it).


Nir, can you please review the discussion? As we have other issues with having lvmetad enabled - do we feel safe enough to disable it for 4.1? or do we want to do that for 4.2?

Thanks,
Liron

(In reply to Liron Aravot from comment #8)
> Hi Ondrej,
> Thanks for your reply.
> Question - when the lvmpolld command is initiated to monitor the pvmove, why
> doesn't the use_lvmetad=0 flag passed to it? regardless of the option of
> disabling lvmetad, passing the flag makes sense to me (if pvmove doesn't use
> the metadata, so should the lvmpolld command monitoring it).

Sure, we were thinking about passing the option and few others as well. Unfortunately in this case any update on lvmpolld side would make cache in lvmetad outdated. Some commands run in lvmpolld environment may update lvm mda (pvmove case included). Mixing commands w/ use_lvmetad=1 and =0 while lvmetad is actually running leads to problems sooner or later I'm affraid. Turning lvmetad off (or restart it after cmd w/ use_lvmetad=0 set) is the only reliable option here.