Bug 1378612

Summary: lvm2-activation-generator doesn't read additional /etc/lvm/lvm*.conf files
Product: [Community] LVM and device-mapper Reporter: Timur Bakeyev <timur>
Component: lvm2Assignee: Peter Rajnoha <prajnoha>
lvm2 sub component: Activating existing Logical Volumes QA Contact: cluster-qe <cluster-qe>
Status: CLOSED UPSTREAM Docs Contact:
Severity: high    
Priority: unspecified CC: agk, heinzm, jbrassow, msnitzer, prajnoha, zkabelac
Version: 2.02.165Flags: rule-engine: lvm-technical-solution?
rule-engine: lvm-test-coverage?
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-24 08:41:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Timur Bakeyev 2016-09-22 23:18:46 UTC
Description of problem:

Placing into /etc/lvm/lvmlocal.conf:

global {
    use_lvmetad = 0
}

entirely breaks activation of the LVM volumes, resulting in various boot problems.

Version-Release number of selected component (if applicable):

LVM TOOLS 2.02.165(2) (2016-09-07) and earlier, backported to the Debian 8.6 Jessie.

How reproducible:

Since recent enough versions of LVM2 support additional local configuration files like /etc/lvm/lvmlocal.conf and earlier versions supported tagged additional configs, like /etc/lvm/lvm_drbd.conf I use those options to put desired changes into separate files to ease maintainance and upgrades of the LVM.

This approach used to work fine, in particular with my DRBD/LVM/NFS cluster setup where some minimal changes were placed into lvm_drbd.conf(and later into lvmlocal.conf):

devices {
    global_filter = [ "a|^/dev/sd.*|", "a|^/dev/drbd.*|", "r|.*|"  ]
    write_cache_state = 0
}

activation {
    volume_list = [ "vg00", "store0", "share0" ]
}

Latest versions of LVM2 resource script for pacemaker started to complain about use of use_lvmetad=1, suggesting to disable it. So, lvmlocal.conf got additional:

global {
    use_lvmetad = 0
}

And that lead to the failure of volumes activation. That's possibly was made worse by my attempt to bring consistency to the content of /etc/lvm/ in a live system and initrd.

Current mkinitramfs LVM2 script(at least on Debian) copies only /etc/lvm/lvm.conf to the resulting image, skipping rest of lvm*.conf files, which may alter the final configuration.

I've added /etc/initramfs-tools/hooks/lvm2-local script to work around this:
#!/bin/sh

PREREQ="lvm2"

prereqs()
{
  echo "$PREREQ"
}

case $1 in
prereqs)
  prereqs
  exit 0
  ;;
esac

. /usr/share/initramfs-tools/hook-functions

# command profile config -> metadata profile config -> tag config -> lvmlocal.conf -> lvm.conf
if [ -d /etc/lvm/ ]; then
  mkdir -p ${DESTDIR}/etc/lvm
  cp /etc/lvm/lvm*.conf ${DESTDIR}/etc/lvm/
fi

So, now I get consistent behvior of LVM2 during initrd stage and the normal boot.

Steps to Reproduce:
1. Place into /etc/lvm/lvmlocal.conf:
global {
    use_lvmetad = 0
}

and make sure that it is also copied into initrd image. LVM in initrd image reads all available config files and doesn't try to use lvmetad(actually, with use_lvmetad=1 it complains that can't access lvmetad and does direct read of meta info).

2. Make sure that initrd is updated and contains all lvm*.conf files.

3. Reboot

Actual results:

# lvs -a
  LV   VG   Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  opt  vg00 -wi------- 16.00g

# systemctl status lvm2-activation.service
● lvm2-activation.service
   Loaded: not-found (Reason: No such file or directory)
   Active: inactive (dead)

Expected results:

# lvs -a
File descriptor 7 (pipe:[15440]) leaked on lvs invocation. Parent PID 796: bash
  LV   VG   Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  opt  vg00 -wi-a----- 16.00g

# systemctl status lvm2-activation.service
● lvm2-activation.service - Activation of LVM2 logical volumes
   Loaded: loaded (/etc/lvm/lvm.conf)
   Active: inactive (dead) since Fri 2016-09-23 01:02:30 CEST; 2min 58s ago
     Docs: man:lvm2-activation-generator(8)
  Process: 390 ExecStart=/sbin/lvm vgchange -aay --ignoreskippedcluster (code=exited, status=0/SUCCESS)
 Main PID: 390 (code=exited, status=0/SUCCESS)

Sep 23 01:02:30 exp01.ad-lab.nl lvm[390]: 1 logical volume(s) in volume group "vg00" now active
Sep 23 01:02:30 exp01.ad-lab.nl systemd[1]: Started Activation of LVM2 logical volumes.

Additional info:

# strace -ff -o /tmp/lvm2 /lib/systemd/system-generators/lvm2-activation-generator /var/tmp /var/tmp /var/tmp

Reveals, that lvm2-activation-generator reads only lvm.conf:
open("/proc/filesystems", O_RDONLY)     = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f392d855000
read(3, "nodev\tsysfs\nnodev\trootfs\nnodev\tr"..., 1024) = 311
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0x7f392d855000, 4096)            = 0
open("/dev/kmsg", O_WRONLY|O_NOCTTY)    = 3
stat("/etc/lvm/lvm.conf", {st_mode=S_IFREG|0644, st_size=93207, ...}) = 0
stat("/etc/lvm/lvm.conf", {st_mode=S_IFREG|0644, st_size=93207, ...}) = 0
open("/etc/lvm/lvm.conf", O_RDONLY)     = 4
mmap(NULL, 93207, PROT_READ, MAP_PRIVATE, 4, 0) = 0x7f392d82f000
munmap(0x7f392d82f000, 93207)           = 0
close(4)                                = 0
stat("/etc/lvm/lvm.conf", {st_mode=S_IFREG|0644, st_size=93207, ...}) = 0
stat("/etc/lvm/lvm.conf", {st_mode=S_IFREG|0644, st_size=93207, ...}) = 0
open("/etc/lvm/lvm.conf", O_RDONLY)     = 4
mmap(NULL, 93207, PROT_READ, MAP_PRIVATE, 4, 0) = 0x7f392d82f000
munmap(0x7f392d82f000, 93207)           = 0
close(4)                                = 0
close(3)                                = 0

Without checking other additional configuraton files, which makes it inconsistent with lvm tool(and friends).

Placing global/use_lvmetad=0 into /etc/lvm/lvm.conf makes it work as desired, generating appropriate set of lvm2-activation-early.service  lvm2-activation-net.service  lvm2-activation.service

So, can it be extended to read the same set of configuration files as the rest of utils do?

Comment 1 Timur Bakeyev 2016-09-23 02:08:02 UTC
Seems it all comes from the lvm2_activation_generator_systemd_red_hat.c:

static void lvm_get_use_lvmetad_and_lvmpolld(int *use_lvmetad, int *use_lvmpolld)
{
        *use_lvmetad = *use_lvmpolld = 0;

        *use_lvmetad = lvm_config_find_bool(NULL, LVM_CONF_USE_LVMETAD, 0);
        *use_lvmpolld = lvm_config_find_bool(NULL, LVM_CONF_USE_LVMPOLLD, 0);
}

where lvm_config_find_bool() creates config context solely from lvm.conf in lvm_base.c:

/*
 * When full lvm connection is not being used, libh can be NULL
 * and this command will internally create a single-use, light-weight
 * cmd struct that only has cmd->cft populated from lvm.conf.
 */
int lvm_config_find_bool(lvm_t libh, const char *config_path, int fail)
{
        int rc = 0;
        struct cmd_context *cmd;
        struct saved_env e;

        if (libh) {
                cmd = (struct cmd_context *)libh;
                e = store_user_env((struct cmd_context *)libh);
        } else {
                if (!(cmd = create_config_context()))
                        return 0;
        }

        rc = dm_config_tree_find_bool(cmd->cft, config_path, fail);

        if (libh)
                restore_user_env(&e);
        else
                destroy_config_context(cmd);
        return rc;
}

I feel that it's abad idea to rely only on lvm.conf in create_config_context(). Or, at least - to use it to determine the status of global/use_lvmetad, which leads to race condition above.

Comment 2 Peter Rajnoha 2016-09-23 13:04:04 UTC
Good catch! Thanks for the report - it should be fixed now with:

https://git.fedorahosted.org/cgit/lvm2.git/commit/?id=b5e093624dc9bc9644903442e59bf049aeaaa3ba

Will appear in upcoming lvm2 v2.02.166.

Comment 3 Timur Bakeyev 2016-09-24 01:37:01 UTC
Thanks a lot for such a quick resolution!

I've tried the patch on two of my affected systems and it works like a charm.