Bug 1766498

What I see is blivet telling lvm to activate rhvh/root, lvm saying it is done, then the device node is missing around .2 sec later. Maybe LVM people can help us see what the problem is...

03:39:50,614 INF program: Running [157] lvm lvchange -ay rhvh/root --config= devices { preferred_names=["^/dev/mapper/", "^/dev/md/", "^/dev/sd"] filter=["r|/360a9800050334c33424b41762d736d45$|","r|/360a9800050334c33424b41762d745551$|"] } log {level=7 file=/tmp/lvm.log syslog=0} ...
03:39:50,707 INF program: stdout[157]: 
03:39:50,708 INF program: stderr[157]:   /dev/sdg: open failed: No medium found
  /dev/sdh: open failed: No medium found
  /dev/sdg: open failed: No medium found
  /dev/sdh: open failed: No medium found

03:39:50,708 INF program: ...done [157] (exit code: 0)
03:39:50,708 INF program: Running... udevadm settle --timeout=300
03:39:50,747 DBG program: Return code: 0
03:39:50,921 INF program: Running... lsblk --perms --fs --bytes
03:39:50,965 INF program: NAME                                            SIZE OWNER GROUP MODE       FSTYPE       LABEL                UUID                                   MOUNTPOINT
03:39:50,966 INF program: loop0                                      499433472 root  disk  brw-rw---- squashfs
03:39:50,966 INF program: loop1                                     3221225472 root  disk  brw-rw---- ext4         Anaconda             8da60120-f0e9-4885-b640-7d5a2a433d5b
03:39:50,966 INF program: |-live-rw                                 3221225472 root  disk  brw-rw---- ext4         Anaconda             8da60120-f0e9-4885-b640-7d5a2a433d5b   /
03:39:50,966 INF program: `-live-base                               3221225472 root  disk  brw-rw---- ext4         Anaconda             8da60120-f0e9-4885-b640-7d5a2a433d5b
03:39:50,966 INF program: loop2                                    34359738368 root  disk  brw-rw----
03:39:50,966 INF program: `-live-rw                                 3221225472 root  disk  brw-rw---- ext4         Anaconda             8da60120-f0e9-4885-b640-7d5a2a433d5b   /
03:39:50,966 INF program: sda                                     213674622976 root  disk  brw-rw---- mpath_member
03:39:50,967 INF program: `-360a9800050334c33424b41762d726954     213674622976 root  disk  brw-rw----
03:39:50,967 INF program: |-360a9800050334c33424b41762d726954p1   1073741824 root  disk  brw-rw---- ext4                              01ec1944-c7d0-4cde-a92e-0ce9e48dd7e4
03:39:50,967 INF program: `-360a9800050334c33424b41762d726954p2 212599832576 root  disk  brw-rw---- LVM2_member                       4EdLtg-x96m-S1zJ-vo2D-6RQM-OoJ6-t2SefJ
03:39:50,967 INF program: |-rhvh-pool00_tmeta                   1073741824 root  disk  brw-rw----
03:39:50,967 INF program: | `-rhvh-pool00-tpool               152999821312 root  disk  brw-rw----
03:39:50,967 INF program: |   `-rhvh-pool00                   152999821312 root  disk  brw-rw----
03:39:50,967 INF program: `-rhvh-pool00_tdata                 152999821312 root  disk  brw-rw----
03:39:50,967 INF program: `-rhvh-pool00-tpool               152999821312 root  disk  brw-rw----
03:39:50,968 INF program: `-rhvh-pool00                   152999821312 root  disk  brw-rw----
03:39:50,968 INF program: sdb                                     106300440576 root  disk  brw-rw---- mpath_member
03:39:50,968 INF program: `-360a9800050334c33424b41762d736d45     106300440576 root  disk  brw-rw----
03:39:50,968 INF program: sdc                                     106300440576 root  disk  brw-rw---- mpath_member
03:39:50,968 INF program: `-360a9800050334c33424b41762d745551     106300440576 root  disk  brw-rw----
03:39:50,968 INF program: sdd                                     213674622976 root  disk  brw-rw---- mpath_member
03:39:50,968 INF program: `-360a9800050334c33424b41762d726954     213674622976 root  disk  brw-rw----
03:39:50,968 INF program: |-360a9800050334c33424b41762d726954p1   1073741824 root  disk  brw-rw---- ext4                              01ec1944-c7d0-4cde-a92e-0ce9e48dd7e4
03:39:50,969 INF program: `-360a9800050334c33424b41762d726954p2 212599832576 root  disk  brw-rw---- LVM2_member                       4EdLtg-x96m-S1zJ-vo2D-6RQM-OoJ6-t2SefJ
03:39:50,969 INF program: |-rhvh-pool00_tmeta                   1073741824 root  disk  brw-rw----
03:39:50,969 INF program: | `-rhvh-pool00-tpool               152999821312 root  disk  brw-rw----
03:39:50,969 INF program: |   `-rhvh-pool00                   152999821312 root  disk  brw-rw----
03:39:50,969 INF program: `-rhvh-pool00_tdata                 152999821312 root  disk  brw-rw----
03:39:50,969 INF program: `-rhvh-pool00-tpool               152999821312 root  disk  brw-rw----
03:39:50,969 INF program: `-rhvh-pool00                   152999821312 root  disk  brw-rw----
03:39:50,970 INF program: sde                                     106300440576 root  disk  brw-rw---- mpath_member
03:39:50,970 INF program: `-360a9800050334c33424b41762d736d45     106300440576 root  disk  brw-rw----
03:39:50,970 INF program: sdf                                     106300440576 root  disk  brw-rw---- mpath_member
03:39:50,970 INF program: `-360a9800050334c33424b41762d745551     106300440576 root  disk  brw-rw----
03:39:50,970 INF program: sr0                                       1348468736 root  cdrom brw-rw---- iso9660      RHVH-4.4 RHVH.x86_64 2019-09-26-15-15-19-00                 /run/install/repo
03:39:50,970 DBG program: Return code: 0

I see this in the lvm.log:

  03:39:50.687354 lvchange[5978] metadata/lv_manip.c:7575  ACTIVATION_SKIP flag set for LV rhvh/root, skipping activation.

Normally, by default, we set this flag only for thin snapshots, but not for the origins. Is there a possibility someone set this flag explicitly before (lvcreate -ky /lvchange -ky)? If this flag is set, then the lvchange returns success even if it doesn't activate the LV itself.

I don't see any lvcreate in the logs though, but the report says that the "automatic partitioning" was selected so I assume this isn't about importing existing layout where the "skip activation" flag could be set before. David, where can I find these in the anaconda logs? I just see pvs/lvs/lvchange.

Also, I see multiple lvchange -ay/-an (some of them are even with "-K" that is "ignore activation skip", but not for rhvh/root). Is this expected?

rhvh sets activation skip for rhvh/root in its postinstall script (after anaconda ends), we basically want to make sure that we only activate the LVs that are needed for the running system.  Is this a problem ? I would think that when we want to destroy existing LVs we should ignoreactivationskip, no ?

(In reply to Yuval Turgeman from comment #7)
> rhvh sets activation skip for rhvh/root in its postinstall script (after
> anaconda ends), we basically want to make sure that we only activate the LVs
> that are needed for the running system.  Is this a problem ?

Well, but we're setting up the LV where the root fs resides (deducing from the LV name) - that one shouldn't really have the "skip activation" flag set because we need to have that activated.

> I would think
> that when we want to destroy existing LVs we should ignoreactivationskip, no
> ?

Nope, the lvremove removes LVs irrespective of the "skip activation" flag. This flag is really useful and respected only during LV activation (vgchagne/lvchagne -ay/-aay ...). The flag's purpose is to have a possibility to mark LVs which are not essential for the system and which can be activate manually later by calling vgchagne/lvchange with the "-K|--ignoreactivationskip" option to force the activation. For now, LVM sets this flag automatically for thin snapshot LVs because it is expected that there may be lots of them which would delay system bootup uselessly (...and snapshots are usually considered as "archives" of the state at certain point in time, so not necessary for current state of the system).

Sorry, I'm not sure I understand what we're doing wrong - basically, the reason we set activationskip on root is because we create a snapshot from it right after installation, and boot from that snapshot.  Doesn't it fit the description in comment 8 ?  In general, I would think that if a disk is good and I'd like anaconda/blivet to make use of it for my installation, it should be able to do so regardless of what was installed previously on this disk, no ?

If you're booting into a root fs which is on thin snapshot LV marked as "skip on activation", then dracut already deals with this by using "lvchange -K ..." in its scripts to force the activation:

  # grep -r lvchange /usr/lib/dracut/modules.d/90lvm
  /usr/lib/dracut/modules.d/90lvm/lvm_scan.sh:        lvm lvchange --yes -K -ay $lvm_quirk_args $LV 2>&1 | vinfo

...so that scenario is covered already. If you used your LVs this way, it's perfectly OK. If you set the activation skip *after* anaconda finished its job, then that's OK too. But here, the issue is *during* the installation, while anaconda does the processing still, as I read from the information in this report.


Then the problem, I think, is in anaconda trying to reuse what was installed before and not using "-K|--ignoreactivationskip" option for any vgchange/lvchange it calls? I'm not sure why any previous VGs/LVs are not simply removed and they're activated first instead.

So is this about reusing disks with some pre-existing VG/LV structure and then passing that to anaconda? And then anaconda tries to activate the pre-existing thin snapshot LVs? (without the "-K" option which in turn causes the LVs to not activate).

I judge only from the logs where I don't see any lvcreate, just lvchange and that one is without "-K|--ignoreactivationskip" option...

IIUC, anaconda doesn't reuse the existing layout, but tries to destroy it and create a new one using blivet, which can't destroy it for some reason

Well, there's this last LVM call logged in program.log (without "-K" option):

  03:39:50,614 INF program: Running [157] lvm lvchange -ay rhvh/root --config= devices { preferred_names=["^/dev/mapper/", "^/dev/md/", "^/dev/sd"] filter=["r|/360a9800050334c33424b41762d736d45$|","r|/360a9800050334c33424b41762d745551$|"] } log {level=7 file=/tmp/lvm.log syslog=0} ...

And no vgcreate/lvcreate or vgremove/lvremove/related wipefs before, just pvs, lvs, vgchange -an, vgchange -ay, lvchange- an, lvchange -ay for various VGs/LVs. So without vgcreate/lvcreate, I suppose it must be trying to activate the already existing VGs/LVs which already exist from previous use.

(In reply to Yuval Turgeman from comment #9)
> description in comment 8 ?  In general, I would think that if a disk is good
> and I'd like anaconda/blivet to make use of it for my installation, it
> should be able to do so regardless of what was installed previously on this
> disk, no ?

No. There absolutely are limits to what we can effectively manage.


The reason we activate the LVs is to remove the file system metadata
from them. Yes, I realize it is possible that LVM at some point in the
relatively-recent past has gained the ability to do this, but doing this in a
special way for LVM because of that is not an approach I am interested in if
it can be avoided. I will see if there is a way for us to determine in
advance that we need to pass -K to lvchange.

FWIW I don't see how this could possibly be a regression.

Also backing up a bit, this will not affect normal installations since normal installations do not configure volumes to ignore activation commands.

(In reply to David Lehman from comment #13)
> I will see if there is a way for us to determine in
> advance that we need to pass -K to lvchange.

This may help - we have lv_skip_activation report field (so it's not necessary to rely on the "lv_attr" field which may be harder to parse as it can change in the future to include more attrs - we should have separate field for each char in the attr "attr" field):

# lvs vg -o+lv_skip_activation
  LV    VG Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert SkipAct        
  lvol1 vg Vwi-a-tz--   1.00g pool        0.00                                                   
  lvol2 vg Vwi---tz--   1.00g pool lvol1                                                         
  lvol3 vg Vwi---tz-k   1.00g pool lvol1                                          skip activation
  pool  vg twi---tz-- 100.00m             0.00   10.94                                   


So it has blank if not or "skip activation" if set. Or it can be reported in binary too:

# lvs vg -o+lv_skip_activation --binary
  LV    VG Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert SkipAct        
  lvol1 vg Vwi-a-tz--   1.00g pool        0.00                                                  0
  lvol2 vg Vwi---tz--   1.00g pool lvol1                                                        0
  lvol3 vg Vwi---tz-k   1.00g pool lvol1                                                        1
  pool  vg twi---tz-- 100.00m             0.00   10.94                                          0

Or the -S|--select may help as well:

# lvs --noheadings --select lv_skip_activation=yes vg
  lvol3 vg Vwi---tz-k 1.00g pool lvol1

# lvs --noheadings --select lv_skip_activation=no vg
  lvol1 vg Vwi-a-tz--   1.00g pool        0.00                                   
  lvol2 vg Vwi---tz--   1.00g pool lvol1                                         
  pool  vg twi---tz-- 100.00m             0.00   10.94

Maybe this can be really simple:

  https://github.com/storaged-project/blivet/pull/811

Any thoughts, Peter?

(In reply to David Lehman from comment #17)
> Maybe this can be really simple:
> 
>   https://github.com/storaged-project/blivet/pull/811
> 
> Any thoughts, Peter?

Looks good to me. I think it makes sense here to ignore this "skip activation" flag altogether and just activate everything - because we need to properly clean things up.

Looks good, thanks - we might also want to set global/thin_check_executable to an empty string when we're activating the lv for teardown, but perhaps on a different bug, it happens once in a while but a little harder to reproduce on a real system (it can be set to an executable that always fail)

Can you confirm that this is a rhel8.2 beta-blocker as it is blocking rhv installations?

(In reply to Rob Evers from comment #25)
> Can you confirm that this is a rhel8.2 beta-blocker as it is blocking rhv
> installations?

Yes please. From customer perspective the user experience without this fix is pretty bad.

I prepared an updates image with a proposed patch. Please test it by adding the following to the boot command line:

  inst.updates=http://people.redhat.com/~dlehman/updates/ignore-skip.0.img

  (the image assumes you are using RHEL-8.1.0 installer images; please advise if this is not the case)

Let me know how it goes.

I found that RHVH had also set the rhvh/root LV as read-only using lvchange, which again prevents us from wiping the fs signature from it. Please try an expanded updates image by adding the following to your boot command line:

  inst.updates=http://people.redhat.com/~dlehman/updates/rhvh.0.img


Please let me know how it goes and attach logs if there is a failure.

(In reply to David Lehman from comment #32)
> I found that RHVH had also set the rhvh/root LV as read-only using lvchange,
> which again prevents us from wiping the fs signature from it. Please try an
> expanded updates image by adding the following to your boot command line:
> 
>   inst.updates=http://people.redhat.com/~dlehman/updates/rhvh.0.img
> 
> 
> Please let me know how it goes and attach logs if there is a failure.

Tested RHVH-4.4-20190926.3-RHVH-x86_64-dvd1.iso, installation succeeded this time.

Both patches have been posted for upstream review (the first is already approved):

  https://github.com/storaged-project/blivet/pull/811

Discussed with David about this bug over IRC and reporting here the outcome:
- Bug is reproducible also on RHEL 8.1 ISO instaling over a dirty disk with a previous RHV-H installation
- From RHV 4.4 planning perspective we are fine not blocking RHEL 8.2 beta on this bug, provided it will be fixed for 8.2 GA.

Can be reproduced by installation of RHEL on to a custom disk image in VM created like this:
truncate -s 9G disk.img
device=$(losetup -f)
losetup -P $device disk.img
pvcreate $device
vgcreate test $device
lvcreate -l 100%FREE --permission r --setactivationskip y -n testlv test
losetup -d $device

(In reply to David Lehman from comment #37)
> Both patches have been posted for upstream review (the first is already
> approved):
> 
>   https://github.com/storaged-project/blivet/pull/811

How is this going?

(In reply to Rob Evers from comment #40)
> (In reply to David Lehman from comment #37)
> > Both patches have been posted for upstream review (the first is already
> > approved):
> > 
> >   https://github.com/storaged-project/blivet/pull/811
> 
> How is this going?

I've reworked the patch substantially, so now I have to re-test.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1701