Created attachment 1079329 [details]
File: anaconda-tb

Created attachment 1079330 [details]
File: anaconda.log

Created attachment 1079331 [details]
File: environ

Created attachment 1079332 [details]
File: ks.cfg

Created attachment 1079333 [details]
File: lsblk_output

Created attachment 1079334 [details]
File: nmcli_dev_list

Created attachment 1079335 [details]
File: os_info

Created attachment 1079336 [details]
File: program.log

Created attachment 1079337 [details]
File: storage.log

Created attachment 1079338 [details]
File: syslog

Created attachment 1079339 [details]
File: ifcfg.log

Created attachment 1079340 [details]
File: packaging.log

To clarify, the kickstart disk config (worked on other identical hosts) is:

zerombr
clearpart --all
bootloader --location=mbr --driveorder=sda

part prepboot --fstype="prepboot" --size=5 --ondisk=sda --asprimary 
part prepboot --fstype="prepboot" --size=5 --ondisk=sdb --asprimary 
part /boot --size 1000 --fstype ext4 --ondisk=sda --asprimary 
part /boot/whocares --size 1000 --fstype ext4 --ondisk=sdb --asprimary 

part raid.008011 --size=65536 --ondisk=sda --grow
part raid.008012 --size=65536 --ondisk=sdb --grow

part raid.008021 --grow --size=200 --ondisk=sdc
part raid.008022 --grow --size=200 --ondisk=sdd
part raid.008023 --grow --size=200 --ondisk=sde
part raid.008024 --grow --size=200 --ondisk=sdf
part raid.008025 --grow --size=200 --ondisk=sdg
part raid.008026 --grow --size=200 --ondisk=sdh
part raid.008027 --grow --size=200 --ondisk=sdi
part raid.008028 --grow --size=200 --ondisk=sdj
part raid.008029 --grow --size=200 --ondisk=sdk
part raid.008030 --grow --size=200 --ondisk=sdl

raid pv.009001 --level=1 --device=md1 raid.008011 raid.008012 
raid pv.009002 --level=6 --device=md2 raid.008021 raid.008022 raid.008023 raid.008024 raid.008025 raid.008026 raid.008027 raid.008028 raid.008029 raid.008030

volgroup vg_guests --pesize=4096 pv.009002
volgroup vg_server --pesize=4096 pv.009001
logvol / --fstype=xfs --name=LogVol00 --vgname=vg_server --size=32768
logvol swap --name=LogVol01 --vgname=vg_server --size=8192

Oh and then it crashed again once it'd reported this bug:

Logging out
Status: NEW https://bugzilla.redhat.com/show_bug.cgi?id=1268190
===============================================================================
===============================================================================

===============================================================================
An unknown error has occurred
===============================================================================
anaconda 21.48.22.35-1 exception report
Traceback (most recent call first): 
  File "/usr/lib/python2.7/site-packages/blivet/devicelibs/lvm.py", line 408, in vgreduce
    raise LVMError("vgreduce failed for %s: %s" % (vg_name, msg))
  File "/usr/lib/python2.7/site-packages/blivet/devices/lvm.py", line 230, in _destroy
    lvm.vgreduce(self.name, None, missing=True)
  File "/usr/lib/python2.7/site-packages/blivet/devices/storage.py", line 512, in destroy
    self._destroy()
  File "/usr/lib/python2.7/site-packages/blivet/deviceaction.py", line 344, in execute
    self.device.destroy()
  File "/usr/lib/python2.7/site-packages/blivet/devicetree.py", line 377, in processActions
    action.execute(callbacks)
  File "/usr/lib/python2.7/site-packages/blivet/__init__.py", line 373, in doIt
    self.devicetree.processActions(callbacks)
  File "/usr/lib/python2.7/site-packages/blivet/__init__.py", line 223, in turnOnFilesystems
    storage.doIt(callbacks)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/install.py", line 186, in doInstall
    turnOnFilesystems(storage, mountOnly=flags.flags.dirInstall, callbacks=callbacks_reg)
  File "/usr/lib64/python2.7/threading.py", line 764, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/threads.py", line 227, in run
    threading.Thread.run(self, *args, **kwargs)
LVMError: vgreduce failed for vg_guests: running lvm vgreduce --config  devices { preferred_names=["^/dev/mapper/", "^/dev/md/", "^/dev/sd"] }  --removemissing --force vg_guests failed

What do you want to do now?
1) Report Bug
2) Debug
3) Quit

It looks like this is having trouble removing a previous install, vg_guests appears to vanish before it can run vg_reduce on it. You should be able to work around this by making sure the disks have been cleaned up before starting the installation.

(In reply to Brian Lane from comment #16)
> It looks like this is having trouble removing a previous install, vg_guests
> appears to vanish before it can run vg_reduce on it. You should be able to
> work around this by making sure the disks have been cleaned up before
> starting the installation.

I wrote out 100Mb of /dev/zero to all hard disks (a-l) and it still crashes in the same way. I have 3 identical servers being installed with a single kick start, two worked just fine, this one crashes in the same way ever with the zeros written out to all the disks.

> I wrote out 100Mb of /dev/zero to all hard disks (a-l) and it still crashes
> in the same way. I have 3 identical servers being installed with a single
> kick start, two worked just fine, this one crashes in the same way ever with
> the zeros written out to all the disks.

Writing to the beginning isn't necessarily good enough.  Backup copies of metadata tend to get scattered all over the place and are sometimes picked up.

Also are you still seeing this with 7.2 final?  I see the original report was against the beta.

> Writing to the beginning isn't necessarily good enough.  Backup copies of
> metadata tend to get scattered all over the place and are sometimes picked
> up.

I'm aware of that, I'd also tried dding /dev/zero to the entire set of disks. It's LVM on a RAID-1 two disk md-array. The non OS disks are md-raid 6,

> Also are you still seeing this with 7.2 final?  I see the original report
> was against the beta.

I need to rebuild all four of the machines with a clean RHEL 7.2 but have not had a change window to do it yet (we're awaiting some network changes in PXH2)

Can't the wipefs tool do the job, it should be able to clean up everything, all metadata?

I think this is a duplicate of bug 1261621. Is the bug reproducible on 7.2 release or 7.3 nightlies?

(In reply to David Lehman from comment #21)
> I think this is a duplicate of bug 1261621. Is the bug reproducible on 7.2
> release or 7.3 nightlies?

I will be rebuilding the Power8 infrastructure in Fedora from clean 7.2+updates in early April so I'll be able to report back then. We're in freeze at the moment for F-24 Alpha, and as it's production I don't have the resources to test 7.3 nightlies.

That works for me -- thanks. I'll mark this as NEEDINFO until then so we don't forget.

Getting the following, just trying to get all the rest of the details off the server.

An unknown error has occured, look at the /tmp/anaconda-tb* file(s) for more details

===============================================================================
An unknown error has occurred
===============================================================================
anaconda 21.48.22.56-1 exception report
Traceback (most recent call first):
  File "/usr/lib/python2.7/site-packages/blivet/devicelibs/lvm.py", line 538, in lvactivate
    raise LVMError("lvactivate failed for %s: %s" % (lv_name, msg))
  File "/usr/lib/python2.7/site-packages/blivet/devices/lvm.py", line 690, in _setup
    lvm.lvactivate(self.vg.name, self._name)
  File "/usr/lib/python2.7/site-packages/blivet/devices/storage.py", line 413, in setup
    self._setup(orig=orig)
  File "/usr/lib/python2.7/site-packages/blivet/deviceaction.py", line 650, in execute
    self.device.setup(orig=True)
  File "/usr/lib/python2.7/site-packages/blivet/devicetree.py", line 377, in processActions
    action.execute(callbacks)
  File "/usr/lib/python2.7/site-packages/blivet/__init__.py", line 374, in doIt
    self.devicetree.processActions(callbacks)
  File "/usr/lib/python2.7/site-packages/blivet/__init__.py", line 224, in turnOnFilesystems
    storage.doIt(callbacks)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/install.py", line 186, in doInstall
    turnOnFilesystems(storage, mountOnly=flags.flags.dirInstall, callbacks=callbacks_reg)
  File "/usr/lib64/python2.7/threading.py", line 764, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/site-packages/pyanaconda/threads.py", line 227, in run
    threading.Thread.run(self, *args, **kwargs)
LVMError: lvactivate failed for LogVol01: running lvm lvchange -a y --config  devices { preferred_names=["^/dev/mapper/", "^/dev/md/", "^/dev/sd"] }  vg_server/LogVol01 failed

Okay, I think I can see what's generally happening. When an md pv gets activated, udev fires off an asynchronous task to run pvscan on the newly-active pv. Until that pvscan has run, lvm has no knowledge of the md pv's existence (even if it was previously active on the system). Since that happens asynchronously, we need to find a way to tell when it's done. I will update the bug when I have more information.

Peter, this is similar to bug 1261621 (how to know when lvmetad has auto-activated lvm on a just-activated md pv) except that the vg appears to contain no lvs. So how can we know when lvmetad has finished its work in this case?

(In reply to David Lehman from comment #28)
> Peter, this is similar to bug 1261621 (how to know when lvmetad has
> auto-activated lvm on a just-activated md pv) except that the vg appears to
> contain no lvs. So how can we know when lvmetad has finished its work in
> this case?

We can't - the activation part is asynchronous operation - we call "pvscan --cache -aay" for each PV that appears (either from udev rule directly [RHEL6] or as part of lvm2-pvscan@major:minor.service [RHEL7]). Each pvscan informs lvmetad about new PV present in the system and lvmetad sends back information to pvscan whether the VG is now complete or not. If it's complete, the pvscan runs VG activation in addition (...while lvmetad continues in parallel with any other queries it may receive from other clients - LVM commands).

Now, the important part is that lvmetad doesn't get the information back from pvscan once pvscan activated the VG - lvmetad just caches LVM metadata, it doesn't store runtime information, like the activation state. So lvmetad doesn't do any waiting for pvscan to do the activation part. So if any LVM command queries lvmetad *after* it got the PV scan result and *before* the pvscan activates the VG, it will see the VG with all LVs and the LVM command itself will look for the activation state (it doens't get this information from lvmetad).

If some concrete device is expected within systemd environment, we can hook a service on .device sytemd unit representing the device itself and wait for it to appear this way (...like it is when there's a .mount unit for a device which waits for .device unit to appear with a timeout so it can do the actual mount).

(In reply to David Lehman from comment #28)
> Peter, this is similar to bug 1261621 (how to know when lvmetad has
> auto-activated lvm on a just-activated md pv) except that the vg appears to
> contain no lvs. So how can we know when lvmetad has finished its work in
> this case?

(sorry, I misread the "contain no lvs")

...well, if there are no LVs in the VG at all, there's nothing extra that pvscan can do - there's nothing to activate. So it's just the scanning part that pvscan does. Then it's just about whether the VG is complete or not and we know that from the "partial" state of the VG (vgs -o+partial or the corresponding "p" char as 4th bit in vg_attr column).

Please try this updates image and report back with the results:

  http://people.redhat.com/dlehman/updates-1268190.0.img

(In reply to David Lehman from comment #31)
> Please try this updates image and report back with the results:
> 
>   http://people.redhat.com/dlehman/updates-1268190.0.img

I don't have the hardware spare to randomly test this at the moment, all 4 devices where we were seeing this problem are currently in production with numerous production services running on top, it would take weeks to schedule maintenance to migrate these or schedule downtime.

I can provide the kickstart that recreated it if that's useful.

I imagine the kickstart would be useful to QE.

This issue is not active for more than a year so I'm closing it now. If it is still requested to be fixed please feel free to re-open it.