Bug 1323024 - Undercloud installation disables LVM activation, which causes boot failure on systems with more than root and swap LVs
Summary: Undercloud installation disables LVM activation, which causes boot failure on...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: instack-undercloud
Version: 8.0 (Liberty)
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: beta
: 10.0 (Newton)
Assignee: Jeff Peeler
QA Contact: Shai Revivo
URL:
Whiteboard:
: 1327964 1333569 1340555 1374292 (view as bug list)
Depends On:
Blocks: 1194008 1274548 1295530
TreeView+ depends on / blocked
 
Reported: 2016-04-01 03:05 UTC by Dan Macpherson
Modified: 2020-12-11 12:08 UTC (History)
21 users (show)

Fixed In Version: instack-undercloud-5.0.0-0.20160907134010.649dc3f
Doc Type: Known Issue
Doc Text:
A puppet manifest bug incorrectly disables LVM partition automounting during the undercloud installation process. As a result, it is possible for undercloud hosts with partitions other than root and swap (activated on kernel command line) to only boot into an emergency shell. There are several ways to work around this issue. Choose one from the following: 1. Remove the mountpoints manually from /etc/fstab. Doing so will prevent the issue from manifesting in all future cases. Other partitions could also be removed, and the space added to other partitions (like root or swap). 2. Configure the partitions to be activated in /etc/lvm.conf. Doing so will work until the next update/upgrade, when the undercloud installation is re-run. 3. Restrict initial deployment to only root and swap partitions. This will avoid the issue completely.
Clone Of:
Environment:
Last Closed: 2016-12-14 15:29:31 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2016:2948 0 normal SHIPPED_LIVE Red Hat OpenStack Platform 10 enhancement update 2016-12-14 19:55:27 UTC

Description Dan Macpherson 2016-04-01 03:05:59 UTC
Description of problem:
The Undercloud installation includes the following section in its Puppet manifest (/etc/puppet/manifests/puppet-stack-config.pp):

# Ensure dm thin-pool is never activated. This avoids an issue
# where the instack host (in this case on a VM) was crashing due to
# activation of the docker thin-pool associated with the atomic host.
augeas { 'lvm.conf':
  require => Package['openstack-nova-compute'],
  context => '/files/etc/lvm/lvm.conf/activation/dict/',
  changes => 'set auto_activation_volume_list/list ""'
}

This disables auto-activation of volumes.

This doesn't affect systems that only have root and swap LVs because these get activated when the kernel starts:

linux16 /vmlinuz-3.10.0-327.10.1.el7.x86_64 root=/dev/mapper/rhel_ospd8test-root ro rd.lvm.lv=rhel_ospd8test/root rd.lvm.lv=rhel_ospd8test/swap rhgb quiet LANG=en_AU.UTF-8

However, systems might use additional LVs, especially since the default RHEL installation auto-creates an LV for home if the target installation disk is over a certain size (50 GB according to the RHEL manual).

So if these additional LVs don't get activated and you reboot, the boot process will attempt to find and mount the home LV, hang for a minute and half, then timeout and drop you to the emergency shell.

If you comment out the line added to /etc/lvm/lvm.conf by puppet-stack-config.pp and reboot, the boot succeeds and the home partition is mounted successfully.

Version-Release number of selected component (if applicable):

OSPd Puddle from 2016-03-29.3

How reproducible:
Always

Steps to Reproduce:
1. Install OSPd on a System with additional LVs (e.g. home)
2. Reboot
3. Boot process attempts to mount, fails, then drops to the emergency shell

Actual results:
Boot process attempts to mount, fails, then drops to the emergency shell

Expected results:
Boot process succeeds and home LV mounts successfully.

Additional info:

Comment 3 Steven Hardy 2016-04-01 10:50:30 UTC
Adding jpeeler as he may have more insight into this:

https://review.openstack.org/#/c/258139/

This was added as a workaround for kernel panics when using atomic images - Jeff do we know if this is still needed?

Comment 5 Jeff Peeler 2016-04-01 16:34:56 UTC
As far as I know, there wouldn't be anything that would cause this change to be unneeded. But maybe an ironic expert would be able to chime in and confirm. The point was brought up by James that we didn't know how the undercloud might be used with additional logical volumes, but I thought the agreement was that the undercloud isn't expected to be "general use".

Comment 7 Mike Burns 2016-04-07 21:36:02 UTC
This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Comment 9 Mike Burns 2016-04-18 11:35:05 UTC
*** Bug 1327964 has been marked as a duplicate of this bug. ***

Comment 10 Mike Burns 2016-05-06 10:57:22 UTC
*** Bug 1333569 has been marked as a duplicate of this bug. ***

Comment 14 Mike Burns 2016-06-06 12:55:10 UTC
*** Bug 1340555 has been marked as a duplicate of this bug. ***

Comment 15 Charles Crouch 2016-07-18 22:02:10 UTC
Mike, remind me how this blocks BZ1194008 again?

Comment 16 Jeff Peeler 2016-07-19 13:46:29 UTC
A more precise fix has been made for this issue here:

https://review.openstack.org/#/c/343100/

Comment 17 Mike Burns 2016-08-04 00:17:41 UTC
(In reply to Charles Crouch from comment #15)
> Mike, remind me how this blocks BZ1194008 again?

Bug 1340555 blocked 1194008.  Bug 1340555 was closed as a duplicate of this bug.  When a bug gets closed as a duplicate, the "Blocks" field gets copied over to the duplicate bug.

Comment 18 Charles Crouch 2016-08-08 13:10:08 UTC
Got it! Thanks Mike for the reminder :-)

Comment 19 Jeff Peeler 2016-08-26 14:54:19 UTC
One more fix: https://review.openstack.org/#/c/360790/

(setting back to post as modified should represent committing into RHOS)

Comment 20 Mike Burns 2016-09-08 12:38:57 UTC
*** Bug 1374292 has been marked as a duplicate of this bug. ***

Comment 22 Tzach Shefi 2016-09-21 10:45:29 UTC
Verified. 
On VM with a 100G disk, following a reboot after installing undercloud I don't hit the usual emergency mode any more, home LV remains active. 

  LV   VG   Attr       LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home rhel -wi-ao---- 39.00g                                                    
  root rhel -wi-ao---- 50.00g                                                    
  swap rhel -wi-ao---- 10.00g  

Version:
RHEL7.3 
instack-undercloud-5.0.0-0.20160907134010.649dc3f.el7ost.noarch

#cat /etc/puppet/manifests/puppet-stack-config.pp
augeas { 'lvm.conf':
  require => Package['openstack-nova-compute'],
  context => '/files/etc/lvm/lvm.conf/devices/dict/',
  changes => 'set global_filter/list/1/str "r|^/dev/disk/by-path/ip.*iscsi.*\.org\.openstack:.*|"'


#cat /etc/lvm/lvm.conf 

        allow_changes_with_duplicate_pvs = 0
global_filter = ["r|^/dev/disk/by-path/ip.*iscsi.*\.org\.openstack:.*|"]
}

Happened to install today, been following this bug as it would hit me every time, glade it's resolved.

Comment 24 errata-xmlrpc 2016-12-14 15:29:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-2948.html


Note You need to log in before you can comment on or make changes to this bug.