Bug 1397294

Summary: Undercloud installation fails when IPv6 is disabled
Product: Red Hat OpenStack Reporter: Petr Barta <pbarta>
Component: instack-undercloudAssignee: Alex Schultz <aschultz>
Status: CLOSED CURRENTRELEASE QA Contact: Gurenko Alex <agurenko>
Severity: medium Docs Contact:
Priority: medium    
Version: 9.0 (Mitaka)CC: agurenko, aschultz, hjensas, jcoufal, mburns, racedoro, rhel-osp-director-maint, rszmigie, sclewis, srevivo, tvignaud
Target Milestone: ---Keywords: Triaged, ZStream
Target Release: 12.0 (Pike)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-12-13 16:28:09 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
undercloud-install.log none

Description Petr Barta 2016-11-22 08:13:29 UTC 
Description of problem:

Customer wants to have IPv6 disabled on each node in his system (using ipv6.disable=1 grub option on RHEL7.3). When he's trying to install undercloud on such a node used as director, installation of undercloud fails with same errors as in BZ 1369293 (https://bugzilla.redhat.com/show_bug.cgi?id=1369293), ie 

17:46:26 Error: Could not prefetch sysctl_runtime provider 'sysctl_runtime': sysctl parameter net.ipv6.ip_nonlocal_bind wasn't found on this system
17:46:26 Notice: /Stage[main]/Main/Sysctl::Value[net.ipv4.ip_nonlocal_bind]/Sysctl_runtime[net.ipv4.ip_nonlocal_bind]/val: val changed '0' to '1'
17:46:26 Notice: /Stage[main]/Main/Sysctl::Value[net.ipv6.ip_nonlocal_bind]/Sysctl[net.ipv6.ip_nonlocal_bind]/ensure: created
17:46:26 Error: Execution of '/sbin/sysctl net.ipv6.ip_nonlocal_bind=1' returned 255: sysctl: cannot stat /proc/sys/net/ipv6/ip_nonlocal_bind: No such file or directory
17:46:26 Error: /Stage[main]/Main/Sysctl::Value[net.ipv6.ip_nonlocal_bind]/Sysctl_runtime[net.ipv6.ip_nonlocal_bind]/val: change from absent to 1 failed: Execution of '/sbin/sysctl net.ipv6.ip_nonlocal_bind=1' returned 255: sysctl: cannot stat /proc/sys/net/ipv6/ip_nonlocal_bind: No such file or directory


  No information found that IPv6 is required for installation.


Version-Release number of selected component (if applicable):

RHOSP9 
python-openstackclient.noarch        2.2.0-1.el7ost
openstack-puppet-modules.noarch      1:8.1.8-3.el7ost
openstack-tripleo-puppet-elements.noarch 3.6.2-2.el7
openstack-packstack-puppet.noarch    8.0.0-4.el7ost
puppet-server.noarch                 3.6.2-2.el7
kernel.x86_64                        3.10.0-514.el7




How reproducible:
Always

Steps to Reproduce:
1. Disable IPv6 on latest RHEL7.3 (https://access.redhat.com/solutions/8709)
2. Install undercloud RHOSP9, by using steps from https://access.redhat.com/documentation/en/red-hat-openstack-platform/9/single/director-installation-and-usage
3. Fail occurs during step 4.6 ("openstack undercloud install") command, log available in /home/stack/.instack/install-undercloud.log

Actual results:

Installation fails

Expected results:

Installation succeeds.

Additional info:

Apparently no IPv6 is in fact required, as the installation is successful even when ip6tables blocks all IPv6 traffic in&out. In that situation I would expect the IPv6 and related sysctl option would not be required for successful installation.

If the sysctl option (net.ipv6.ip_nonlocal_bind) is really needed and therefore IPv6 must be enabled for the installation to succeed, it needs to be mentioned as requirement in docs (probably 2.2. section of the Director Installation manual).
Comment 4 Alex Schultz 2017-03-01 20:51:13 UTC 
*** Bug 1401986 has been marked as a duplicate of this bug. ***
Comment 6 Alex Schultz 2017-03-24 20:26:57 UTC 
I've created an upstream bug to track this effort upstream. The current workaround is to temporarily enable ipv6 when installing or upgrading the undercloud and then disabling it afterwards.
Comment 13 Gurenko Alex 2017-12-11 09:53:12 UTC 
I still see undercloud deployment with IPv6 disabled failing on RHEL 7.4 and RHOS 12 with following error:

2017-12-11 10:44:08,432 INFO: [mNotice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[998 log all]/Firewall[998 log all ipv4]: Dependency Service[ip6tables] has failures: true[0m
2017-12-11 10:44:08,432 INFO: [1;33mWarning: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[998 log all]/Firewall[998 log all ipv4]: Skipping because of failed dependencies[0m
2017-12-11 10:44:08,433 INFO: [mNotice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[998 log all]/Firewall[998 log all ipv6]: Dependency Service[ip6tables] has failures: true[0m
2017-12-11 10:44:08,433 INFO: [1;33mWarning: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[998 log all]/Firewall[998 log all ipv6]: Skipping because of failed dependencies[0m
2017-12-11 10:44:08,433 INFO: [mNotice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[999 drop all]/Firewall[999 drop all ipv4]: Dependency Service[ip6tables] has failures: true[0m
2017-12-11 10:44:08,434 INFO: [1;33mWarning: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[999 drop all]/Firewall[999 drop all ipv4]: Skipping because of failed dependencies[0m
2017-12-11 10:44:08,434 INFO: [mNotice: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[999 drop all]/Firewall[999 drop all ipv6]: Dependency Service[ip6tables] has failures: true[0m
2017-12-11 10:44:08,434 INFO: [1;33mWarning: /Stage[main]/Tripleo::Firewall::Post/Tripleo::Firewall::Rule[999 drop all]/Firewall[999 drop all ipv6]: Skipping because of failed dependencies[0m
2017-12-11 10:44:08,435 INFO: [mNotice: /Stage[main]/Firewall::Linux::Redhat/File[/etc/sysconfig/iptables]: Dependency Service[ip6tables] has failures: true[0m
2017-12-11 10:44:08,435 INFO: [1;33mWarning: /Stage[main]/Firewall::Linux::Redhat/File[/etc/sysconfig/iptables]: Skipping because of failed dependencies[0m
2017-12-11 10:44:08,436 INFO: [mNotice: /Stage[main]/Firewall::Linux::Redhat/File[/etc/sysconfig/ip6tables]: Dependency Service[ip6tables] has failures: true[0m
2017-12-11 10:44:08,436 INFO: [1;33mWarning: /Stage[main]/Firewall::Linux::Redhat/File[/etc/sysconfig/ip6tables]: Skipping because of failed dependencies[0m
2017-12-11 10:44:09,726 INFO: [mNotice: Applied catalog in 775.18 seconds[0m
2017-12-11 10:44:19,859 INFO: + rc=6
2017-12-11 10:44:19,860 INFO: + set -e
2017-12-11 10:44:19,860 INFO: + echo 'puppet apply exited with exit code 6'
2017-12-11 10:44:19,860 INFO: puppet apply exited with exit code 6
2017-12-11 10:44:19,861 INFO: + '[' 6 '!=' 2 -a 6 '!=' 0 ']'
2017-12-11 10:44:19,861 INFO: + exit 6
2017-12-11 10:44:19,861 INFO: [2017-12-11 10:44:19,860] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 1]
2017-12-11 10:44:19,861 INFO: 
2017-12-11 10:44:19,861 INFO: [2017-12-11 10:44:19,860] (os-refresh-config) [ERROR] Aborting...
2017-12-11 10:44:19,869 DEBUG: An exception occurred

full log attached
Comment 14 Gurenko Alex 2017-12-11 09:54:01 UTC 
Created attachment 1365864 [details]
undercloud-install.log
Comment 15 Alex Schultz 2017-12-11 17:56:57 UTC 
How was ipv6 disabled in the test? If you do it via kernel command line in grub config it should work. If you only disabled via sysctl it may have exhibited this error.

https://access.redhat.com/solutions/8709#rhel7disable

I confirmed it works if you do the kernel commandline and grub configuration.
Comment 17 Gurenko Alex 2017-12-12 07:40:18 UTC 
(In reply to Alex Schultz from comment #15)
> How was ipv6 disabled in the test? If you do it via kernel command line in
> grub config it should work. If you only disabled via sysctl it may have
> exhibited this error.
> 
> https://access.redhat.com/solutions/8709#rhel7disable
> 
> I confirmed it works if you do the kernel commandline and grub configuration.

 I did it with grub modification and encountered that error. I will re-try today with other builds to see if it matters.
Comment 18 Alex Schultz 2017-12-12 17:28:34 UTC 
This is failing for a different reason around iptables trying to start. I'll have to look later.  Let's drop this from 12.0 and we can see if there's a follow up to a zstream that is needed

2017-12-12 19:04:45,023 INFO: ESC[mNotice: /Stage[main]/Firewall::Linux::Redhat/Service[iptables]/ensure: ensure changed 'stopped' to 'running'ESC[0m
2017-12-12 19:04:45,112 INFO: ESC[1;31mError: Systemd start for ip6tables failed!
2017-12-12 19:04:45,112 INFO: journalctl log for ip6tables:
2017-12-12 19:04:45,112 INFO: -- Logs begin at Tue 2017-12-12 18:50:45 IST, end at Tue 2017-12-12 19:04:45 IST. --
2017-12-12 19:04:45,112 INFO: Dec 12 19:04:44 undercloud-0.local systemd[1]: [/usr/lib/systemd/system/ip6tables.service:3] Failed to add dependency on syslog.target,iptables.service, ignoring: Invalid argument
2017-12-12 19:04:45,112 INFO: Dec 12 19:04:44 undercloud-0.local systemd[1]: [/usr/lib/systemd/system/ip6tables.service:3] Failed to add dependency on syslog.target,iptables.service, ignoring: Invalid argument
2017-12-12 19:04:45,112 INFO: Dec 12 19:04:44 undercloud-0.local systemd[1]: [/usr/lib/systemd/system/ip6tables.service:3] Failed to add dependency on syslog.target,iptables.service, ignoring: Invalid argument
2017-12-12 19:04:45,113 INFO: Dec 12 19:04:44 undercloud-0.local systemd[1]: [/usr/lib/systemd/system/ip6tables.service:3] Failed to add dependency on syslog.target,iptables.service, ignoring: Invalid argument
2017-12-12 19:04:45,113 INFO: Dec 12 19:04:45 undercloud-0.local systemd[1]: [/usr/lib/systemd/system/ip6tables.service:3] Failed to add dependency on syslog.target,iptables.service, ignoring: Invalid argument
2017-12-12 19:04:45,113 INFO: Dec 12 19:04:45 undercloud-0.local systemd[1]: [/usr/lib/systemd/system/ip6tables.service:3] Failed to add dependency on syslog.target,iptables.service, ignoring: Invalid argument
2017-12-12 19:04:45,113 INFO: Dec 12 19:04:45 undercloud-0.local systemd[1]: Starting IPv6 firewall with ip6tables...
2017-12-12 19:04:45,113 INFO: Dec 12 19:04:45 undercloud-0.local ip6tables.init[13122]: ip6tables: Applying firewall rules: ip6tables-restore v1.4.21: ip6tables-restore: unable to initialize table 'filter'
2017-12-12 19:04:45,113 INFO: Dec 12 19:04:45 undercloud-0.local ip6tables.init[13122]: Error occurred at line: 4
2017-12-12 19:04:45,113 INFO: Dec 12 19:04:45 undercloud-0.local ip6tables.init[13122]: Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.
2017-12-12 19:04:45,113 INFO: Dec 12 19:04:45 undercloud-0.local ip6tables.init[13122]: [FAILED]
2017-12-12 19:04:45,114 INFO: Dec 12 19:04:45 undercloud-0.local systemd[1]: ip6tables.service: main process exited, code=exited, status=1/FAILURE
2017-12-12 19:04:45,114 INFO: Dec 12 19:04:45 undercloud-0.local systemd[1]: Failed to start IPv6 firewall with ip6tables.
2017-12-12 19:04:45,114 INFO: Dec 12 19:04:45 undercloud-0.local systemd[1]: Unit ip6tables.service entered failed state.
2017-12-12 19:04:45,114 INFO: Dec 12 19:04:45 undercloud-0.local systemd[1]: ip6tables.service failed.
2017-12-12 19:04:45,114 INFO: ESC[0m
2017-12-12 19:04:45,114 INFO: ESC[1;31mError: /Stage[main]/Firewall::Linux::Redhat/Service[ip6tables]/ensure: change from stopped to running failed: Systemd start for ip6tables failed!
Comment 20 Alex Schultz 2018-12-13 16:28:09 UTC 
Closing as there was a patch upstream to address this and has been merged for some time.