Bug 1119351

Summary: Installing on a system with ipv6 issues causes odd failures
Product: Red Hat Satellite Reporter: Justin Sherrill <jsherril>
Component: InstallationAssignee: Justin Sherrill <jsherril>
Status: CLOSED CURRENTRELEASE QA Contact: Tazim Kolhar <tkolhar>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.0.3CC: bbuckingham, cshao, cwelton, dsulliva, fdeutsch, hartsjc, huiwa, jmontleo, leiwang, mmccune, shughes, tkolhar, xdmoon, ycui
Target Milestone: UnspecifiedKeywords: Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
URL: http://projects.theforeman.org/issues/7145
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-09-11 12:26:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1115190    

Description Justin Sherrill 2014-07-14 15:05:38 UTC
Description of problem:

When installing katello if ipv6 is enabled but the network is not configured properly the install will fail with these errors:

[ERROR 2014-07-08 08:41:52 main]  Could not start Service[elasticsearch]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait elasticsearch start' returned 5: Starting elasticsearch: [  OK  ]


[ERROR 2014-07-08 08:45:35 main]  Could not start Service[tomcat6]: Execution of '/usr/share/katello-installer/modules/service_wait/bin/service-wait tomcat6 start' returned 5: Starting tomcat6: [  OK  ]


The result of:

ping6 localhost 

is:
"socket: Address family not supported by protocol" 


The root cause is that in the installer's service-wait script, wget is used to retry the connection on started services for 30 seconds.  It uses 'localhost' which resolves for both ipv4 and ipv6 and so the wget fails (with the same 'Address family not supported by protocol') initially and does not retry.

In this user's case commenting out the ipv6 localhost from /etc/hosts allowed the installer to run completely. 

How reproducible:

Not entirely sure, but somehow ipv6 needs to be enabled but the network needs to not support it.

Steps to Reproduce:
1.  Get into the state above 
2.  Attempt to install katello


Actual results:
Partially installs with errors.

Expected results:
Should either install correctly, or immediately error telling me that ipv6 is not supported and should be disabled.

Comment 2 James Hartsock 2014-07-14 21:50:32 UTC
=== In Red Hat Customer Portal Case 01140979 ===
--- Comment by Hartsock, James on 7/14/2014 4:49 PM ---

With ::1 commented out of /etc/hosts we can see that wget retries until it gets connected (I removed the -q so we can see output)
~~~
# sed -i 's/^::/#::/' /etc/hosts

# service tomcat6 restart ; /usr/bin/wget --timeout=1 --tries=30 --retry-connrefused --no-check-certificate https://localhost:8443/candlepin/status
Stopping tomcat6:                                          [  OK  ]
Starting tomcat6:                                          [  OK  ]
--2014-07-14 21:42:45--  https://localhost:8443/candlepin/status
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:8443... failed: Connection refused.
Retrying.

--2014-07-14 21:42:46--  (try: 2)  https://localhost:8443/candlepin/status
Connecting to localhost|127.0.0.1|:8443... failed: Connection refused.
Retrying.

--2014-07-14 21:42:48--  (try: 3)  https://localhost:8443/candlepin/status
Connecting to localhost|127.0.0.1|:8443... connected.
Unable to establish SSL connection.
~~~


Now with ::1 in /etc/hosts, but ipv6 disabled we can see wget fails and aborts immediately.
~~~
# grep ^:: /etc/hosts
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

# service tomcat6 restart ; /usr/bin/wget --timeout=1 --tries=30 --retry-connrefused --no-check-certificate https://localhost:8443/candlepin/status ; echo -e "\n$?"
Stopping tomcat6:                                          [  OK  ]
Starting tomcat6:                                          [  OK  ]
--2014-07-14 21:44:16--  https://localhost:8443/candlepin/status
Resolving localhost... 127.0.0.1, ::1
Connecting to localhost|127.0.0.1|:8443... failed: Connection refused.
Connecting to localhost|::1|:8443... failed: Address family not supported by protocol.
~~~



Here is an strace for the socket() call and can see it is the PF_INET6 port that is causing the issue
~~~
# service tomcat6 restart ; strace -e socket /usr/bin/wget --timeout=1 --tries=30 --retry-connrefused --no-check-certificate https://localhost:8443/candlepin/status
Stopping tomcat6:                                          [  OK  ]
Starting tomcat6:                                          [  OK  ]
--2014-07-14 21:47:19--  https://localhost:8443/candlepin/status
Resolving localhost... socket(PF_NETLINK, SOCK_RAW, 0)         = 3
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 3
socket(PF_INET6, SOCK_DGRAM, IPPROTO_IP) = -1 EAFNOSUPPORT (Address family not supported by protocol)
127.0.0.1, ::1
Connecting to localhost|127.0.0.1|:8443... socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
failed: Connection refused.
Connecting to localhost|::1|:8443... socket(PF_INET6, SOCK_STREAM, IPPROTO_IP) = -1 EAFNOSUPPORT (Address family not supported by protocol)
failed: Address family not supported by protocol.
~~~






Another solution would be to limit wget to IPv4 address only by using the -4 option
~~~
# service tomcat6 restart ; /usr/bin/wget -4 --timeout=1 --tries=30 --retry-connrefused --no-check-certificate https://localhost:8443/candlepin/status
Stopping tomcat6:                                          [  OK  ]
Starting tomcat6:                                          [  OK  ]
--2014-07-14 21:48:26--  https://localhost:8443/candlepin/status
Resolving localhost... 127.0.0.1, 127.0.0.1
Connecting to localhost|127.0.0.1|:8443... failed: Connection refused.
Connecting to localhost|127.0.0.1|:8443... failed: Connection refused.
Retrying.

--2014-07-14 21:48:27--  (try: 2)  https://localhost:8443/candlepin/status
Connecting to localhost|127.0.0.1|:8443... failed: Connection refused.
Connecting to localhost|127.0.0.1|:8443... failed: Connection refused.
Retrying.

--2014-07-14 21:48:29--  (try: 3)  https://localhost:8443/candlepin/status
Connecting to localhost|127.0.0.1|:8443... connected.
Unable to establish SSL connection.
~~~

Comment 3 Justin Sherrill 2014-08-18 20:19:13 UTC
Easy way to reproduce:

In /etc/sysconfig/network-scripts/ifcfg-eth0 add:

IPV6INIT=no


add reboot

Comment 4 Justin Sherrill 2014-08-18 23:58:20 UTC
Created redmine issue http://projects.theforeman.org/issues/7145 from this bug

Comment 5 Justin Sherrill 2014-08-19 00:07:26 UTC
https://github.com/Katello/puppet-service_wait/pull/3

Comment 6 Justin Sherrill 2014-08-19 00:08:28 UTC
Steps to reproduce:

1.  In /etc/sysconfig/network-scripts/ifcfg-eth0 add:

IPV6INIT=no

2.  Reboot
3.  Confirm ipv6 is disabled:

[root@katello katello-installer]# ping6 localhost
socket: Address family not supported by protocol

4.  Install katello

Comment 7 Bryan Kearney 2014-08-19 22:03:22 UTC
Moving to POST since upstream bug http://projects.theforeman.org/issues/7145 has been closed
-------------
Justin Sherrill
Applied in changeset commit:katello-installer|b45872663ee1dbdd45fa6ce1a37aa689ca40b1b3.

Comment 10 Tazim Kolhar 2014-08-28 06:12:38 UTC
VERIFIED:

ipv6 is disabled
katello install works

Comment 11 Bryan Kearney 2014-09-11 12:26:38 UTC
This was delivered with Satellite 6.0 which was released on 10 September 2014.