Bug 1298533

Summary: fusor-undercloud-installer freezes when rerun with different IP
Product: Red Hat Quickstart Cloud Installer Reporter: Antonin Pagac <apagac>
Component: fusor-installerAssignee: Jason Montleon <jmontleo>
Status: CLOSED UPSTREAM QA Contact: Tasos Papaioannou <tpapaioa>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 1.0CC: apagac, dymurray, jmatthew, jmontleo, smallamp, tpapaioa, tsanders
Target Milestone: gaKeywords: Triaged
Target Release: 1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-08-05 15:38:26 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1321046    
Attachments:
Description Flags
fusor-undercloud-install.log none

Description Antonin Pagac 2016-01-14 11:32:05 UTC 
Description of problem:
When re-running fusor-undercloud-installer and using different IP on prompts (192.1.x.x/24) the installer freezes with 'connection timed out' errors. It seems that the connection is created using old IP:

ERROR: openstack Command 'instack-install-undercloud' returned non-zero exit status 1
Unable to establish connection to http://192.0.2.1:9696/v2.0/networks.json?fields=id&name=ctlplane
ERROR (shell) Exiting due to an error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 248, in main
    TuskarShell(sys.argv[1:]).run()
  File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 76, in run
    args.func(tuskar_client, args)
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans_shell.py", line 43, in do_plan_show
    plan = utils.find_resource(tuskar.plans, args.plan)
  File "/usr/lib/python2.7/site-packages/tuskarclient/common/utils.py", line 81, in find_resource
    listing = manager.list()
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans.py", line 86, in list
    return self._list(self._path())
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/base.py", line 131, in _list
    body = self.client.get(url).json()
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 355, in get
    return self.client_request("GET", url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 345, in client_request
    self, method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 261, in client_request
    method, self.concat_url(endpoint, url), **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 189, in request
    resp = self.http.request(method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(110, 'Connection timed out'))
ERROR (shell) Exiting due to an error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 248, in main
    TuskarShell(sys.argv[1:]).run()
  File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 76, in run
    args.func(tuskar_client, args)
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans_shell.py", line 131, in do_plan_create
    description=vars(args).get('description')
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans.py", line 97, in create
    return self._post(self._path(), fields)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/base.py", line 178, in _post
    body = self.client.post(url, json=json).json()
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 358, in post
    return self.client_request("POST", url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 345, in client_request
    self, method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 261, in client_request
    method, self.concat_url(endpoint, url), **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 189, in request
    resp = self.http.request(method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(110, 'Connection timed out'))
ERROR (shell) Exiting due to an error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 248, in main
    TuskarShell(sys.argv[1:]).run()
  File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 76, in run
    args.func(tuskar_client, args)
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans_shell.py", line 43, in do_plan_show
    plan = utils.find_resource(tuskar.plans, args.plan)
  File "/usr/lib/python2.7/site-packages/tuskarclient/common/utils.py", line 81, in find_resource
    listing = manager.list()
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans.py", line 86, in list
    return self._list(self._path())
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/base.py", line 131, in _list
    body = self.client.get(url).json()
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 355, in get
    return self.client_request("GET", url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 345, in client_request
    self, method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 261, in client_request
    method, self.concat_url(endpoint, url), **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 189, in request
    resp = self.http.request(method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(110, 'Connection timed out'))
ERROR (shell) Exiting due to an error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 248, in main
    TuskarShell(sys.argv[1:]).run()
  File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 76, in run
    args.func(tuskar_client, args)
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/roles_shell.py", line 24, in do_role_list
    roles = tuskar.roles.list()
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/roles.py", line 47, in list
    return self._list(self._path())
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/base.py", line 131, in _list
    body = self.client.get(url).json()
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 355, in get
    return self.client_request("GET", url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 345, in client_request
    self, method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 261, in client_request
    method, self.concat_url(endpoint, url), **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 189, in request
    resp = self.http.request(method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(110, 'Connection timed out'))


Version-Release number of selected component (if applicable):
RHCI-6.0-RHEL-7-20160107.t.1-RHCI-x86_64-dvd1.iso
RHCIOOO-7-RHEL-7-20151208.t.0-RHCIOOO-x86_64-dvd1.iso

How reproducible:
?

Steps to Reproduce:
1. Run fusor-undercloud-installer, let it finish
2. Run the installer again, but choose different IP
3.

Actual results:
fusor-undercloud-installer freezes, tries to connect to the old IP

Expected results:
Installer finishes successfully with new IP

Additional info:
Comment 1 Jason Montleon 2016-03-21 16:24:41 UTC 
Do you still see this? Believe it or not, 192.0.2.1 was hardcoded in the egon gem. I put in a temporary fix to egon until they fix it in the way they desire to.
Comment 2 Antonin Pagac 2016-03-22 10:57:14 UTC 
Jason,

for TP3 RC1: I was able to run fusor-undercloud-installer on 192.0.4.0 network without problems, it finished successfully and I'm now registering nodes.
That said, this was first run of the installer, not re-run as stated in the original issue. I will also attempt re-running later on.
Comment 3 Jason Montleon 2016-04-25 13:46:37 UTC 
Given what I believe was the problem in C1 I believe this should be fixed. Moving to ONQA so it can be tested.
Comment 4 Tasos Papaioannou 2016-07-12 20:15:02 UTC 
Created attachment 1178992 [details]
fusor-undercloud-install.log
Comment 5 Tasos Papaioannou 2016-07-12 20:15:36 UTC 
There still appears to be a problem when re-running fusor-undercloud-installer and choosing a different IP address. During the first run I chose 192.168.101.1 on 192.168.101.0/24, then on the second run 192.168.103.1 / 192.168.103.0/24, and the second run failed with:

---
Unable to establish connection to http://192.168.101.1:5000/v2.0/tokens
usage: neutron subnet-update [-h] [--request-format {json,xml}] [--name NAME]
                             [--gateway GATEWAY_IP | --no-gateway]
                             [--allocation-pool start=IP_ADDR,end=IP_ADDR]
                             [--host-route destination=CIDR,nexthop=IP_ADDR]
                             [--dns-nameserver DNS_NAMESERVER]
                             [--disable-dhcp] [--enable-dhcp]
                             SUBNET
neutron subnet-update: error: too few arguments
----

The full fusor-undercloud-install.log is attached. It looks like the following section is relevant:


----
^[[mNotice: Finished catalog run in 1366.36 seconds^[[0m
+ rc=6
+ set -e
+ echo 'puppet apply exited with exit code 6'
puppet apply exited with exit code 6
+ '[' 6 '!=' 2 -a 6 '!=' 0 ']'
+ exit 6
[2016-07-12 15:36:59,102] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 6]

[2016-07-12 15:36:59,103] (os-refresh-config) [ERROR] Aborting...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 815, in install
    _run_orc(instack_env)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 699, in _run_orc
    _run_live_command(args, instack_env, 'os-refresh-config')
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 370, in _run_live_command
    raise RuntimeError('%s failed. See log for details.' % name)
RuntimeError: os-refresh-config failed. See log for details.
Command 'instack-install-undercloud' returned non-zero exit status 1
----
Comment 6 Jason Montleon 2016-07-15 15:18:00 UTC 
This may be a limitation of the instack installer in OSP-D. Possibly related to https://bugzilla.redhat.com/show_bug.cgi?id=1238483
Comment 12 Jason Montleon 2016-08-05 15:38:26 UTC 
This is still a problem in OSP 8, though the symptoms are different.

Filed https://bugzilla.redhat.com/show_bug.cgi?id=1364528 and adding to our OSP tracker.