Bug 1298533 - fusor-undercloud-installer freezes when rerun with different IP
fusor-undercloud-installer freezes when rerun with different IP
Status: CLOSED UPSTREAM
Product: Red Hat Quickstart Cloud Installer
Classification: Red Hat
Component: fusor-installer (Show other bugs)
1.0
Unspecified Unspecified
unspecified Severity unspecified
: ga
: 1.0
Assigned To: Jason Montleon
Tasos Papaioannou
: Triaged
Depends On:
Blocks: rhci-sprint-16
  Show dependency treegraph
 
Reported: 2016-01-14 06:32 EST by Antonin Pagac
Modified: 2016-08-12 16:58 EDT (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-08-05 11:38:26 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
fusor-undercloud-install.log (244.41 KB, text/plain)
2016-07-12 16:15 EDT, Tasos Papaioannou
no flags Details

  None (edit)
Description Antonin Pagac 2016-01-14 06:32:05 EST
Description of problem:
When re-running fusor-undercloud-installer and using different IP on prompts (192.1.x.x/24) the installer freezes with 'connection timed out' errors. It seems that the connection is created using old IP:

ERROR: openstack Command 'instack-install-undercloud' returned non-zero exit status 1
Unable to establish connection to http://192.0.2.1:9696/v2.0/networks.json?fields=id&name=ctlplane
ERROR (shell) Exiting due to an error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 248, in main
    TuskarShell(sys.argv[1:]).run()
  File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 76, in run
    args.func(tuskar_client, args)
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans_shell.py", line 43, in do_plan_show
    plan = utils.find_resource(tuskar.plans, args.plan)
  File "/usr/lib/python2.7/site-packages/tuskarclient/common/utils.py", line 81, in find_resource
    listing = manager.list()
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans.py", line 86, in list
    return self._list(self._path())
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/base.py", line 131, in _list
    body = self.client.get(url).json()
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 355, in get
    return self.client_request("GET", url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 345, in client_request
    self, method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 261, in client_request
    method, self.concat_url(endpoint, url), **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 189, in request
    resp = self.http.request(method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(110, 'Connection timed out'))
ERROR (shell) Exiting due to an error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 248, in main
    TuskarShell(sys.argv[1:]).run()
  File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 76, in run
    args.func(tuskar_client, args)
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans_shell.py", line 131, in do_plan_create
    description=vars(args).get('description')
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans.py", line 97, in create
    return self._post(self._path(), fields)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/base.py", line 178, in _post
    body = self.client.post(url, json=json).json()
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 358, in post
    return self.client_request("POST", url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 345, in client_request
    self, method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 261, in client_request
    method, self.concat_url(endpoint, url), **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 189, in request
    resp = self.http.request(method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(110, 'Connection timed out'))
ERROR (shell) Exiting due to an error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 248, in main
    TuskarShell(sys.argv[1:]).run()
  File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 76, in run
    args.func(tuskar_client, args)
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans_shell.py", line 43, in do_plan_show
    plan = utils.find_resource(tuskar.plans, args.plan)
  File "/usr/lib/python2.7/site-packages/tuskarclient/common/utils.py", line 81, in find_resource
    listing = manager.list()
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans.py", line 86, in list
    return self._list(self._path())
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/base.py", line 131, in _list
    body = self.client.get(url).json()
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 355, in get
    return self.client_request("GET", url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 345, in client_request
    self, method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 261, in client_request
    method, self.concat_url(endpoint, url), **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 189, in request
    resp = self.http.request(method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(110, 'Connection timed out'))
ERROR (shell) Exiting due to an error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 248, in main
    TuskarShell(sys.argv[1:]).run()
  File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 76, in run
    args.func(tuskar_client, args)
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/roles_shell.py", line 24, in do_role_list
    roles = tuskar.roles.list()
  File "/usr/lib/python2.7/site-packages/tuskarclient/v2/roles.py", line 47, in list
    return self._list(self._path())
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/base.py", line 131, in _list
    body = self.client.get(url).json()
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 355, in get
    return self.client_request("GET", url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 345, in client_request
    self, method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 261, in client_request
    method, self.concat_url(endpoint, url), **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 189, in request
    resp = self.http.request(method, url, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send
    raise ConnectionError(err, request=request)
ConnectionError: ('Connection aborted.', error(110, 'Connection timed out'))


Version-Release number of selected component (if applicable):
RHCI-6.0-RHEL-7-20160107.t.1-RHCI-x86_64-dvd1.iso
RHCIOOO-7-RHEL-7-20151208.t.0-RHCIOOO-x86_64-dvd1.iso

How reproducible:
?

Steps to Reproduce:
1. Run fusor-undercloud-installer, let it finish
2. Run the installer again, but choose different IP
3.

Actual results:
fusor-undercloud-installer freezes, tries to connect to the old IP

Expected results:
Installer finishes successfully with new IP

Additional info:
Comment 1 Jason Montleon 2016-03-21 12:24:41 EDT
Do you still see this? Believe it or not, 192.0.2.1 was hardcoded in the egon gem. I put in a temporary fix to egon until they fix it in the way they desire to.
Comment 2 Antonin Pagac 2016-03-22 06:57:14 EDT
Jason,

for TP3 RC1: I was able to run fusor-undercloud-installer on 192.0.4.0 network without problems, it finished successfully and I'm now registering nodes.
That said, this was first run of the installer, not re-run as stated in the original issue. I will also attempt re-running later on.
Comment 3 Jason Montleon 2016-04-25 09:46:37 EDT
Given what I believe was the problem in C1 I believe this should be fixed. Moving to ONQA so it can be tested.
Comment 4 Tasos Papaioannou 2016-07-12 16:15 EDT
Created attachment 1178992 [details]
fusor-undercloud-install.log
Comment 5 Tasos Papaioannou 2016-07-12 16:15:36 EDT
There still appears to be a problem when re-running fusor-undercloud-installer and choosing a different IP address. During the first run I chose 192.168.101.1 on 192.168.101.0/24, then on the second run 192.168.103.1 / 192.168.103.0/24, and the second run failed with:

---
Unable to establish connection to http://192.168.101.1:5000/v2.0/tokens
usage: neutron subnet-update [-h] [--request-format {json,xml}] [--name NAME]
                             [--gateway GATEWAY_IP | --no-gateway]
                             [--allocation-pool start=IP_ADDR,end=IP_ADDR]
                             [--host-route destination=CIDR,nexthop=IP_ADDR]
                             [--dns-nameserver DNS_NAMESERVER]
                             [--disable-dhcp] [--enable-dhcp]
                             SUBNET
neutron subnet-update: error: too few arguments
----

The full fusor-undercloud-install.log is attached. It looks like the following section is relevant:


----
^[[mNotice: Finished catalog run in 1366.36 seconds^[[0m
+ rc=6
+ set -e
+ echo 'puppet apply exited with exit code 6'
puppet apply exited with exit code 6
+ '[' 6 '!=' 2 -a 6 '!=' 0 ']'
+ exit 6
[2016-07-12 15:36:59,102] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 6]

[2016-07-12 15:36:59,103] (os-refresh-config) [ERROR] Aborting...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 815, in install
    _run_orc(instack_env)
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 699, in _run_orc
    _run_live_command(args, instack_env, 'os-refresh-config')
  File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 370, in _run_live_command
    raise RuntimeError('%s failed. See log for details.' % name)
RuntimeError: os-refresh-config failed. See log for details.
Command 'instack-install-undercloud' returned non-zero exit status 1
----
Comment 6 Jason Montleon 2016-07-15 11:18:00 EDT
This may be a limitation of the instack installer in OSP-D. Possibly related to https://bugzilla.redhat.com/show_bug.cgi?id=1238483
Comment 12 Jason Montleon 2016-08-05 11:38:26 EDT
This is still a problem in OSP 8, though the symptoms are different.

Filed https://bugzilla.redhat.com/show_bug.cgi?id=1364528 and adding to our OSP tracker.

Note You need to log in before you can comment on or make changes to this bug.