Description of problem: When re-running fusor-undercloud-installer and using different IP on prompts (192.1.x.x/24) the installer freezes with 'connection timed out' errors. It seems that the connection is created using old IP: ERROR: openstack Command 'instack-install-undercloud' returned non-zero exit status 1 Unable to establish connection to http://192.0.2.1:9696/v2.0/networks.json?fields=id&name=ctlplane ERROR (shell) Exiting due to an error: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 248, in main TuskarShell(sys.argv[1:]).run() File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 76, in run args.func(tuskar_client, args) File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans_shell.py", line 43, in do_plan_show plan = utils.find_resource(tuskar.plans, args.plan) File "/usr/lib/python2.7/site-packages/tuskarclient/common/utils.py", line 81, in find_resource listing = manager.list() File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans.py", line 86, in list return self._list(self._path()) File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/base.py", line 131, in _list body = self.client.get(url).json() File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 355, in get return self.client_request("GET", url, **kwargs) File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 345, in client_request self, method, url, **kwargs) File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 261, in client_request method, self.concat_url(endpoint, url), **kwargs) File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 189, in request resp = self.http.request(method, url, **kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send raise ConnectionError(err, request=request) ConnectionError: ('Connection aborted.', error(110, 'Connection timed out')) ERROR (shell) Exiting due to an error: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 248, in main TuskarShell(sys.argv[1:]).run() File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 76, in run args.func(tuskar_client, args) File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans_shell.py", line 131, in do_plan_create description=vars(args).get('description') File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans.py", line 97, in create return self._post(self._path(), fields) File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/base.py", line 178, in _post body = self.client.post(url, json=json).json() File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 358, in post return self.client_request("POST", url, **kwargs) File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 345, in client_request self, method, url, **kwargs) File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 261, in client_request method, self.concat_url(endpoint, url), **kwargs) File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 189, in request resp = self.http.request(method, url, **kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send raise ConnectionError(err, request=request) ConnectionError: ('Connection aborted.', error(110, 'Connection timed out')) ERROR (shell) Exiting due to an error: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 248, in main TuskarShell(sys.argv[1:]).run() File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 76, in run args.func(tuskar_client, args) File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans_shell.py", line 43, in do_plan_show plan = utils.find_resource(tuskar.plans, args.plan) File "/usr/lib/python2.7/site-packages/tuskarclient/common/utils.py", line 81, in find_resource listing = manager.list() File "/usr/lib/python2.7/site-packages/tuskarclient/v2/plans.py", line 86, in list return self._list(self._path()) File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/base.py", line 131, in _list body = self.client.get(url).json() File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 355, in get return self.client_request("GET", url, **kwargs) File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 345, in client_request self, method, url, **kwargs) File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 261, in client_request method, self.concat_url(endpoint, url), **kwargs) File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 189, in request resp = self.http.request(method, url, **kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send raise ConnectionError(err, request=request) ConnectionError: ('Connection aborted.', error(110, 'Connection timed out')) ERROR (shell) Exiting due to an error: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 248, in main TuskarShell(sys.argv[1:]).run() File "/usr/lib/python2.7/site-packages/tuskarclient/shell.py", line 76, in run args.func(tuskar_client, args) File "/usr/lib/python2.7/site-packages/tuskarclient/v2/roles_shell.py", line 24, in do_role_list roles = tuskar.roles.list() File "/usr/lib/python2.7/site-packages/tuskarclient/v2/roles.py", line 47, in list return self._list(self._path()) File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/base.py", line 131, in _list body = self.client.get(url).json() File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 355, in get return self.client_request("GET", url, **kwargs) File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 345, in client_request self, method, url, **kwargs) File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 261, in client_request method, self.concat_url(endpoint, url), **kwargs) File "/usr/lib/python2.7/site-packages/tuskarclient/openstack/common/apiclient/client.py", line 189, in request resp = self.http.request(method, url, **kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 465, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 573, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 415, in send raise ConnectionError(err, request=request) ConnectionError: ('Connection aborted.', error(110, 'Connection timed out')) Version-Release number of selected component (if applicable): RHCI-6.0-RHEL-7-20160107.t.1-RHCI-x86_64-dvd1.iso RHCIOOO-7-RHEL-7-20151208.t.0-RHCIOOO-x86_64-dvd1.iso How reproducible: ? Steps to Reproduce: 1. Run fusor-undercloud-installer, let it finish 2. Run the installer again, but choose different IP 3. Actual results: fusor-undercloud-installer freezes, tries to connect to the old IP Expected results: Installer finishes successfully with new IP Additional info:
Do you still see this? Believe it or not, 192.0.2.1 was hardcoded in the egon gem. I put in a temporary fix to egon until they fix it in the way they desire to.
Jason, for TP3 RC1: I was able to run fusor-undercloud-installer on 192.0.4.0 network without problems, it finished successfully and I'm now registering nodes. That said, this was first run of the installer, not re-run as stated in the original issue. I will also attempt re-running later on.
Given what I believe was the problem in C1 I believe this should be fixed. Moving to ONQA so it can be tested.
Created attachment 1178992 [details] fusor-undercloud-install.log
There still appears to be a problem when re-running fusor-undercloud-installer and choosing a different IP address. During the first run I chose 192.168.101.1 on 192.168.101.0/24, then on the second run 192.168.103.1 / 192.168.103.0/24, and the second run failed with: --- Unable to establish connection to http://192.168.101.1:5000/v2.0/tokens usage: neutron subnet-update [-h] [--request-format {json,xml}] [--name NAME] [--gateway GATEWAY_IP | --no-gateway] [--allocation-pool start=IP_ADDR,end=IP_ADDR] [--host-route destination=CIDR,nexthop=IP_ADDR] [--dns-nameserver DNS_NAMESERVER] [--disable-dhcp] [--enable-dhcp] SUBNET neutron subnet-update: error: too few arguments ---- The full fusor-undercloud-install.log is attached. It looks like the following section is relevant: ---- ^[[mNotice: Finished catalog run in 1366.36 seconds^[[0m + rc=6 + set -e + echo 'puppet apply exited with exit code 6' puppet apply exited with exit code 6 + '[' 6 '!=' 2 -a 6 '!=' 0 ']' + exit 6 [2016-07-12 15:36:59,102] (os-refresh-config) [ERROR] during configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/configure.d']' returned non-zero exit status 6] [2016-07-12 15:36:59,103] (os-refresh-config) [ERROR] Aborting... Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 815, in install _run_orc(instack_env) File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 699, in _run_orc _run_live_command(args, instack_env, 'os-refresh-config') File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 370, in _run_live_command raise RuntimeError('%s failed. See log for details.' % name) RuntimeError: os-refresh-config failed. See log for details. Command 'instack-install-undercloud' returned non-zero exit status 1 ----
This may be a limitation of the instack installer in OSP-D. Possibly related to https://bugzilla.redhat.com/show_bug.cgi?id=1238483
This is still a problem in OSP 8, though the symptoms are different. Filed https://bugzilla.redhat.com/show_bug.cgi?id=1364528 and adding to our OSP tracker.