When running openstack undercloud install a second time, it fails with the error below. [2015/06/05 02:48:42 PM] [WARNING] DEPRECATED: falling back to /var/run/os-collect-config/os_config_files.json + NETWORK_GATEWAY=172.16.173.1 + METADATA_SERVER=10.10.8.1 + PHYSICAL_NETWORK=ctlplane ++ mktemp + NETWORK_JSON=/tmp/tmp.R3PuH2T5UD + jq . + setup-neutron -n /tmp/tmp.R3PuH2T5UD /usr/lib/python2.7/site-packages/novaclient/v1_1/__init__.py:30: UserWarning: Module novaclient.v1_1 is deprecated (taken as a basis for novaclient.v2). The preferable way to get client class or object you can find in novaclient.client module. warnings.warn("Module novaclient.v1_1 is deprecated (taken as a basis for " 2015-06-05 14:48:43 - root - ERROR - Unexpected error during command execution Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/os_cloud_config/cmd/setup_neutron.py", line 77, in main keystone_client=keystone_client) File "/usr/lib/python2.7/site-packages/os_cloud_config/neutron.py", line 46, in initialize_neutron net = _create_net(neutron_client, network_desc, network_type, admin_tenant) File "/usr/lib/python2.7/site-packages/os_cloud_config/neutron.py", line 95, in _create_net return neutron.create_network({'network': network}) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 102, in with_params ret = self.function(instance, *args, **kwargs) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 571, in create_network return self.post(self.networks_path, body=body) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 298, in post headers=headers, params=params) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 211, in do_request self._handle_fault_response(status_code, replybody) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 185, in _handle_fault_response exception_handler_v20(status_code, des_error_body) File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 70, in exception_handler_v20 status_code=status_code) Conflict: Unable to create the flat network. Physical network ctlplane is in use. [2015-06-05 14:48:43,419] (os-refresh-config) [ERROR] during post-configure phase. [Command '['dib-run-parts', '/usr/libexec/os-refresh-config/post-configure.d']' returned non-zero exit status 1] [2015-06-05 14:48:43,420] (os-refresh-config) [ERROR] Aborting... ERROR: openstack Command 'instack-install-undercloud' returned non-zero exit status 1
I saw the same error on the rdo-list [1]. If ctlplane network is in use because neutron ports were allocated on the assigned subnet because it's a 2nd run through of instack-install-undercloud after a failed deployment [2], then could instack-install-undercloud have a --force-clean option so that re-kicking the box is not necessary? Is there a simpler workaround than re-starting clean? John [1] https://www.redhat.com/archives/rdo-list/2015-April/msg00169.html [2] In this case the install failed the first time because of reasons documented in bz1228438
Is deleting the ctlplane network (as shown by `neutron net-list`) via `neutron net-delete` on the undercloud node an alternative workaround that's easier than re-starting clean?
Keith, would you mind looking into this to see if you want the feature added for A1? John, James will add info for a workaround.
Any update on the workaround James?
delete the subnet first: neutron subnet-list neutron subnet-delete <subnet-uuid> neutron net-delete ctlplane that should get you around this issue.
what is the behavior of --force-clean that is being requested? should it completely uninstall/reinstall the undercloud (would obviously lose any ability to manage a deployed overcloud).
The desired behavior of the --force-clean option that is being requested is that it should completely uninstall/reinstall the undercloud. Thus, if the physical network ctlplane was in use, it would do something like the following: neutron subnet-list neutron subnet-delete <subnet-uuid> neutron net-delete ctlplane But it would also remove other changes that came about from installing the undercloud which might get in the way when `openstack undercloud install` is run on top of a partial installation. The issue is that it is time consuming for someone installing an undercloud to have to re-install RHEL, just to be sure anything that the undercloud put in place is removed. You said "[you] would obviously lose any ability to manage a deployed overcloud". If the undercould isn't yet installed, why would we have an overcloud to loose the ability to manage? This usecase would apply to a situation where the overcloud doesn't yet exist.
The goal of the installer is to be re-runnable without having to manually clean anything up. The specific issue that caused the error message about the neutron network being in use has been fixed (bug 1228438). We can also account for the neutron network possibly being in use due to another future unexpected failure from a previous install attempt in 7.1 (that's tracked by this bug). We can do without the need for a --force-clean option that uninstalls and cleans up any system changes that were made by the installer. To accomplish the use case that was driving the request of that option, we will document how to make a change to the undercloud.conf configuration file and rerun the installer so that the requested changes are applied. If further testing of the documented solution uncovers additional bugs with this method, we can address those as needed. The recommended solution for a true --force-clean option that will completely reset the system to a pristine state is to run the undercloud in a vm and use vm snapshots to rollback to a known good state.
Please note that there is another use case when this occurs. When doing backups and restoration of the undercloud, you restore the configuration files and databases, before finally running openstack undercloud install so that it will recreate the ovs bridges etc. Overall what needs to happen is two things * openstack undercloud install by default needs to be fully idempotent. No matter what, I should be able to run it multiple times and it shouldn't fail (it should detect everything is done/correct and not do anything, only making the changes it needs to do. This would solve the case where you are restoring the undercloud from backup (deleting the ctlplane network in this case is not an option as you want the ctlplane network to have the same uuid in neutron as it always has, not changed) * openstack cli needs a --force-cleanup or something similar in the cases where you do want it to trash all existing data and completely start from a clean slate (for whatever reason) Regards, Graeme
So as user I figured it may be worthwhile to weigh in my €0.02 - a --force-cleanup option would be highly desirable! I've been re-kickstarting my RDO lab box several times because you have no clue in what state the undercloud deployment is. Upgrading RPMs which require an altered database will also put you in a spot where dropping the databases itself will only get the tables re-created, but the tables itself are not filled with the data the undercloud expects. I figured out that removing /opt/stack/.undercloud-setup and rerunning undercloud install is a way to get the tables filled again, but it's quite time consuming to figure out this stuff by yourself. So yeah, being able to consistently start from scratch, especially in a lab environment, is very useful indeed.
(In reply to Graeme Gillies from comment #15) > Please note that there is another use case when this occurs. > > When doing backups and restoration of the undercloud, you restore the > configuration files and databases, before finally running openstack > undercloud install so that it will recreate the ovs bridges etc. > > Overall what needs to happen is two things > > * openstack undercloud install by default needs to be fully idempotent. No > matter what, I should be able to run it multiple times and it shouldn't fail > (it should detect everything is done/correct and not do anything, only > making the changes it needs to do. This would solve the case where you are > restoring the undercloud from backup (deleting the ctlplane network in this > case is not an option as you want the ctlplane network to have the same uuid > in neutron as it always has, not changed) you may like to file a rfe for this if there isn't one already. I'm in agreement with the spirit of what you're asking for, but not sure it's quite this simple. There are valid times I'd think the installer should stop running and not apply requested changes. Such as if the ctlplane is in use (as you say), yet someone is mistakingly trying to change the subnet IP range via undercloud.conf and rerunning the installer. > > * openstack cli needs a --force-cleanup or something similar in the cases > where you do want it to trash all existing data and completely start from a > clean slate (for whatever reason) i'm not convinced a --force-cleanup flag is the correct technical solution to the feature you're requesting. I'd like to focus on the feature, and not the requested technical implementation. It sounds like the feature is "trash all existing data and completely start from a clean slate". We do already have superior solutions to that, far more effective than any programmatic/scripted (puppet, whatever) based solution could ever accomplish. One such wrinkle is that puppet knows about the resources it's told about. Suppose wrong repos were enabled, packages/deps installed from the repos, and an error encountered. My feeling around the expectation of a --force-clean would be that it would completely clean all of that up. That problem is probably solvable with enough code, but I do contend that such a solution might not be the best way to accomplish such a thing. There may very well be existing puppet modules that solve this problem, but it's a question of identifying all such scenarios and solving them all in order to at least say we strive for 100% consistency with a --force-clean. Otherwise, we could say, it's best effort. But, again, that doesn't sound like what you're asking for. I think a container based undercloud could/would offer a --force-clean. So perhaps the answer is wrapped up in that. Anyway, the plan for this particular bugzilla was to fix the reported problem and solve it as I laid out in comment 14. Please file an RFE if you'd like to continue the discussion about these other 2 points in a new bugzilla.
(In reply to James Slagle from comment #17) > (In reply to Graeme Gillies from comment #15) > > Please note that there is another use case when this occurs. > > > > When doing backups and restoration of the undercloud, you restore the > > configuration files and databases, before finally running openstack > > undercloud install so that it will recreate the ovs bridges etc. > > > > Overall what needs to happen is two things > > > > * openstack undercloud install by default needs to be fully idempotent. No > > matter what, I should be able to run it multiple times and it shouldn't fail > > (it should detect everything is done/correct and not do anything, only > > making the changes it needs to do. This would solve the case where you are > > restoring the undercloud from backup (deleting the ctlplane network in this > > case is not an option as you want the ctlplane network to have the same uuid > > in neutron as it always has, not changed) > > you may like to file a rfe for this if there isn't one already. > > I'm in agreement with the spirit of what you're asking for, but not sure > it's quite this simple. There are valid times I'd think the installer should > stop running and not apply requested changes. Such as if the ctlplane is in > use (as you say), yet someone is mistakingly trying to change the subnet IP > range via undercloud.conf and rerunning the installer. Oh yes definitely agreed. > > > > > * openstack cli needs a --force-cleanup or something similar in the cases > > where you do want it to trash all existing data and completely start from a > > clean slate (for whatever reason) > > i'm not convinced a --force-cleanup flag is the correct technical solution > to the feature you're requesting. I'd like to focus on the feature, and not > the requested technical implementation. It sounds like the feature is "trash > all existing data and completely start from a clean slate". > > We do already have superior solutions to that, far more effective than any > programmatic/scripted (puppet, whatever) based solution could ever > accomplish. > > One such wrinkle is that puppet knows about the resources it's told about. > Suppose wrong repos were enabled, packages/deps installed from the repos, > and an error encountered. My feeling around the expectation of a > --force-clean would be that it would completely clean all of that up. That > problem is probably solvable with enough code, but I do contend that such a > solution might not be the best way to accomplish such a thing. There may > very well be existing puppet modules that solve this problem, but it's a > question of identifying all such scenarios and solving them all in order to > at least say we strive for 100% consistency with a --force-clean. > > Otherwise, we could say, it's best effort. But, again, that doesn't sound > like what you're asking for. > > I think a container based undercloud could/would offer a --force-clean. So > perhaps the answer is wrapped up in that. > > Anyway, the plan for this particular bugzilla was to fix the reported > problem and solve it as I laid out in comment 14. > > Please file an RFE if you'd like to continue the discussion about these > other 2 points in a new bugzilla. Agreed also. Your right I probably got too much into the implementation and not expressing the idea. We want to be able to "revert" the undercloud state in some fashion, and as you mentioned, doing that completely correctly is incredibly hard to do without immutable infrastructure patterns, so deferring it until we achieve that (which I believe is on the roadmap) makes sense Regards, Graeme
i still need to add the doc text here
Verified with : ---------------- instack-0.0.7-1.el7ost.noarch instack-undercloud-2.1.2-26.el7ost.noarch After undercloud was installed, I've changed the parameters of undercloud.conf and rerun - openstack undercloud install . Results: ---------- undercloud re-installed successfully and the changes from undercloud.conf were effective.
*** Bug 1266201 has been marked as a duplicate of this bug. ***
moving back to MODIFIED to indicate there's a new build available. the additional verification should be covered by doing a HA deploy on baremetal per: https://bugzilla.redhat.com/show_bug.cgi?id=1266201
although the spec was updated, the patch wasn't included in the build. not sure how that happened, but I didn't notice. rebuilding again into instack-undercloud-2.1.2-29.el7ost
The workaround for this if using an older build is to manually restart the openstack-nova-api service prior to deploying.
Verified with: instack-undercloud-2.1.2-29.el7ost.noarch Deployed the undercloud and then re-deploy it with the same command and then successfully deployed Overcloud HA.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2015:1862