Bug 967369

Summary: RFE packstack should disable NetworkManager
Product: Red Hat OpenStack Reporter: Steven Dake <sdake>
Component: openstack-puppet-modulesAssignee: Martin Magr <mmagr>
Status: CLOSED NOTABUG QA Contact: Roey Dekel <rdekel>
Severity: high Docs Contact:
Priority: high    
Version: 3.0CC: aberezin, aortega, breeler, danw, dcbw, derekh, fdinitto, hateya, jkt, jlibosva, lyarwood, mkollaro, mmagr, mnewby, oblaut, rkhan, rkukura, sdake, sgordon, tgraf, twilson, yeylon
Target Milestone: rcKeywords: FutureFeature
Target Release: 4.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Enhancement
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-12-04 16:11:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1010316    
Bug Blocks: 975499    

Description Steven Dake 2013-05-26 21:40:05 UTC
Description of problem:
NetworkManager causes all kinds of havoc with OpenStack Networking.

Version-Release number of selected component (if applicable):
openstack-packstack-2013.1.1-0.5.dev538.el6ost

How reproducible:
Haven't got a fully operating system yet

Steps to Reproduce:
1. Install with the RHEL6.4 operation 
2.
3.

Actual results:
In some configurations of RHEL, NetworkManager is enabled by default

Expected results:
NetworkManager should always be disabled

Additional info:
NetworkManager must be disabled if the installation chosen is:
* Desktop
* Software Development Workstation

Disable network manager by running the commands

$ sudo chkconfig NetworkManager off
$ sudo service NetworkManager stop

Modify the network script related to the network interfaces in 
/etc/sysconfig/network-scripts/ifcfg-ethX where X is the interface
1. Remove the line NM_CONTROLLED=yes
2. Change ONBOOT=no to ONBOOT=yes

Comment 2 Ryan O'Hara 2013-05-28 15:46:43 UTC
What problems occur? I'm never disabled NetworkManager and I've never run into problems.

Comment 3 Steven Dake 2013-05-28 17:03:56 UTC
Chris Wright indicated he has run into problems when running with Network Manager.  We had lots of problems with NM with RHCS so we ended up disabling there (in the docs I believe).  I can't totally get my quantum setup working NM or no NM because of a separate problem.

Comment 4 Ryan O'Hara 2013-05-28 18:52:03 UTC
We need more information about what problems occur with NM is enabled. If there are problems, what are they? Under what circumstances does NM need to be disabled. This bug seems to suggest that NM should be disabled for all OpenStack deployments. Is that true, or is this specific to Quantum?

Comment 6 Ryan O'Hara 2013-05-29 16:20:16 UTC
Is this really Quantum specific? What about multi-node OpenStack deployments with, say, multiple Nova compute nodes? My understanding is that NM will need to be disabled there as well. We need to define where NM should be disabled before we can provide a solution.

Comment 18 yeylon@redhat.com 2013-10-01 09:57:07 UTC
so what is the fix here?

packstack should refuse to install with NM, 
should disable NM, 
issue a warning about NM.

Comment 19 Perry Myers 2013-10-01 10:13:57 UTC
It's not clear to me that this is a Packstack bug vs. an openstack-puppet-modules bug, since the same problems would exist when Foreman deploys RHOS

Because both Foreman/Packstack are distributed deployment tools, and expect to be able to arbitrarily reconfigure machines that you are installing, I think that the puppet modules should just disable NetworkManager if they find it enabled.

We can make it clear in a Technical Note that OpenStack is currently incompatible with NetworkManager and that until some bug fixing and further testing is done to validate NetworkManager with OpenStack, that all OpenStack nodes will have NM disabled on them.

Note: This does not necessarily mean that the host running packstack should have NM disabled.  It could be that the host _running_ packstack is just a client machine and is not having any OpenStack services installed on it.  (This is another reason why it makes sense to put disabling NM into puppet vs. packstack itself)

Comment 20 Martin Magr 2013-10-01 10:29:20 UTC
Patch which is currently waiting for review is disabling NetworkManager service (this is done via Puppet) and setting NM_CONTROLLED=no in network script of each interface on hosts where OpenStack components are being installed (this is done in Python part of Packstack).

The second part is probably not required. It will just make sure the interfaces won't be touched by NM even if service is started with some third party software.

Comment 22 Terry Wilson 2013-11-05 05:49:44 UTC
According to post: https://www.redhat.com/archives/rhos-list/2013-November/msg00007.html the provided solution may be incomplete--it may be necessary ensure also that the "network" service is enabled/running so that ifcfg-scripts actually run?

Comment 24 Martin Magr 2013-11-07 10:39:26 UTC
I've commented code which disables NetworkManager since it's bringing more issues than it solves.

Comment 25 Martin Magr 2013-12-04 10:11:06 UTC
According to Matthias not disabling NetworkManager is also wrong and killing NM and enabling network service is also wrong. According to his suggestion I'm posting needinfo for Neutron guys to tell us what should we do with it. Please check core review [1] to get a context of this.

[1] https://review.openstack.org/#/c/55528/

Comment 26 Jakub Libosvar 2013-12-04 13:08:36 UTC
I don't have a good knowledge about NetworkManager, I simply don't use it. I haven't run into any issues in my environment but the truth is I don't run any complex scenarios for Neutron.

From what I saw so far is that Neutron doesn't require NetworkManager, thus I see getting rid of it on servers is a good thing to do.

I also think it would be very helpful to have more inputs on what issues are brought by disabling NetworkManager stated in comment 24.

Also a good input could be from QE to see whether they do or don't use NetworkManager on their nodes and if not using it brings any problems. Martina or anybody who's in charge of nodes?

Comment 29 Terry Wilson 2013-12-04 16:04:02 UTC
Meh. Packstack is repeatedly described as being only for "proof of concept" installs anyway. So I don't really care one way or another. I'd just wait until someone runs into a specific problem *now* with NetworkManager. Just reporting a bug that "NetworkManager is bad, or at least has been in the past" doesn't really do it for me.

Comment 30 Perry Myers 2013-12-04 16:11:50 UTC
I think we should just document that as of right now it is recommended to provision your bare metal machines with NetworkManager disabled if running on RHEL 6.

This could be part of the kickstart that is used in a Foreman like install but does not (should not) necessarily be part of Packstack or the Puppet Modules directly.

In RHEL 7, we should try to coexist with NetworkManager if we can, but for RHEL 6 telling folks to disable as part of base provisioning may make sense just to eliminate potential issues.

Documentation that disabling NetworkManager is a best practice via a Release Note and formal docs makes sense.

@rkhan, afaik there was a specific issue where creating network interfaces (which Neutron and Nova Networking needs to do) with NM_CONTROLLED=no in them were still taken down by the NetworkManager daemon.  Afair, @tgraf knew about this issue and I think it is fixed in Fedora but don't know if that fix propagated back to RHEL 6.

@tgraf, can you remind me of the bz# for that?

Comment 32 Thomas Graf 2013-12-05 10:01:58 UTC
(In reply to Perry Myers from comment #30)
> @tgraf, can you remind me of the bz# for that?

https://bugzilla.redhat.com/show_bug.cgi?id=979181

Comment 33 Bob Kukura 2013-12-06 19:11:29 UTC
I missed most of this discussion, but, for the record, I completely agree with Perry's assessment in comment 30. 

Furthermore, I don't think OpenStack-specific configuration tools (such as puppet modules for neutron or nova) should strive to be general-purpose network configuration tools. It must remain possible to deploy OpenStack in any suitable network environment, however that is configured and managed. Instead, top-level tools (i.e. Foreman or OpenStack-m) that are responsible for the entire deployment should combine the lower-lever OpenStack component configuration tools with some suitable network configuration solution.

Comment 34 Martin Magr 2014-06-26 15:14:38 UTC
*** Bug 1113636 has been marked as a duplicate of this bug. ***