Description of problem:
Our QE Openstack environment, on OSP4, is down and we cannot start the openstack-nova-network service.
openstack-nova-network_stop_0 on openstack-network-01 'unknown error' (1): call=-1, status=Timed Out, last-rc-change='Tue Mar 15 18:52:30 2016', queued=0ms, exec=0ms
There isn't anything in /var/log/nova/network.log from any recent period. We had been seeing timeout issues (No protocol received, closing) on the controllers, but increasing the timeout stopped that. We've also seen conntrack table full errors on the network node, but that doesn't appear to be preventing us from starting the service and I've increased the max on that as well.
Version-Release number of selected component (if applicable):
openstack-nova-network-2013.2.4-4.el6ost.noarch
How reproducible:
100%
Steps to Reproduce:
1. pcs resource disable openstack-nova-network
2. pcs resource enable openstack-nova-network
3. Still hasn't started.
Actual results:
openstack-nova-network fails to start
Expected results:
service starts
Additional info:
One of the behaviors we're seeing is that we cannot create instances. They appear to go from Task State networking to Status: ERROR.
I've unmanaged nova-networking in PCS and started the service manually via standard sysv commands, but we get the same behavior. The service appears to be running here, though.