Bug 970500

Summary: quantum server log errors post installation.
Product: Red Hat OpenStack Reporter: Nir Magnezi <nmagnezi>
Component: openstack-neutronAssignee: Jakub Libosvar <jlibosva>
Status: CLOSED WORKSFORME QA Contact: Ofer Blaut <oblaut>
Severity: low Docs Contact:
Priority: low    
Version: 3.0CC: chrisw, jkt, jlibosva, lpeer
Target Milestone: ---   
Target Release: 4.0   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-11-24 13:27:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Nir Magnezi 2013-06-04 07:50:04 UTC
Description of problem:
=======================
I noticed many repeated errors ans exception in server.log post installation.
I don't know what are the functional implications of such errors.

Version-Release number of selected component (if applicable):
=============================================================
# rpm -qa | grep quantum
python-quantumclient-2.2.1-1.el6ost.noarch
openstack-quantum-2013.1.1-9.el6ost.noarch
openstack-quantum-openvswitch-2013.1.1-9.el6ost.noarch
python-quantum-2013.1.1-9.el6ost.noarch

How reproducible:
=================
1/1

Steps to Reproduce:
===================
1. Used packstack[1] to install Openstack all-in-one
2. packstack Quantum related configuration
CONFIG_QUANTUM_INSTALL=y
CONFIG_QUANTUM_SERVER_HOST=<IP_Address>
CONFIG_QUANTUM_USE_NAMESPACES=n
CONFIG_QUANTUM_KS_PW=22cfa5c4bc304a09
CONFIG_QUANTUM_DB_PW=d4dc90e3e9d84753
CONFIG_QUANTUM_L3_HOSTS=<IP_Address>
CONFIG_QUANTUM_L3_EXT_BRIDGE=br-ex
CONFIG_QUANTUM_DHCP_HOSTS=<IP_Address>
CONFIG_QUANTUM_L2_PLUGIN=openvswitch
CONFIG_QUANTUM_METADATA_HOSTS=<IP_Address>
CONFIG_QUANTUM_METADATA_PW=7f5c2eef86824c76
CONFIG_QUANTUM_LB_TENANT_NETWORK_TYPE=local
CONFIG_QUANTUM_LB_VLAN_RANGES=
CONFIG_QUANTUM_LB_INTERFACE_MAPPINGS=
CONFIG_QUANTUM_OVS_TENANT_NETWORK_TYPE=local
CONFIG_QUANTUM_OVS_VLAN_RANGES=
CONFIG_QUANTUM_OVS_BRIDGE_MAPPINGS=physnet1:1000:2000
3. Check /var/log/quantum/server.log

[1] Version: openstack-packstack-2013.1.1-0.7.dev580.el6ost.noarch

Actual results:
===============
server.log

RuntimeError: Second simultaneous write on fileno 8 detected.  Unless you really know what you're doing, make sure that only one greenthread can write any particular socket.  Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.debug.hub_multiple_reader_prevention(False)

2013-06-04 06:36:59    ERROR [quantum.agent.dhcp_agent] Failed reporting state!
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/quantum/agent/dhcp_agent.py", line 700, in _report_state
    self.agent_state)
  File "/usr/lib/python2.6/site-packages/quantum/agent/rpc.py", line 66, in report_state
    topic=self.topic)
  File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/proxy.py", line 80, in call
    return rpc.call(context, self._get_topic(topic), msg, timeout)
  File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/__init__.py", line 140, in call
    return _get_impl().call(CONF, context, topic, msg, timeout)
  File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py", line 611, in call
    rpc_amqp.get_connection_pool(conf, Connection))
  File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 611, in call
    rv = multicall(conf, context, topic, msg, timeout, connection_pool)
  File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 594, in multicall
    conn = ConnectionContext(conf, connection_pool)
  File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 122, in __init__
    self.connection = connection_pool.get()
  File "/usr/lib/python2.6/site-packages/eventlet/pools.py", line 119, in get
    created = self.create()
  File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/amqp.py", line 76, in create
    return self.connection_cls(self.conf)
  File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py", line 308, in __init__
    self.reconnect()
  File "/usr/lib/python2.6/site-packages/quantum/openstack/common/rpc/impl_qpid.py", line 349, in reconnect
    self.connection.open()
  File "<string>", line 6, in open
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 256, in open
    self.attach()
  File "<string>", line 6, in attach
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 274, in attach
    self._ewait(lambda: self._transport_connected and not self._unlinked())
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 209, in _ewait
    self.check_error()
  File "/usr/lib/python2.6/site-packages/qpid/messaging/endpoints.py", line 202, in check_error
    raise self.error
InternalError: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 509, in dispatch
    self.connect()
  File "/usr/lib/python2.6/site-packages/qpid/messaging/driver.py", line 533, in connect
    self._transport = trans(self.connection, host, port)
  File "/usr/lib/python2.6/site-packages/qpid/messaging/transports.py", line 28, in __init__
    self.socket = connect(host, port)
  File "/usr/lib/python2.6/site-packages/qpid/util.py", line 49, in connect
    sock.connect(sa)
  File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 168, in connect
    trampoline(fd, write=True)
  File "/usr/lib/python2.6/site-packages/eventlet/hubs/__init__.py", line 119, in trampoline
    listener = hub.add(hub.WRITE, fileno, current.switch)
  File "/usr/lib/python2.6/site-packages/eventlet/hubs/epolls.py", line 48, in add
    listener = BaseHub.add(self, evtype, fileno, cb)
  File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 126, in add
    evtype, fileno, evtype))
RuntimeError: Second simultaneous write on fileno 8 detected.  Unless you really know what you're doing, make sure that only one greenthread can write any particular socket.  Consider using a pools.Pool. If you do know what you're doing and want to disable this error, call eventlet.debug.hub_multiple_reader_prevention(False)

2013-06-04 06:39:35  WARNING [quantum.openstack.common.loopingcall] task run outlasted interval by 0.866072 sec
2013-06-04 08:53:55  WARNING [quantum.openstack.common.loopingcall] task run outlasted interval by 0.320455 sec
2013-06-04 10:27:06  WARNING [quantum.openstack.common.loopingcall] task run outlasted interval by 4.169436 sec


Expected results:
=================
No Errors should be expected

Comment 4 lpeer 2013-08-13 07:18:42 UTC
After further discussion with Ofer it seems like this has no functionality impact.
The exceptions are there because the different processes are not in sync, after the processes go up the exceptions stop.

Leaving this as a low priority bug as the logs are not clean and for someone who is not familiar with the product it can appear like there is an error.

Comment 5 Jakub Libosvar 2013-11-19 16:47:44 UTC
Does it happen always after installation? I cannot reproduce that.

How important is mentioned packstack answer file configuration?

Comment 6 Nir Magnezi 2013-11-24 13:27:08 UTC
The bug no longer reproduce.

# rpm -qa| grep quantum
python-quantumclient-2.2.1-2.el6ost.noarch
python-quantum-2013.1.4-3.el6ost.noarch
openstack-quantum-openvswitch-2013.1.4-3.el6ost.noarch
openstack-quantum-2013.1.4-3.el6ost.noarch


I installed Openstack via packstack [1], an all-in-one installation.
I compared and found some differences in quantum configuration files (compared the ones I attached to the bug with the newly created config files)

Difference #1:
quantum.conf: only the old file contains the line: root_helper=sudo quantum-rootwrap /etc/quantum/rootwrap.conf
quantum.conf: only the new file contains the line: qpid_reconnect=True

Does not seem like nothing major.

Difference #2:
When I filed this BZ, namespaces where disabled by default. now the default value is to enable namespaces.
This change is expressed in quantum configuration files, but since we no longer work without namespaces, There is no point in trying that configuration.

Closing the bug.

[1] openstack-packstack-2013.1.1-0.35.dev696.el6ost.noarch