Attachment 903232 Details for Bug 1105850 – metadata-agent.log

metadata-agent.log

file_1105850.txt (text/plain), 12.45 KB, created by Ofer Blaut on 2014-06-08 08:00:22 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Ofer Blaut

Created: 2014-06-08 08:00:22 UTC

Size: 12.45 KB

patch

obsolete

>[root@puma04 ~(keystone_admin_tenant1)]$rpm -qa | grep neutron
>python-neutronclient-2.3.4-2.el7ost.noarch
>openstack-neutron-2014.1-26.el7ost.noarch
>python-neutron-2014.1-26.el7ost.noarch
>openstack-neutron-openvswitch-2014.1-26.el7ost.noarch
>openstack-neutron-linuxbridge-2014.1-26.el7ost.noarch
>openstack-neutron-ml2-2014.1-26.el7ost.noarch
>
>
>== neutron services ==
>neutron-server:                         active
>neutron-dhcp-agent:                     active
>neutron-l3-agent:                       active
>neutron-metadata-agent:                 active
>neutron-lbaas-agent:                    inactive  (disabled on boot)
>neutron-openvswitch-agent:              active
>neutron-linuxbridge-agent:              inactive  (disabled on boot)
>neutron-ryu-agent:                      inactive  (disabled on boot)
>neutron-nec-agent:                      inactive  (disabled on boot)
>neutron-mlnx-agent:                     inactive  (disabled on boot)
>
>
>/var/log/neutron/neutron-ns-metadata-proxy-7d311111-f227-4acc-ba6c-8e969a2f5094.log 
>
>2014-06-08 08:43:08.560 9498 INFO neutron.wsgi [-] (9498) accepted ('21.0.0.11', 57223)
>
>2014-06-08 08:43:08.562 9498 ERROR neutron.agent.metadata.namespace_proxy [-] Unexpected error.
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy Traceback (most recent call last):
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/namespace_proxy.py", line 74, in __call__
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy     req.body)
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/namespace_proxy.py", line 105, in _proxy_request
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy     connection_type=UnixDomainHTTPConnection)
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1605, in request
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy     (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1353, in _request
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy     (response, content) = self._conn_request(conn, request_uri, method, body, headers)
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1327, in _conn_request
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy     conn.connect()
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/namespace_proxy.py", line 48, in connect
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy     self.sock.connect(cfg.CONF.metadata_proxy_socket)
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 190, in connect
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy     while not socket_connect(fd, address):
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 39, in socket_connect
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy     raise socket.error(err, errno.errorcode[err])
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy error: [Errno 13] EACCES
>2014-06-08 08:43:08.562 9498 TRACE neutron.agent.metadata.namespace_proxy 
>2014-06-08 08:43:08.563 9498 INFO neutron.wsgi [-] 21.0.0.11 - - [08/Jun/2014 08:43:08] "GET /2009-04-04/meta-data/instance-id HTTP/1.1" 500 343 0.002016
>
>2014-06-08 08:43:09.672 9498 INFO neutron.wsgi [-] (9498) accepted ('21.0.0.12', 35267)
>
>2014-06-08 08:43:09.673 9498 ERROR neutron.agent.metadata.namespace_proxy [-] Unexpected error.
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy Traceback (most recent call last):
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/namespace_proxy.py", line 74, in __call__
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy     req.body)
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/namespace_proxy.py", line 105, in _proxy_request
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy     connection_type=UnixDomainHTTPConnection)
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1605, in request
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy     (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1353, in _request
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy     (response, content) = self._conn_request(conn, request_uri, method, body, headers)
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1327, in _conn_request
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy     conn.connect()
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/namespace_proxy.py", line 48, in connect
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy     self.sock.connect(cfg.CONF.metadata_proxy_socket)
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 190, in connect
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy     while not socket_connect(fd, address):
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/site-packages/eventlet/greenio.py", line 39, in socket_connect
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy     raise socket.error(err, errno.errorcode[err])
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy error: [Errno 13] EACCES
>2014-06-08 08:43:09.673 9498 TRACE neutron.agent.metadata.namespace_proxy 
>2014-06-08 08:43:09.688 9498 INFO neutron.wsgi [-] 21.0.0.12 - - [08/Jun/2014 08:43:09] "GET /2009-04-04/meta-data/instance-id HTTP/1.1" 500 343 0.015121
>
>
>
>
>[root@puma04 ~(keystone_admin)]# cat /var/log/neutron/metadata-agent.log 
>2014-06-08 08:23:56.858 24941 INFO neutron.common.config [-] Logging enabled!
>2014-06-08 08:23:56.963 24941 INFO eventlet.wsgi.server [req-4249adf0-3f12-42f3-87bb-caa0465573b8 None] (24941) wsgi starting up on http:///:v/
>
>2014-06-08 08:23:56.971 24941 INFO neutron.openstack.common.rpc.common [req-4249adf0-3f12-42f3-87bb-caa0465573b8 None] Connected to AMQP server on 10.35.160.17:5672
>2014-06-08 08:23:56.983 24941 INFO neutron.openstack.common.rpc.common [req-4249adf0-3f12-42f3-87bb-caa0465573b8 None] Connected to AMQP server on 10.35.160.17:5672
>2014-06-08 08:24:56.988 24941 ERROR neutron.agent.metadata.agent [req-4249adf0-3f12-42f3-87bb-caa0465573b8 None] Failed reporting state!
>2014-06-08 08:24:56.988 24941 TRACE neutron.agent.metadata.agent Traceback (most recent call last):
>2014-06-08 08:24:56.988 24941 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 359, in _report_state
>2014-06-08 08:24:56.988 24941 TRACE neutron.agent.metadata.agent     use_call=self.agent_state.get('start_flag'))
>2014-06-08 08:24:56.988 24941 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 72, in report_state
>2014-06-08 08:24:56.988 24941 TRACE neutron.agent.metadata.agent     return self.call(context, msg, topic=self.topic)
>2014-06-08 08:24:56.988 24941 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/proxy.py", line 129, in call
>2014-06-08 08:24:56.988 24941 TRACE neutron.agent.metadata.agent     exc.info, real_topic, msg.get('method'))
>2014-06-08 08:24:56.988 24941 TRACE neutron.agent.metadata.agent Timeout: Timeout while waiting on RPC response - topic: "q-plugin", RPC method: "report_state" info: "<unknown>"
>2014-06-08 08:24:56.988 24941 TRACE neutron.agent.metadata.agent 
>2014-06-08 08:24:56.989 24941 WARNING neutron.openstack.common.loopingcall [req-4249adf0-3f12-42f3-87bb-caa0465573b8 None] task run outlasted interval by 30.09215 sec
>2014-06-08 08:25:57.051 24941 ERROR neutron.agent.metadata.agent [req-4249adf0-3f12-42f3-87bb-caa0465573b8 None] Failed reporting state!
>2014-06-08 08:25:57.051 24941 TRACE neutron.agent.metadata.agent Traceback (most recent call last):
>2014-06-08 08:25:57.051 24941 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 359, in _report_state
>2014-06-08 08:25:57.051 24941 TRACE neutron.agent.metadata.agent     use_call=self.agent_state.get('start_flag'))
>2014-06-08 08:25:57.051 24941 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/rpc.py", line 72, in report_state
>2014-06-08 08:25:57.051 24941 TRACE neutron.agent.metadata.agent     return self.call(context, msg, topic=self.topic)
>2014-06-08 08:25:57.051 24941 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/site-packages/neutron/openstack/common/rpc/proxy.py", line 129, in call
>2014-06-08 08:25:57.051 24941 TRACE neutron.agent.metadata.agent     exc.info, real_topic, msg.get('method'))
>2014-06-08 08:25:57.051 24941 TRACE neutron.agent.metadata.agent Timeout: Timeout while waiting on RPC response - topic: "q-plugin", RPC method: "report_state" info: "<unknown>"
>2014-06-08 08:25:57.051 24941 TRACE neutron.agent.metadata.agent 
>2014-06-08 08:25:57.053 24941 WARNING neutron.openstack.common.loopingcall [req-4249adf0-3f12-42f3-87bb-caa0465573b8 None] task run outlasted interval by 30.06305 sec
>[root@puma04 ~(keystone_admin)]# 
>
>
>[root@puma04 ~(keystone_admin_tenant1)]$ip netns exec qrouter-7d311111-f227-4acc-ba6c-8e969a2f5094 iptables -nL -t nat
>Chain PREROUTING (policy ACCEPT)
>target     prot opt source               destination         
>neutron-l3-agent-PREROUTING  all  --  0.0.0.0/0            0.0.0.0/0           
>
>Chain INPUT (policy ACCEPT)
>target     prot opt source               destination         
>
>Chain OUTPUT (policy ACCEPT)
>target     prot opt source               destination         
>neutron-l3-agent-OUTPUT  all  --  0.0.0.0/0            0.0.0.0/0           
>
>Chain POSTROUTING (policy ACCEPT)
>target     prot opt source               destination         
>neutron-l3-agent-POSTROUTING  all  --  0.0.0.0/0            0.0.0.0/0           
>neutron-postrouting-bottom  all  --  0.0.0.0/0            0.0.0.0/0           
>
>Chain neutron-l3-agent-OUTPUT (1 references)
>target     prot opt source               destination         
>
>Chain neutron-l3-agent-POSTROUTING (1 references)
>target     prot opt source               destination         
>ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ! ctstate DNAT
>
>Chain neutron-l3-agent-PREROUTING (1 references)
>target     prot opt source               destination         
>REDIRECT   tcp  --  0.0.0.0/0            169.254.169.254      tcp dpt:80 redir ports 9697
>
>Chain neutron-l3-agent-float-snat (1 references)
>target     prot opt source               destination         
>
>Chain neutron-l3-agent-snat (1 references)
>target     prot opt source               destination         
>neutron-l3-agent-float-snat  all  --  0.0.0.0/0            0.0.0.0/0           
>SNAT       all  --  21.0.0.0/24          0.0.0.0/0            to:10.35.180.20
>SNAT       all  --  22.0.0.0/24          0.0.0.0/0            to:10.35.180.20
>
>Chain neutron-postrouting-bottom (1 references)
>target     prot opt source               destination         
>neutron-l3-agent-snat  all  --  0.0.0.0/0            0.0.0.0/0    
>

Actions: View

Attachments on bug 1105850: 903232 | 907416