Description of problem: L3 agent is in failed status after installation RDO kilo via packstack. FWAAS was enabled in answer file. Version-Release number of selected component (if applicable): OS= RHEL7.1 repo : http://rdoproject.org/repos/openstack-kilo/rdo-testing-kilo.rpm # rpm -qa |grep neutron openstack-neutron-common-2015.1.0-1.el7.noarch python-neutronclient-2.3.11-1.el7.noarch python-neutron-lbaas-2015.1.0-1.el7.noarch openstack-neutron-lbaas-2015.1.0-1.el7.noarch python-neutron-fwaas-2015.1.0-1.el7.noarch python-neutron-2015.1.0-1.el7.noarch openstack-neutron-fwaas-2015.1.0-1.el7.noarch openstack-neutron-openvswitch-2015.1.0-1.el7.noarch openstack-neutron-2015.1.0-1.el7.noarch openstack-neutron-ml2-2015.1.0-1.el7.noarch How reproducible: Steps to Reproduce: 1.Install RHEL 7.1 {Bare metal} 2.install the repo: yum install -y http://rdoproject.org/repos/openstack-kilo/rdo-testing-kilo.rpm 3.yum update -y 4.install packstack 5.Generate answer file 6.enable in answer file FWAAS= y 7.run packstack with the answer file: packstack --answer-file=answer 8.When installation complete check openstack-status you can see that l3-agent is failed Actual results: l3-agent is failed in log file I got this error : 2015-05-05 10:42:27.719 17694 ERROR neutron_fwaas.services.firewall.agents.l3reference.firewall_l3_agent [req-642f4abc-2705-42a9-b25c-d8a4621e64e2 ] FWaaS plugin is configured in the server side, but FWaaS is disabled in L3-agent. ~ ~ [root@puma16 ~(keystone_admin)]# systemctl status neutron-l3-agent neutron-l3-agent.service - OpenStack Neutron Layer 3 Agent Loaded: loaded (/usr/lib/systemd/system/neutron-l3-agent.service; enabled) Active: failed (Result: exit-code) since Tue 2015-05-05 10:42:27 IDT; 45min ago Main PID: 17694 (code=exited, status=1/FAILURE) CGroup: /system.slice/neutron-l3-agent.service May 05 10:42:26 puma16.scl.lab.tlv.redhat.com systemd[1]: Started OpenStack Neutron Layer 3 Agent. May 05 10:42:27 puma16.scl.lab.tlv.redhat.com systemd[1]: neutron-l3-agent.service: main process exited, code=exited, status=1/FAILURE May 05 10:42:27 puma16.scl.lab.tlv.redhat.com systemd[1]: Unit neutron-l3-agent.service entered failed state. Expected results: no error all services in active status Additional info:
Workaround : when adding the config file /etc/neutron/fwaas_driver.ini to /usr/lib/systemd/system/neutron-l3-agent.service and restarted the l3 agent the l3 agent is started without error # vim /usr/lib/systemd/system/neutron-l3-agent.service [Unit] Description=OpenStack Neutron Layer 3 Agent After=syslog.target network.target [Service] Type=simple User=neutron ExecStart=/usr/bin/neutron-l3-agent --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/l3_agent --config-file /etc/neutron/neutron.conf --config-dir /etc/neutron/conf.d/neutron-l3-agent --log-file /var/log/neutron/l3-agent.log --config-file /etc/neutron/fwaas_driver.ini PrivateTmp=false KillMode=process [Install] WantedBy=multi-user.target # systemctl daemon-reload # systemctl restart neutron-l3-agent
Eran, could you please: 0) Upload the current /etc/neutron as a tar.gz when FWaaS is enabled 1) check if server side disabled, and l3-agent side enabled works, 2) and upload a log for the l3-agent in such case?. I'm trying to decide what's the best solution here or if packstack is configuring outdated settings anywhere.
Ajo 0. I upload tar file as you ask + answer file 1. I disabled FW in neutron.conf and after neutron restart all neutron service are in active status 2.upload the log file . as you can see Ihave some new error messages : 2015-05-05 14:30:51.070 5514 ERROR neutron.callbacks.manager [-] Error during notification for neutron.agent.metadata.driver.after_router_added router, after_create 2015-05-05 14:30:51.070 5514 TRACE neutron.callbacks.manager IOError: [Errno 13] Permission denied: u'/var/lib/neutron/lock/neutron-iptables-qrouter-7c738e45-cdc0-4f26-9e97-5993946b1031'
Created attachment 1022159 [details] tar etc/neutron
Created attachment 1022160 [details] log l3-agent
Created attachment 1022161 [details] answer file
(In reply to Eran Kuris from comment #3) > Ajo > 0. I upload tar file as you ask + answer file > 1. I disabled FW in neutron.conf and after neutron restart all neutron > service are in active status > 2.upload the log file . as you can see Ihave some new error messages : > > 2015-05-05 14:30:51.070 5514 ERROR neutron.callbacks.manager [-] Error > during notification for neutron.agent.metadata.driver.after_router_added > router, after_create > 2015-05-05 14:30:51.070 5514 TRACE neutron.callbacks.manager IOError: [Errno > 13] Permission denied: > u'/var/lib/neutron/lock/neutron-iptables-qrouter-7c738e45-cdc0-4f26-9e97- > 5993946b1031' After examining the network node, we found out l3-agent was run under root that created lock for iptables with root owner. After changing lock ownership to neutron, l3 agent was able to use iptables. With regarding to original issue, if FWaaS service plugin is used in neutron-server, it must be also enabled in l3 agent. That's part of deployment tool. As written in description packstack was used, I'm moving this bug to packstack.
After checking this and discussing it with Miguel Angel on IRC, it looks like the issue is on the openstack-neutron-fwaas package. An older release (openstack-neutron-fwaas-2015.1-0rc2, built on Delorean) included /usr/share/neutron/l3_agent/fwaas_driver.conf as a symlink to /etc/neutron/fwaas_driver.ini . This symlink is required to start the L3 agent when FWaaS is configured. However, openstack-neutron-fwaas-2015.1.0-1 (built on CBS) does not include it, so we suspect the spec file has some differences. So, a quick workaround would be: ln -s /etc/neutron/fwaas_driver.ini /usr/share/neutron/l3_agent/fwaas_driver.conf Alan, can you comment on the packaging differences? Reassigning the bug to openstack-neutron.
https://github.com/openstack-packages/neutron-fwaas/compare/rpm-kilo...f20-master We need to cherry pick the missing commits into rpm-kilo, it seems we branched it out from a wrong commit id.
My bad, I had out of date git checkout when branching rpm-kilo, fixing it now.
in openstack-kilo/testing repodata 06-May-2015 15:52 UTC