Created attachment 806528 [details] log files Description of problem: Nova services fail to start: [root@rhel6 ~(keystone_demo)]# openstack-status == Nova services == openstack-nova-api: dead openstack-nova-cert: dead openstack-nova-compute: dead openstack-nova-network: dead (disabled on boot) openstack-nova-scheduler: dead openstack-nova-volume: dead (disabled on boot) openstack-nova-conductor: dead == Glance services == openstack-glance-api: active openstack-glance-registry: active == Keystone service == openstack-keystone: active == Horizon service == openstack-dashboard: 500 == neutron services == neutron-server: active neutron-dhcp-agent: active neutron-l3-agent: dead neutron-metadata-agent: active neutron-openvswitch-agent: active == Cinder services == openstack-cinder-api: active openstack-cinder-scheduler: active openstack-cinder-volume: active == Ceilometer services == openstack-ceilometer-api: active openstack-ceilometer-central: active openstack-ceilometer-compute: active openstack-ceilometer-collector:active == Support services == mysqld: active libvirtd: active openvswitch: active messagebus: active tgtd: active qpidd: active memcached: active == Keystone users == See attached logs for the import errors.. Under /var/log/nova
So the issue here was the newly added dependency on the parallel installable python-jinja2-26. The simplest short term fix would be just to revert that change back to python-jinja (2.2), but since we're testing this fix we might as well align more with modern systems (and upstream test) and use the available python-jinja2-26 correctly. RDO was updated accordingly also.
It seems like I encountered the same issue: Version ======= rhos 4.0 on rhel 6.5, 2013-11-08.1 puddle, openstack-neutron-2013.2-5.el6ost openstack-neutron-vpn-agent-2013.2-5.el6ost openstack-nova-api-2013.2-4.el6ost.noarch python-novaclient-2.15.0-1.el6ost.noarch openstack-nova-common-2013.2-4.el6ost.noarch openstack-nova-conductor-2013.2-4.el6ost.noarch openstack-nova-novncproxy-2013.2-4.el6ost.noarch openstack-nova-cert-2013.2-4.el6ost.noarch python-nova-2013.2-4.el6ost.noarch openstack-nova-console-2013.2-4.el6ost.noarch openstack-nova-scheduler-2013.2-4.el6ost.noarch The default vpn_device_driver is an openswag driver (vpn_device_driver=neutron.services.vpn.device_drivers.ipsec.OpenSwanDriver), however, it's neutron vpn agent fails to load/import it because this driver uses the puthon jinja2 driver which fails to be imported. It looks like the following jinja2 rpm (python-jinja2-26-2.6-2.el6ost.noarch) is installed in my env, but it's installed into an egg instead of regualarly on the site-packages directory: /usr/lib/python2.6/site-packages/Jinja2-2.6-py2.6.egg, hence any import of jinja2 fails - even manually when the site-packages is in my PYTHONPATH. The class load exception from /var/log/neutron/vpn-agent.log: 2013-11-13 11:02:32.472 26715 CRITICAL neutron [-] Can not load driver :neutron.services.vpn.device_drivers.ipsec.OpenSwanDriver 2013-11-13 11:02:32.472 26715 TRACE neutron Traceback (most recent call last): 2013-11-13 11:02:32.472 26715 TRACE neutron File "/usr/bin/neutron-vpn-agent", line 10, in <module> 2013-11-13 11:02:32.472 26715 TRACE neutron sys.exit(main()) 2013-11-13 11:02:32.472 26715 TRACE neutron File "/usr/lib/python2.6/site-packages/neutron/services/vpn/agent.py", line 148, in main 2013-11-13 11:02:32.472 26715 TRACE neutron manager='neutron.services.vpn.agent.VPNAgent') 2013-11-13 11:02:32.472 26715 TRACE neutron File "/usr/lib/python2.6/site-packages/neutron/agent/l3_agent.py", line 876, in main 2013-11-13 11:02:32.472 26715 TRACE neutron manager=manager) 2013-11-13 11:02:32.472 26715 TRACE neutron File "/usr/lib/python2.6/site-packages/neutron/service.py", line 204, in create 2013-11-13 11:02:32.472 26715 TRACE neutron periodic_fuzzy_delay=periodic_fuzzy_delay) 2013-11-13 11:02:32.472 26715 TRACE neutron File "/usr/lib/python2.6/site-packages/neutron/service.py", line 137, in __init__ 2013-11-13 11:02:32.472 26715 TRACE neutron self.manager = manager_class(host=host, *args, **kwargs) 2013-11-13 11:02:32.472 26715 TRACE neutron File "/usr/lib/python2.6/site-packages/neutron/services/vpn/agent.py", line 37, in __init__ 2013-11-13 11:02:32.472 26715 TRACE neutron self.setup_device_drivers(host) 2013-11-13 11:02:32.472 26715 TRACE neutron File "/usr/lib/python2.6/site-packages/neutron/services/vpn/agent.py", line 57, in setup_device_drivers 2013-11-13 11:02:32.472 26715 TRACE neutron device_driver=device_driver) 2013-11-13 11:02:32.472 26715 TRACE neutron DeviceDriverImportError: Can not load driver :neutron.services.vpn.device_drivers.ipsec.OpenSwanDriver
https://github.com/redhat-openstack/quantum/commit/5c5194e0b5257939f55ff04f6d1d6011d376f1f8 needs to be adjusted: --- a/neutron/__init__.py +++ b/neutron/__init__.py @@ -37,6 +37,7 @@ def replace_dist(requirement): replace_dist("WebOb >= 1.0") replace_dist("SQLAlchemy >= 0.6.3") replace_dist("Routes >= 1.12.3") +replace_dist("Jinja2 >= 2.6") replace_dist("PasteDeploy >= 1.5.0") # This hack is needed because replace_dist() results in
(In reply to Alan Pevec from comment #5) > https://github.com/redhat-openstack/quantum/commit/ > 5c5194e0b5257939f55ff04f6d1d6011d376f1f8 needs to be adjusted: > > --- a/neutron/__init__.py > +++ b/neutron/__init__.py > @@ -37,6 +37,7 @@ def replace_dist(requirement): > replace_dist("WebOb >= 1.0") > replace_dist("SQLAlchemy >= 0.6.3") > replace_dist("Routes >= 1.12.3") > +replace_dist("Jinja2 >= 2.6") > > replace_dist("PasteDeploy >= 1.5.0") > # This hack is needed because replace_dist() results in I've tried to add this line manually, now neutron vpn agent loads the openswag driver well.
Thanks Rami. Padraig, I've noticed el6-havana-patches wasn't rebased to 2013.2 so I did that and amended "use parallel installed versions in RHEL6" patch.
Looks correct Alan thanks. jinja2 is in upstream neutron requirements.txt, and the parallel package is already depended on by the neutron package. So we should just need to respin RDO and then RHOS packages with this. Will do now
Adding to beta milestone for MODIFIED/ON-QA bugs
Rami, plz try to verify tomorrow
I verified that this does not reproduce on rhos 4.0 on rhel 6.5 with puddle 2013-11-18.8. Pbrady, could you please change this bug to ON_QA for me to be able to change it to Verified?
All clear now, there was related followup fix in openstack-neutron rhbz 1031370
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. http://rhn.redhat.com/errata/RHEA-2013-1859.html