Bug 1080438

Summary: f20: packstack allinone instalation failed: neutron-ovs-cleanup start' returned 1
Product: [Community] RDO Reporter: Ido Ovadia <iovadia>
Component: openstack-packstackAssignee: Martin Magr <mmagr>
Status: CLOSED CURRENTRELEASE QA Contact: Ido Ovadia <iovadia>
Severity: urgent Docs Contact:
Priority: high    
Version: unspecifiedCC: aortega, chrisw, derekh, jlibosva, jslagle, mmagr, pbrady, whayutin, yeylon
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-04-29 14:33:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
packstack install logs for recreate none

Description Ido Ovadia 2014-03-25 12:47:37 UTC 
Description of problem:
-----------------------
Installation of packstack --allinone is failing on Fedora20


ERROR : Error appeared during Puppet run: 10.35.64.158_neutron.pp
Error: Could not start Service[ovs-cleanup-service]: Execution of '/sbin/service neutron-ovs-cleanup start' returned 1: 
You will find full trace in log /var/tmp/packstack/20140325-144031-XD3KyR/manifests/10.35.64.158_neutron.pp.log
Please check log file /var/tmp/packstack/20140325-144031-XD3KyR/openstack-setup.log for more information



Version-Release number of selected component (if applicable):

openstack-packstack-2014.1.1-0.4.dev1018.fc21.noarch

How reproducible:


Steps to Reproduce:
-------------------
1. sudo yum install -y openstack-packstack
2. packstack --allinone



Actual results:
---------------
Full output of packstack --allinone is mentioned below:

======================================================================
packstack --allinone
Welcome to Installer setup utility
Packstack changed given value  to required value /root/.ssh/id_rsa.pub

Installing:
Clean Up                                             [ DONE ]
root.64.158's password: 
Setting up ssh keys                                  [ DONE ]
Discovering hosts' details                           [ DONE ]
Adding pre install manifest entries                  [ DONE ]
Adding MySQL manifest entries                        [ DONE ]
Adding AMQP manifest entries                         [ DONE ]
Adding Keystone manifest entries                     [ DONE ]
Adding Glance Keystone manifest entries              [ DONE ]
Adding Glance manifest entries                       [ DONE ]
Installing dependencies for Cinder                   [ DONE ]
Adding Cinder Keystone manifest entries              [ DONE ]
Adding Cinder manifest entries                       [ DONE ]
Checking if the Cinder server has a cinder-volumes vg[ DONE ]
Adding Nova API manifest entries                     [ DONE ]
Adding Nova Keystone manifest entries                [ DONE ]
Adding Nova Cert manifest entries                    [ DONE ]
Adding Nova Conductor manifest entries               [ DONE ]
Adding Nova Compute manifest entries                 [ DONE ]
Adding Nova Scheduler manifest entries               [ DONE ]
Adding Nova VNC Proxy manifest entries               [ DONE ]
Adding Nova Common manifest entries                  [ DONE ]
Adding Openstack Network-related Nova manifest entries[ DONE ]
Adding Neutron API manifest entries                  [ DONE ]
Adding Neutron Keystone manifest entries             [ DONE ]
Adding Neutron L3 manifest entries                   [ DONE ]
Adding Neutron L2 Agent manifest entries             [ DONE ]
Adding Neutron DHCP Agent manifest entries           [ DONE ]
Adding Neutron LBaaS Agent manifest entries          [ DONE ]
Adding Neutron Metadata Agent manifest entries       [ DONE ]
Adding OpenStack Client manifest entries             [ DONE ]
Adding Horizon manifest entries                      [ DONE ]
Adding Provisioning manifest entries                 [ DONE ]
Adding Ceilometer manifest entries                   [ DONE ]
Adding Ceilometer Keystone manifest entries          [ DONE ]
Adding Nagios server manifest entries                [ DONE ]
Adding Nagios host manifest entries                  [ DONE ]
Adding post install manifest entries                 [ DONE ]
Preparing servers                                    [ DONE ]
Installing Dependencies                              [ DONE ]
Copying Puppet modules and manifests                 [ DONE ]
Applying 10.35.64.158_prescript.pp
10.35.64.158_prescript.pp:                           [ DONE ]        
Applying 10.35.64.158_mysql.pp
Applying 10.35.64.158_amqp.pp
10.35.64.158_mysql.pp:                               [ DONE ]    
10.35.64.158_amqp.pp:                                [ DONE ]    
Applying 10.35.64.158_keystone.pp
Applying 10.35.64.158_glance.pp
Applying 10.35.64.158_cinder.pp
10.35.64.158_keystone.pp:                            [ DONE ]       
10.35.64.158_glance.pp:                              [ DONE ]       
10.35.64.158_cinder.pp:                              [ DONE ]       
Applying 10.35.64.158_api_nova.pp
10.35.64.158_api_nova.pp:                            [ DONE ]       
Applying 10.35.64.158_nova.pp
10.35.64.158_nova.pp:                                [ DONE ]   
Applying 10.35.64.158_neutron.pp
10.35.64.158_neutron.pp:                          [ ERROR ]        
Applying Puppet manifests                         [ ERROR ]

ERROR : Error appeared during Puppet run: 10.35.64.158_neutron.pp
Error: Could not start Service[ovs-cleanup-service]: Execution of '/sbin/service neutron-ovs-cleanup start' returned 1: 
You will find full trace in log /var/tmp/packstack/20140325-144031-XD3KyR/manifests/10.35.64.158_neutron.pp.log
Please check log file /var/tmp/packstack/20140325-144031-XD3KyR/openstack-setup.log for more information

======================================================================


Expected results:
-----------------
Packstack should install and bringup openstack successfully.


Additional info:
---------------

The following is the content of:
/var/tmp/packstack/20140325-145608-rshEpx/manifests/10.35.64.158_neutron.pp.log
======================================================================
connect: No such file or directory
Please make sure that the zfs-fuse daemon is running.
internal error: failed to initialize ZFS library
connect: No such file or directory
Please make sure that the zfs-fuse daemon is running.
internal error: failed to initialize ZFS library
^[[1;31mWarning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults^[[0m
^[[1;31mWarning: Scope(Class[Neutron::Server]): sql_connection deprecated for connection^[[0m
^[[mNotice: Compiled catalog for aqua-vds9.qa.lab.tlv.redhat.com in environment production in 0.98 seconds^[[0m
connect: No such file or directory
Please make sure that the zfs-fuse daemon is running.
internal error: failed to initialize ZFS library
connect: No such file or directory
Please make sure that the zfs-fuse daemon is running.
internal error: failed to initialize ZFS library
^[[1;31mError: Could not start Service[ovs-cleanup-service]: Execution of '/sbin/service neutron-ovs-cleanup start' returned 1: ^[[0m
^[[1;31mError: /Stage[main]/Neutron::Agents::Ovs/Service[ovs-cleanup-service]/ensure: change from stopped to running failed: Could not start Service[ovs-cleanup-service]: Execution of '/sbin/service neutron-ovs-cleanup start' returned 1: ^[[0m
^[[mNotice: /Stage[main]/Main/Exec[neutron-db-manage upgrade]/returns: Traceback (most recent call last):^[[0m
^[[mNotice: /Stage[main]/Main/Exec[neutron-db-manage upgrade]/returns:   File "/usr/bin/neutron-db-manage", line 10, in <module>^[[0m
^[[mNotice: /Stage[main]/Main/Exec[neutron-db-manage upgrade]/returns:     sys.exit(main())^[[0m
^[[mNotice: /Stage[main]/Main/Exec[neutron-db-manage upgrade]/returns:   File "/usr/lib/python2.7/site-packages/neutron/db/migration/cli.py", line 138, in main^[[0m
^[[mNotice: /Stage[main]/Main/Exec[neutron-db-manage upgrade]/returns:     CONF()^[[0m
^[[mNotice: /Stage[main]/Main/Exec[neutron-db-manage upgrade]/returns:   File "/usr/lib/python2.7/site-packages/oslo/config/cfg.py", line 1599, in __call__^[[0m
^[[mNotice: /Stage[main]/Main/Exec[neutron-db-manage upgrade]/returns:     raise ConfigFilesNotFoundError(self._namespace.files_not_found)^[[0m
^[[mNotice: /Stage[main]/Main/Exec[neutron-db-manage upgrade]/returns: oslo.config.cfg.ConfigFilesNotFoundError: Failed to read some config files: /etc/neutron/plugin.ini^[[0m
^[[1;31mError: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head returned 1 instead of one of [0]^[[0m
^[[1;31mError: /Stage[main]/Main/Exec[neutron-db-manage upgrade]/returns: change from notrun to 0 failed: neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini upgrade head returned 1 instead of one of [0]^[[0m
^[[mNotice: /Stage[main]/Neutron::Agents::Ovs/Service[neutron-plugin-ovs-service]/ensure: ensure changed 'stopped' to 'running'^[[0m
^[[mNotice: /Stage[main]/Neutron::Agents::Dhcp/Service[neutron-dhcp-service]/ensure: ensure changed 'stopped' to 'running'^[[0m
^[[mNotice: /Stage[main]/Neutron::Server/Service[neutron-server]: Dependency Exec[neutron-db-manage upgrade] has failures: true^[[0m
^[[1;31mWarning: /Stage[main]/Neutron::Server/Service[neutron-server]: Skipping because of failed dependencies^[[0m
^[[mNotice: Finished catalog run in 1.29 seconds^[[0m
======================================================================

OS: Fedora20
SELINUX=enforcing
Comment 1 wes hayutin 2014-03-25 12:53:58 UTC 
Created attachment 878454 [details]
packstack install logs for recreate

recreated the issue.. see log file
Comment 2 Pádraig Brady 2014-03-25 13:14:27 UTC 
From the log in description:

oslo.config.cfg.ConfigFilesNotFoundError: Failed to read some config files: /etc/neutron/plugin.ini

That's was also seen by eglynn and was due to the permissions causing /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini being unreadable by the neutron user.

My guess is that openstack-puppet-modules is replacing that file (with wrong owner)
Comment 3 Jakub Libosvar 2014-03-25 14:30:49 UTC 
I confirm that this was issue on Martin Magr's machine while hitting this issue. Changing ownership of /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini to neutron:neutron fixed the issue.
Comment 4 Pádraig Brady 2014-03-25 17:01:12 UTC 
The package sets that file to root:neutron
Something is intermittently setting it to root:root
That might happen if (puppet?) logic runs against that file before the package is installed
Comment 5 James Slagle 2014-03-25 17:02:32 UTC 
*** Bug 1080560 has been marked as a duplicate of this bug. ***
Comment 6 James Slagle 2014-03-25 17:05:44 UTC 
I don't think this is puppet specific. See the steps i've used to reproduce in the bug i marked as a duplicate of this one above. 

Has something to do with the neutron group not existing on the very first install of the package.
Comment 7 Martin Magr 2014-03-25 17:55:10 UTC 
<mmagr> pixelb, so you were write, puppet-neutron's neutron_plugin_ovs provider uses ini_file resource from inifile module which simply opens config file with 'w', which is what I think rewrites permission on the config file ... since all openstack related puppet modules are using ini_file to modify configs we will have this problem with all components ...
<mmagr> pixelb, will implement fix for pupet-neutron, but we will have to fix that on ini_file level in future
Comment 8 Martin Magr 2014-03-25 18:41:33 UTC 
Submitted patch is to make sure ownership is in state it is supposed to be. Bug #1080560 should be reopened so Neutron guys can fix that on packaging side ... if it is problem in Neutron package.
Comment 9 Pádraig Brady 2014-04-04 00:24:31 UTC 
I think this is just a neutron packaging bug Martin and so the packstack workaround might best be left out.
Comment 10 Jakub Libosvar 2014-04-07 14:37:25 UTC 
Neutron side fixed in bug 1080560