Bug 1304088 - rhel-osp-director: OC with cinder and ceph nodes update 7.0->7.3 failed "Exec[ceph-key-client.openstack]/returns: sh: -c: line 2: unexpected EOF while looking for matching `'"
rhel-osp-director: OC with cinder and ceph nodes update 7.0->7.3 failed "Exec...
Status: CLOSED INSUFFICIENT_DATA
Product: Red Hat OpenStack
Classification: Red Hat
Component: rhosp-director (Show other bugs)
7.0 (Kilo)
Unspecified Unspecified
high Severity unspecified
: y3
: 7.0 (Kilo)
Assigned To: Sofer Athlan-Guyot
yeylon@redhat.com
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2016-02-02 15:29 EST by Alexander Chuzhoy
Modified: 2017-04-12 09:40 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-02-09 05:00:06 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Alexander Chuzhoy 2016-02-02 15:29:59 EST
rhel-osp-director: OC with cinder and ceph nodes update 7.0->7.3 failed
Environment:
openstack-tripleo-heat-templates-0.8.6-112.el7ost.noarch
instack-undercloud-2.1.2-37.el7ost.noarch
openstack-puppet-modules-2015.1.8-45.el7ost.noarch

Steps to reproduce:
1. Deploy overcloud
2. Populate the overcloud with some objects
3. Attempt to update the setup to 7.3

Result:
IN_PROGRESS                                                                                                           
FAILED                                                                                                                
update finished with status FAILED    


[stack@instack ~]$ heat resource-list -n5 overcloud|grep -v COMPLE                                                                                                                                              
+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+-----------------------------------------------+                                                                                                                                                                                            
| resource_name                                 | physical_resource_id                          | resource_type                                     | resource_status | updated_time         | parent_resource                               |                                                                                                                                                                                            
+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+-----------------------------------------------+                                                                                                                                                                                            
| ComputeNodesPostDeployment                    | 32c4d76a-b8a3-43f7-97ef-941cb7200389          | OS::TripleO::ComputePostDeployment                | UPDATE_FAILED   | 2016-02-02T18:21:57Z |                                               |                                                                                                                                                                                            
| ComputePuppetDeployment                       | 7bd1644e-6975-45ed-a3b4-4bbd95eb652e          | OS::Heat::StructuredDeployments                   | UPDATE_FAILED   | 2016-02-02T18:22:13Z | ComputeNodesPostDeployment                    |                                                                                                                                                                                            
| 0                                             | ace372ec-855c-4d5a-972b-8572f4117703          | OS::Heat::StructuredDeployment                    | UPDATE_FAILED   | 2016-02-02T18:22:18Z | ComputePuppetDeployment                       |                                                                                                                                                                                            
| ControllerNodesPostDeployment                 | c56124af-903f-47e0-a49b-69f570d3718a          | OS::TripleO::ControllerPostDeployment             | UPDATE_FAILED   | 2016-02-02T18:22:48Z |                                               |                                                                                                                                                                                            
| ControllerPostPuppet                          | 8a871534-fd61-46cd-b380-47fa37a255e1          | OS::TripleO::Tasks::ControllerPostPuppet          | CREATE_FAILED   | 2016-02-02T18:40:30Z | ControllerNodesPostDeployment                 |                                                                                                                                                                                            
| ControllerPostPuppetRestartDeployment         | 094f747e-23f3-4f72-ab34-d7315f16fbc5          | OS::Heat::SoftwareDeployments                     | CREATE_FAILED   | 2016-02-02T18:40:33Z | ControllerPostPuppet                          |                                                                                                                                                                                            
| 0                                             | 8e3b2db3-7c0c-4267-874a-c4ba7a49ff29          | OS::Heat::SoftwareDeployment                      | CREATE_FAILED   | 2016-02-02T18:41:51Z | ControllerPostPuppetRestartDeployment         |                                                                                                                                                                                            
+-----------------------------------------------+-----------------------------------------------+---------------------------------------------------+-----------------+----------------------+-----------------------------------------------+   

heat deployment-show ace372ec-855c-4d5a-972b-8572f4117703
{                                                                           
  "status": "FAILED",                                                       
  "server_id": "0ae85f9a-2e0c-47d4-9a6e-997d4a2e61a9",                      
  "config_id": "2276b8e0-cc60-48f9-a1eb-80955e29e0cf",                      
  "output_values": {                                                        
    "deploy_stdout": "\u001b[mNotice: Compiled catalog for overcloud-compute-0.localdomain in environment production in 4.60 seconds\u001b[0m\n\u001b[mNotice: /Stage[main]/Tripleo::Packages/Exec[package-upgrade]/returns: executed successfully\u001b[0m\n\u001b[mNotice: /Stage[main]/Main/Package_manifest[/var/lib/tripleo/installed-packages/overcloud_compute]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Neutron/File[/etc/neutron]/group: group changed 'root' to 'neutron'\u001b[0m\n\u001b[mNotice: /Stage[main]/Neutron/Neutron_config[oslo_messaging_rabbit/heartbeat_rate]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Neutron/Neutron_config[DEFAULT/service_plugins]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Neutron/Neutron_config[DEFAULT/core_plugin]/value: value changed 'openvswitch' to 'ml2'\u001b[0m\n\u001b[mNotice: /Stage[main]/Nova::Compute/Package[pm-utils]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph/Ceph_config[global/osd_pool_default_pgp_num]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph::Keys/Ceph::Key[client.openstack]/File[/etc/ceph/ceph.client.openstack.keyring]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph/Ceph_config[global/osd_pool_default_min_size]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph/Ceph_config[global/auth_service_required]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph/Ceph_config[global/fsid]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph/Ceph_config[global/auth_supported]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph/Ceph_config[global/auth_cluster_required]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Nova/Nova_config[DEFAULT/use_ipv6]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph::Keys/Ceph::Key[client.admin]/File[/etc/ceph/ceph.client.admin.keyring]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Nova/Nova_config[oslo_messaging_rabbit/heartbeat_rate]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Nova/Nova_config[oslo_messaging_rabbit/heartbeat_timeout_threshold]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph/Ceph_config[global/mon_host]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph::Keys/Ceph::Key[client.openstack]/Exec[ceph-key-client.openstack]/returns: sh: -c: line 2: unexpected EOF while looking for matching `''\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer/File[/etc/ceilometer/]/owner: owner changed 'root' to 'ceilometer'\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer/File[/etc/ceilometer/]/group: group changed 'root' to 'ceilometer'\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer/File[/etc/ceilometer/]/mode: mode changed '0755' to '0750'\u001b[0m\n\u001b[mNotice: /Stage[main]/Nova::Network::Neutron/Nova_config[DEFAULT/vif_plugging_is_fatal]/value: value changed 'False' to 'True'\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph::Keys/Ceph::Key[client.bootstrap-osd]/File[/var/lib/ceph/bootstrap-osd/ceph.keyring]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph::Keys/Ceph::Key[client.bootstrap-osd]/Exec[ceph-key-client.bootstrap-osd]/returns: sh: -c: line 2: unexpected EOF while looking for matching `''\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph/Ceph_config[global/auth_client_required]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer/Ceilometer_config[oslo_messaging_rabbit/heartbeat_timeout_threshold]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph/Ceph_config[global/osd_pool_default_size]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph::Keys/Ceph::Key[client.admin]/Exec[ceph-key-client.admin]/returns: sh: -c: line 2: unexpected EOF while looking for matching `''\u001b[0m\n\u001b[mNotice: /Stage[main]/Nova::Compute::Rbd/Nova_config[libvirt/images_rbd_pool]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Nova::Compute::Rbd/Nova_config[libvirt/images_type]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Nova::Compute::Rbd/Nova_config[libvirt/images_rbd_ceph_conf]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer::Agent::Auth/Ceilometer_config[service_credentials/os_endpoint_type]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceph/Ceph_config[global/osd_pool_default_pg_num]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Neutron::Plugins::Ml2/Neutron_plugin_ml2[ml2/type_drivers]/value: value changed 'flat,gre,vxlan,vlan' to 'vxlan,vlan,flat,gre'\u001b[0m\n\u001b[mNotice: /Stage[main]/Neutron/Neutron_config[oslo_messaging_rabbit/heartbeat_timeout_threshold]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer/Ceilometer_config[oslo_messaging_rabbit/heartbeat_rate]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Nova::Network::Neutron/Nova_config[DEFAULT/vif_plugging_timeout]/value: value changed '30' to '300'\u001b[0m\n\u001b[mNotice: /Stage[main]/Nova::Compute::Rbd/Nova_config[libvirt/rbd_user]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Ceilometer::Agent::Compute/Service[ceilometer-agent-compute]: Triggered 'refresh' from 3 events\u001b[0m\n\u001b[mNotice: /Stage[main]/Nova/Exec[post-nova_config]: Triggered 'refresh' from 9 events\u001b[0m\n\u001b[mNotice: /Stage[main]/Nova::Compute/Nova::Generic_service[compute]/Service[nova-compute]: Triggered 'refresh' from 1 events\u001b[0m\n\u001b[mNotice: Finished catalog run in 608.05 seconds\u001b[0m\n",
    "deploy_stderr": "Device \"br_ex\" does not exist.\nDevice \"br_int\" does not exist.\nDevice \"br_tun\" does not exist.\nDevice \"ovs_system\" does not exist.\nDevice \"qbrd23abf0e_bb\" does not exist.\nDevice \"qvbd23abf0e_bb\" does not exist.\nDevice \"qvod23abf0e_bb\" does not exist.\nDevice \"tapd23abf0e_bb\" does not exist.\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::host'; class ::nova::vncproxy has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::vncproxy_protocol'; class ::nova::vncproxy has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::port'; class ::nova::vncproxy has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::vncproxy_path'; class ::nova::vncproxy has not been evaluated\u001b[0m\n\u001b[1;31mWarning: Scope(Class[Ceilometer::Agent::Compute]): This class is deprecated. Please use ceilometer::agent::polling with compute namespace instead.\u001b[0m\n\u001b[1;31mWarning: The package type's allow_virtual parameter will be changing its default value from false to true in a future release. If you do not want to allow virtual packages, please explicitly set allow_virtual to false.\n   (at /usr/share/ruby/vendor_ruby/puppet/type.rb:816:in `set_default')\u001b[0m\n\u001b[1;31mError: /bin/true # comment to satisfy puppet syntax requirements\nset -ex\nceph-authtool /etc/ceph/ceph.client.openstack.keyring --name 'client.openstack' --add-key ''' --cap mon 'allow r' --cap osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rwx pool=images'  returned 2 instead of one of [0]\u001b[0m\n\u001b[1;31mError: /Stage[main]/Ceph::Keys/Ceph::Key[client.openstack]/Exec[ceph-key-client.openstack]/returns: change from notrun to 0 failed: /bin/true # comment to satisfy puppet syntax requirements\nset -ex\nceph-authtool /etc/ceph/ceph.client.openstack.keyring --name 'client.openstack' --add-key ''' --cap mon 'allow r' --cap osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rwx pool=images'  returned 2 instead of one of [0]\u001b[0m\n\u001b[1;31mError: /bin/true # comment to satisfy puppet syntax requirements\nset -ex\nceph-authtool /var/lib/ceph/bootstrap-osd/ceph.keyring --name 'client.bootstrap-osd' --add-key ''' --cap mon 'allow profile bootstrap-osd'  returned 2 instead of one of [0]\u001b[0m\n\u001b[1;31mError: /Stage[main]/Ceph::Keys/Ceph::Key[client.bootstrap-osd]/Exec[ceph-key-client.bootstrap-osd]/returns: change from notrun to 0 failed: /bin/true # comment to satisfy puppet syntax requirements\nset -ex\nceph-authtool /var/lib/ceph/bootstrap-osd/ceph.keyring --name 'client.bootstrap-osd' --add-key ''' --cap mon 'allow profile bootstrap-osd'  returned 2 instead of one of [0]\u001b[0m\n\u001b[1;31mError: /bin/true # comment to satisfy puppet syntax requirements\nset -ex\nceph-authtool /etc/ceph/ceph.client.admin.keyring --name 'client.admin' --add-key ''' --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *'  returned 2 instead of one of [0]\u001b[0m\n\u001b[1;31mError: /Stage[main]/Ceph::Keys/Ceph::Key[client.admin]/Exec[ceph-key-client.admin]/returns: change from notrun to 0 failed: /bin/true # comment to satisfy puppet syntax requirements\nset -ex\nceph-authtool /etc/ceph/ceph.client.admin.keyring --name 'client.admin' --add-key ''' --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *'  returned 2 instead of one of [0]\u001b[0m\n",
    "deploy_status_code": 6
  },
  "creation_time": "2016-02-02T16:08:34Z",
  "updated_time": "2016-02-02T18:33:06Z",
  "input_values": {},
  "action": "UPDATE",
  "status_reason": "deploy_status_code : Deployment exited with non-zero status code: 6",
  "id": "ace372ec-855c-4d5a-972b-8572f4117703"
}
Comment 2 Sofer Athlan-Guyot 2016-02-03 12:43:45 EST
Can you paste the command (and eventually any modified yaml) used for:
 - the initial setup;
 - the upgrade;

Thanks.
Comment 3 Sofer Athlan-Guyot 2016-02-03 12:46:14 EST
In the meantime, looking at the code, the problem seems to be that the ceph_admin_key is set to "'".  Will need the above information to get further and try reproduce the problem.
Comment 7 Angus Thomas 2016-02-07 11:14:04 EST
Given that we're not yet able to reproducr this, I'm going to remove this from the 7.3 release blockers on on February 9th unless we get sufficient information for further analysis.
Comment 9 Angus Thomas 2016-02-09 05:00:06 EST
I'm closing this bug, as it hasn't been possible to reproduce it and to provide additional diagnostic information. 

Please re-open the bug if the problem can be reproduced.

Note You need to log in before you can comment on or make changes to this bug.