Bug 1387378

Summary: [rhos-osp-9][Assure updates to rhel7.3] Minor update that includes rhel72 to rhel73 switch fails with running instance (on SSL environment ) .
Product: Red Hat OpenStack Reporter: Omri Hochman <ohochman>
Component: openstack-puppet-modulesAssignee: Emilien Macchi <emacchi>
Status: CLOSED CURRENTRELEASE QA Contact: Alexander Chuzhoy <sasha>
Severity: urgent Docs Contact:
Priority: high    
Version: 9.0 (Mitaka)CC: augol, cpaquin, dbecker, emacchi, jguiditt, jslagle, mburns, mfuruta, morazi, rhel-osp-director-maint, srevivo
Target Milestone: ---Keywords: TestOnly, Triaged, ZStream
Target Release: 9.0 (Mitaka)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-puppet-modules-8.1.8-3.el7ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-29 15:07:21 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 1386373    

Description Omri Hochman 2016-10-20 17:47:36 UTC
[rhos-osp-9][Assure updates to rhel7.3] Minor update that includes rhel72 to rhel73 switch fails with running instance (on SSL environment )  . 



Environment:
------------
openstack-puppet-modules-8.1.8-2.el7ost.noarch
openstack-tripleo-puppet-elements-2.0.0-4.el7ost.noarch
puppet-3.6.2-4.el7sat.noarch
openstack-heat-templates-0-0.8.20150605git.el7ost.noarch
openstack-heat-api-cfn-6.0.0-11.el7ost.noarch
openstack-tripleo-heat-templates-2.0.0-35.el7ost.noarch
heat-cfntools-1.3.0-2.el7ost.noarch
openstack-tripleo-heat-templates-liberty-2.0.0-35.el7ost.noarch
python-heat-tests-6.0.0-11.el7ost.noarch
openstack-heat-common-6.0.0-11.el7ost.noarch
openstack-heat-api-6.0.0-11.el7ost.noarch
python-heatclient-1.2.0-1.el7ost.noarch
openstack-heat-engine-6.0.0-11.el7ost.noarch


Steps: 
------
(1) Deploy Osp9 environment on BM with SSL  (use : rhel7.2 overcloud-images)
(2) launch_instance to run on the compute_node 
(3) Run Minor-update of the environment to get it to rhel7.3 

Results: 
--------
[stack@undercloud72 ~]$ heat stack-list
+--------------------------------------+------------+---------------+---------------------+---------------------+
| id                                   | stack_name | stack_status  | creation_time       | updated_time        |
+--------------------------------------+------------+---------------+---------------------+---------------------+
| 1738cee5-d984-4af5-b9b1-ca49d903b4fc | overcloud  | UPDATE_FAILED | 2016-08-23T17:43:03 | 2016-10-19T20:31:41 |
+--------------------------------------+------------+---------------+---------------------+---------------------+

[stack@undercloud72 ~]$ heat resource-list overcloud -n 5 | grep -v COMPLETE
-----------------+---------------------+----------------------------------------
| resource_name                                 | physical_resource_id                          | resource_type                                                                   | resource_status | updated_time        | stack_name                                                                                                                                        |
+-----------------------------------------------+-------------------------------
| ComputeNodesPostDeployment                    | f33fa7eb-e20b-4828-8d10-3cc9b4162911          | OS::TripleO::ComputePostDeployment                                              | UPDATE_FAILED   | 2016-10-19T22:00:26 | overcloud                                                                                                                                         |
+-----------------------------------------------+-------------------------------





Seems to be related to the following error:
---------------------------------------------
Error: /Stage[main]/Neutron/Resources[neutron_config]: Failed to generate additional resources using 'generate': OpenStackConfig only support collecting instances when a file path is hard coded
(heat-config) [INFO] Completed /var/lib/heat-config/heat-config-puppet/3ef310a3-cc47-4d22-b52f-c79881d9f0e5.pp
 

from compute-node:
------------------
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::vncproxy_protocol'; class ::nova::vncproxy has not been evaluated
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::port'; class ::nova::vncproxy has not been evaluated
Warning: Scope(Class[Nova::Vncproxy::Common]): Could not look up qualified variable '::nova::vncproxy::vncproxy_path'; class ::nova::vncproxy has not been evaluated
Warning: Scope(Class[Ceilometer::Agent::Compute]): This class is deprecated. Please use ceilometer::agent::polling with compute namespace instead.
Warning: The package type's allow_virtual parameter will be changing its default value from false to true in a future release. If you do not want to allow virtual packages, please explicitly set allow_virtual to false.
(at /usr/share/ruby/vendor_ruby/puppet/type.rb:816:in `set_default')
Error: /Stage[main]/Neutron/Resources[neutron_config]: Failed to generate additional resources using 'generate': OpenStackConfig only support collecting instances when a file path is hard coded
(heat-config) [INFO] Completed /var/lib/heat-config/heat-config-puppet/3ef310a3-cc47-4d22-b52f-c79881d9f0e5.pp
(heat-config) [INFO] Completed /var/lib/heat-config/hooks/puppet
(heat-config) [DEBUG] Running heat-config-notify /var/lib/heat-config/deployed/3ef310a3-cc47-4d22-b52f-c79881d9f0e5.json < /var/lib/heat-config/deployed/3ef310a3-cc47-4d22-b52f-c79881d9f0e5.notify.json
192.168.0.9]:161
192.168.0.9]:161
(heat-config) [INFO]
(heat-config) [DEBUG] [2016-10-20 00:58:06,139] (heat-config-notify) [DEBUG] Signaling to http://192.168.0.1:8000/v1/signal/arn%3Aopenstack%3Aheat%3A%3A5c97a30686f249ea86a25bc5b724aa58%3Astacks%2Fovercloud-ComputeNode
(heat-config-notify) [DEBUG] Response <Response [200]>
(heat-config) [WARNING] Skipping group os-apply-config with no hook script /var/lib/heat-config/hooks/os-apply-config
(heat-config) [WARNING] Skipping group os-apply-config with no hook script /var/lib/heat-config/hooks/os-apply-config
(heat-config) [WARNING] Skipping config ad45292f-439c-482d-aa2b-ebf8d175b0c0, already deployed
(heat-config) [WARNING] To force-deploy, rm /var/lib/heat-config/deployed/ad45292f-439c-482d-aa2b-ebf8d175b0c0.json
(heat-config) [WARNING] Skipping config 0c5112f4-765e-45b4-b1fd-39ba5a3e300c, already deployed
(heat-config) [WARNING] To force-deploy, rm /var/lib/heat-config/deployed/0c5112f4-765e-45b4-b1fd-39ba5a3e300c.json
8 UTC 2016 55-heat-config completed

Comment 2 Lukas Bezdicka 2016-10-20 19:20:44 UTC
openstack-puppet-modules-8.1.8-2.el7ost.noarch
file neutron/lib/puppet/provider/neutron_config/openstackconfig.rb does not seem to contain https://review.openstack.org/#/c/359475/

Comment 4 Jason Guiditta 2016-10-21 14:04:43 UTC
puppet-neutron updated

Comment 6 Masaki Furuta 2016-11-09 08:28:36 UTC
Emilien Macchi,

I also get similar error report from customer on the customer portal case when they're using template file they used with osp8 for upgrade testing from Red Hat OpenStack Platform 8 to 9.

They saw the followings:

> [director01-osp9.dev-sirius:stack/788]$openstack software deployment output show 5d070b47-f404-471f-91e3-b7ff62c2e9a2 deploy_stderr
> output_value:
> 
> deploy_stderr: |
> ...
> Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
> Warning: Not collecting exported resources without storeconfigs
> Warning: Not collecting exported resources without storeconfigs
> Warning: Not collecting exported resources without storeconfigs
> Error: /Stage[main]/Neutron/Resources[neutron_config]: Failed to generate additional resources using 'generate': OpenStackConfig only support collecting instances when a file path is hard coded
> Error: Could not start Service[httpd]: Execution of '/usr/bin/systemctl start httpd' returned 1: Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
> Wrapped exception:
> Execution of '/usr/bin/systemctl start httpd' returned 1: Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
> Error: /Stage[main]/Apache::Service/Service[httpd]/ensure: change from stopped to running failed: Could not start Service[httpd]: Execution of '/usr/bin/systemctl start httpd' returned 1: Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
> Warning: /Stage[main]/Keystone::Deps/Anchor[keystone::service::end]: Skipping because of failed dependencies
(truncated, view all with --long)


Now they'd know if there is any workaround , if it's taken over without SSL ?

Thank you.

Comment 7 James Slagle 2016-11-09 12:29:03 UTC
(In reply to Masaki Furuta from comment #6)
> Emilien Macchi,
> 
> I also get similar error report from customer on the customer portal case
> when they're using template file they used with osp8 for upgrade testing
> from Red Hat OpenStack Platform 8 to 9.
> 
> They saw the followings:
> 
> > [director01-osp9.dev-sirius:stack/788]$openstack software deployment output show 5d070b47-f404-471f-91e3-b7ff62c2e9a2 deploy_stderr
> > output_value:
> > 
> > deploy_stderr: |
> > ...
> > Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
> > Warning: Not collecting exported resources without storeconfigs
> > Warning: Not collecting exported resources without storeconfigs
> > Warning: Not collecting exported resources without storeconfigs
> > Error: /Stage[main]/Neutron/Resources[neutron_config]: Failed to generate additional resources using 'generate': OpenStackConfig only support collecting instances when a file path is hard coded
> > Error: Could not start Service[httpd]: Execution of '/usr/bin/systemctl start httpd' returned 1: Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
> > Wrapped exception:
> > Execution of '/usr/bin/systemctl start httpd' returned 1: Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
> > Error: /Stage[main]/Apache::Service/Service[httpd]/ensure: change from stopped to running failed: Could not start Service[httpd]: Execution of '/usr/bin/systemctl start httpd' returned 1: Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
> > Warning: /Stage[main]/Keystone::Deps/Anchor[keystone::service::end]: Skipping because of failed dependencies
> (truncated, view all with --long)
> 
> 
> Now they'd know if there is any workaround , if it's taken over without SSL ?
> 
> Thank you.

there could be many reasons why httpd might fail to start. i'd suggest filing a new bug with all the relevant information (logs, etc) so that we can track it as a new issues, especially since they are using a different openstack version than this bug was reported against.

if needed, you can escalate the bugzilla through the rhos-prio escalation procedure.

Comment 8 Masaki Furuta 2016-11-10 08:44:14 UTC
(In reply to James Slagle from comment #7)
> 
> there could be many reasons why httpd might fail to start. i'd suggest
> filing a new bug with all the relevant information (logs, etc) so that we
> can track it as a new issues, especially since they are using a different
> openstack version than this bug was reported against.
> 
> if needed, you can escalate the bugzilla through the rhos-prio escalation
> procedure.

Sure, Thanks for it, I'll bring out with httpd error on another bz !

Btw ,based on Error regarding to Neutron in the log, I guess customer also hit this bz too.

If it's true, can we work neutron related error around with disabling SSL or any suggestion worth trying to do ?

Comment 9 James Slagle 2016-11-10 18:27:22 UTC
(In reply to Masaki Furuta from comment #8)
> (In reply to James Slagle from comment #7)
> > 
> > there could be many reasons why httpd might fail to start. i'd suggest
> > filing a new bug with all the relevant information (logs, etc) so that we
> > can track it as a new issues, especially since they are using a different
> > openstack version than this bug was reported against.
> > 
> > if needed, you can escalate the bugzilla through the rhos-prio escalation
> > procedure.
> 
> Sure, Thanks for it, I'll bring out with httpd error on another bz !
> 
> Btw ,based on Error regarding to Neutron in the log, I guess customer also
> hit this bz too.
> 
> If it's true, can we work neutron related error around with disabling SSL or
> any suggestion worth trying to do ?

I wouldn't recommend disabling SSL at this point. There's no indication that SSL might be related to the issue.

Comment 12 Alexander Chuzhoy 2016-11-29 02:13:32 UTC
Verified:

Environment:
openstack-puppet-modules-8.1.8-3.el7ost

Was able to minor update a deployment with ssl.