Bug 1659112

Summary: OSP12 -> OSP13 Prevalidation check 'Check if nova_wsgi is running' tries to validate a service that is already in container
Product: Red Hat OpenStack Reporter: ggrimaux
Component: openstack-tripleo-heat-templatesAssignee: Rajesh Tailor <ratailor>
Status: CLOSED WORKSFORME QA Contact: Gurenko Alex <agurenko>
Severity: high Docs Contact:
Priority: unspecified    
Version: 12.0 (Pike)CC: ccamacho, emacchi, ggrimaux, jfrancoa, lyarwood, mburns, mschuppe, owalsh, ratailor
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-02-28 06:03:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description ggrimaux 2018-12-13 15:34:03 UTC
Description of problem:
During an OSP12 to OSP13 upgrade the following prevalidation check fails:

        - name: "PreUpgrade step0,validation: Check if nova_wsgi is running"
          tags: step0,validation
          shell: systemctl status 'httpd' | grep -q 'nova'
          when: httpd_enabled.rc == 0 and httpd_running.rc == 0

But on OSP12 nova_api is already containerized.

# systemctl status httpd | grep -q nova
# echo $?
1

The controller's Apache is running but just not handling that vhost anymore.

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-7.0.12-17.el7ost

How reproducible:
100%

Steps to Reproduce:
1. On a OSP12 environment run 'systemctl status httpd | grep -q nova' && echo $?
2.
3.

Actual results:
Upgrade fails

Expected results:
Upgrade should continue 

Additional info:

Comment 1 Carlos Camacho 2018-12-17 13:18:06 UTC
For what I can see[1], you are running THT from Pike instead of Queens.
When you run an upgrade from 12 to 13 you must have THT from 13 instead 12, thus the validation[2] should work.

Please, can you confirm you are following the documentation correctly?

[1]: https://github.com/openstack/tripleo-heat-templates/blob/stable/pike/docker/services/nova-api.yaml#L385
[2]: https://github.com/openstack/tripleo-heat-templates/blob/stable/queens/docker/services/nova-api.yaml#L378

Comment 2 ggrimaux 2018-12-19 11:32:10 UTC
I'm sorry I don't know why I put the wrong version the client is using:

It should be openstack-tripleo-heat-templates-8.0.7-4.el7ost

[ggrimaux@collab-shell undercloud]$ grep openstack-tripleo-heat-templates installed-rpms 
openstack-tripleo-heat-templates-8.0.7-4.el7ost.noarch      Fri Dec  7 10:33:21 2018

So it's for OSP13 as it should.

Thank you.

Comment 3 Carlos Camacho 2019-01-02 10:50:08 UTC
Please feel free to reopen this in the future if you have issues upgrading THT.

Comment 7 Carlos Camacho 2019-01-21 11:41:19 UTC
Can you double confirm that this deployment was never upgraded before?

Thanks!

Comment 8 ggrimaux 2019-01-21 11:48:46 UTC
This was done on a new OSP12 installation. Not upgraded from OSP11 to OSP12 previously.

I confirmed it with the client.

Thank you!

Comment 9 Carlos Camacho 2019-01-21 13:23:59 UTC
After checking the templates I'm not able to spot why the deployment failed in this way, eve if its working it should be in that way.

So after checking that the services are actually running under apache `httpd -t -D DUMP_VHOSTS | grep -q 'nova'` and `systemctl is-active --quiet openstack-nova-api`

We should proceed to disable them `sudo systemctl stop openstack-nova-api` and `sudo systemctl disable openstack-nova-api`. This will make the service still run from the container and the validation should pass.

In my opinion, when running the Overcloud upgrade for going to 12->13 we should not skip the pre-upgrade validations.

Comment 14 Rajesh Tailor 2019-02-28 06:03:16 UTC
Closing this BZ as it is not reproducible with latest THT templates.

Please feel free to reopen this in the future if you have face this issue again.