Bug 1560573 - Overcloud deployment RHOS12 failed on THT compat installation 'stack (overcloud) could not be found'
Summary: Overcloud deployment RHOS12 failed on THT compat installation 'stack (overclo...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates-compat
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
medium
high
Target Milestone: beta
: 13.0 (Queens)
Assignee: Jiri Stransky
QA Contact: Ronnie Rasouli
URL:
Whiteboard: DFG:upgrades
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-03-26 13:18 UTC by Ronnie Rasouli
Modified: 2018-06-27 13:49 UTC (History)
6 users (show)

Fixed In Version: openstack-tripleo-heat-templates-compat-7.0.10-0.20180221180818.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-27 13:48:49 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:2086 None None None 2018-06-27 13:49:42 UTC

Description Ronnie Rasouli 2018-03-26 13:18:02 UTC
Description of problem:

Deploying containerized RHOS12 overcloud on undercloud RHOS13 (Compat installation) fails with error:

404 Client Error: Not Found for url: https://192.168.24.2:13808/v1/AUTH_21131430022a48338d4f0c323332ad4a/overcloud/common/services.yaml

Version-Release number of selected component (if applicable):

The file exist on /usr/share/openstack-tripleo-heat-templates/common/services.yaml

However on /usr/share/openstack-tripleo-heat-templates/compat it isn't there

There is a job for this scenario on staging server 

http://staging-jenkins2-qe-playground.usersys.redhat.com/view/DFG/view/upgrades/view/bw-compat/job/DFG-upgrades-bw-compat-mixed-versions-13-director-compat-deploy-12-rhel-7.5-virt-HA-ipv4-poc/9/

Deployment on 3 CEPH, 2 COMPUTES, 3 CONTROLLERS, 1 undercloud



How reproducible:
100%

Steps to Reproduce:
1. install the undercloud
2. prepare the THT for compat package
3. deploy RHOS12 containers with CEPH

Actual results:

Overcloud deployment failed

Expected results:

Include the services.yaml 

Additional info:

Comment 1 Ronnie Rasouli 2018-03-26 13:34:11 UTC
Deployment command:
openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates/compat/ \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /home/stack/virt/internal.yaml \
-e /usr/share/openstack-tripleo-heat-templates/compat//environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/hostnames.yml \
-e /usr/share/openstack-tripleo-heat-templates/compat//environments/ceph-ansible/ceph-ansible.yaml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/docker-images.yaml \
--log-file overcloud_deployment_29.log

Comment 2 Jiri Stransky 2018-03-26 15:59:17 UTC
The RPM spec had outdated list of directories.

Comment 7 Ronnie Rasouli 2018-03-29 15:40:28 UTC
The services.yaml exist on /usr/share/openstack-tripleo-heat-templates/compat/common/services.yaml.

The deployment passed that stage and found services.yaml

Comment 10 Ronnie Rasouli 2018-06-04 13:30:58 UTC
Issue hasn't been resolved
run-os-net-config.sh is missing


2018-06-04 08:22:17.703 20266 WARNING tripleoclient.plugin [  admin] Waiting for messages on queue 'tripleo' with no timeout.
2018-06-04 08:23:12.359 20266 ERROR openstack [  admin] Action tripleo.parameters.update execution failed: 404 Client Error: Not Found for url: https://192.168.24.2:13808/v1/AUTH_d8d1aafbb90347ce88ccb43106741a3a/overcloud/user-files/usr/share/openstack-tripleo-heat-templates/compat/network/scripts/run-os-net-config.sh

Comment 12 Jiri Stransky 2018-06-04 13:41:45 UTC
I just checked the RPM spec and the file should be getting copied over. This is in the RPM spec:

cp -ar network %{buildroot}/%{_datadir}/openstack-%{upstream_name}/compat

So the root cause is probably not the same as the last time.

Comment 13 Jiri Stransky 2018-06-04 13:58:57 UTC
I'm looking at yum.log on undercloud and the only -compat mention i see there is:

Jun 04 07:35:57 Installed: openstack-tripleo-heat-templates-compat-7.0.9-8.1.el7ost.noarch

^ That's older than the fixed-in-version field of the BZ. Maybe the repos were misconfigured? Should we move back to ON_QA and retest?

Comment 14 Jiri Stransky 2018-06-05 07:51:48 UTC
Investigating further, the file in question is present:

[stack@undercloud-0 ~]$ ll /usr/share/openstack-tripleo-heat-templates/compat/network/scripts/run-os-net-config.sh
-rwxr-xr-x. 1 root root 4683 12. úno 09.09 /usr/share/openstack-tripleo-heat-templates/compat/network/scripts/run-os-net-config.sh

but for some reason it seems that we're trying to look for it under user-files in the deployment plan (see error message in comment #10). I'm not sure but i suspect the double slashes in deploy command might be the issue:

[stack@undercloud-0 ~]$ cat overcloud_deploy.sh 
#!/bin/bash

openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates/compat/ \
--stack overcloud \
--libvirt-type kvm \
--ntp-server clock.redhat.com \
-e /home/stack/virt/internal.yaml \
-e /usr/share/openstack-tripleo-heat-templates/compat//environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/hostnames.yml \
-e /usr/share/openstack-tripleo-heat-templates/compat//environments/ceph-ansible/ceph-ansible.yaml \
-e /home/stack/virt/debug.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/docker-images.yaml \
--log-file overcloud_deployment_40.log

If we control this with an ansible variable set to:

/usr/share/openstack-tripleo-heat-templates/compat/

let's rather try to set it to

/usr/share/openstack-tripleo-heat-templates/compat

and see if it gets us further.

Comment 15 Ronnie Rasouli 2018-06-05 15:07:04 UTC
Closing this bug since another issue with CEPH has been revealed

Comment 17 errata-xmlrpc 2018-06-27 13:48:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086


Note You need to log in before you can comment on or make changes to this bug.