Bug 1624081 - Openshift deployment via Director fails when re-running the overcloud deploy command used for initial deployment
Summary: Openshift deployment via Director fails when re-running the overcloud deploy ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 14.0 (Rocky)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: rc
: 14.0 (Rocky)
Assignee: John Trowbridge
QA Contact: Gurenko Alex
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-30 21:21 UTC by Marius Cornea
Modified: 2019-01-11 11:53 UTC (History)
8 users (show)

Fixed In Version: openstack-tripleo-heat-templates-9.0.1-0.20181013060882.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-01-11 11:51:51 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
ansible.log (19.32 MB, text/plain)
2018-08-30 21:24 UTC, Marius Cornea
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1794824 0 None None None 2018-09-27 16:06:58 UTC
OpenStack gerrit 605796 0 None None None 2018-09-27 16:06:34 UTC
OpenStack gerrit 608658 0 None None None 2018-10-08 18:48:43 UTC
Red Hat Product Errata RHEA-2019:0045 0 None None None 2019-01-11 11:53:01 UTC

Description Marius Cornea 2018-08-30 21:21:54 UTC
Description of problem:
Openshift deployment via Director fails when re-running the overcloud deploy command with:

TASK [generate openshift playbook] *********************************************
Thursday 30 August 2018  15:59:27 -0400 (0:00:00.050)       0:02:30.779 ******* 
fatal: [undercloud]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'openshift_upgrade' is undefined\n\nThe error appears to have been in '/var/lib/mistral/openshift/external_deploy_steps_tasks.yaml': line 124, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n      openshift_upgrade_playbook_path: /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade.yml\n  - copy:\n    ^ here\n"}


Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-9.0.0-0.20180818200902.cb08cb1.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
1. Deploy Openshift with 1 master + 1 worker:
openstack overcloud deploy \
--stack openshift \
--templates \
-r /home/stack/openshift_roles_data.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/openshift.yaml \
-e /home/stack/openshift_env.yaml \
-e /home/stack/containers-default-parameters.yaml

2. Make sure deployment is successful

3. Re-run the same overcloud deploy command

Actual results:
Fails.

Expected results:
Doesn't fail. Deployment should be idempotent.

Additional info:
Same failure occurs in case of a configuration change.

Comment 1 Marius Cornea 2018-08-30 21:24:50 UTC
Created attachment 1479913 [details]
ansible.log

Comment 2 John Trowbridge 2018-09-26 13:31:36 UTC
I have seen this as well. It is because we have different behaviour for a stack update[1]. Would it be possible to test with the fix to new node detection?[2]

[1] https://github.com/openstack/tripleo-heat-templates/blob/master/extraconfig/services/openshift-master.yaml#L341-L355
[2] https://review.openstack.org/600012

Comment 3 John Trowbridge 2018-09-27 16:09:59 UTC
I was able to reproduce this upstream even with the patch in my comment above (600012). I made and upstream bug[1] and patch[2] for it.

[1] https://bugs.launchpad.net/tripleo/+bug/1794824 
[2] https://review.openstack.org/605796

Comment 4 Marius Cornea 2018-09-28 22:08:17 UTC
(In reply to John Trowbridge from comment #3)
> I was able to reproduce this upstream even with the patch in my comment
> above (600012). I made and upstream bug[1] and patch[2] for it.
> 
> [1] https://bugs.launchpad.net/tripleo/+bug/1794824 
> [2] https://review.openstack.org/605796

I tried applying but next it fails on this task:

TASK [generate openshift playbook] *********************************************
Friday 28 September 2018  18:01:40 -0400 (0:00:00.052)       0:03:47.760 ****** 
fatal: [undercloud]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'new_masters' is undefined\n\nThe error appears to have been in '/var/lib/mistral/openshift/external_deploy_steps_tasks.yaml': line 158, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n      openshift_worker_scaleup_playbook_path: /usr/share/ansible/openshift-ansible/playbooks/openshift-node/scaleup.yml\n  - copy:\n    ^ here\n"}

Comment 5 John Trowbridge 2018-10-08 18:50:26 UTC
I was able to get past this second issue with Martin's fix for the update tasks:

https://review.openstack.org/608658

With that patch and the previous one I posted, I am able to rerun the deploy command as expected.

Comment 6 Martin André 2018-10-23 16:40:27 UTC
This is still missing https://review.openstack.org/#/c/605796/ from upstream.

Comment 25 Martin André 2019-01-10 10:26:45 UTC
No doc text required.

Comment 26 errata-xmlrpc 2019-01-11 11:51:51 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:0045


Note You need to log in before you can comment on or make changes to this bug.