Bug 1688461

Summary: config-download backports from RHOSP 14 to RHOSP 13 (openstack-tripleo-common)
Product: Red Hat OpenStack Reporter: Luke Short <lshort>
Component: openstack-tripleo-commonAssignee: Luke Short <lshort>
Status: CLOSED ERRATA QA Contact: David Rosenfeld <drosenfe>
Severity: medium Docs Contact:
Priority: medium    
Version: 13.0 (Queens)CC: bfournie, dvd, emacchi, jjoyce, jschluet, mburns, mcornea, mgeary, slinaber, tvignaud, vcojot
Target Milestone: z7Keywords: FeatureBackport, Triaged, ZStream
Target Release: 13.0 (Queens)   
Hardware: Unspecified   
OS: Linux   
Whiteboard:
Fixed In Version: openstack-tripleo-common-8.6.8-6.el7ost Doc Type: Enhancement
Doc Text:
There are two new CLI arguments you can use with `config-download`: - Monitor the deployment in a separate CLI session or with the API with `openstack overcloud status`. - Log and save Ansible errors for future analysis with `openstack overcloud failures`.
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-07-10 13:01:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Luke Short 2019-03-13 18:58:53 UTC
For providing a more streamlined, reliable, and faster experience of using Ansible for configuration management (config-download) in RHOSP 13, select patches from upstream Rocky were backported to upstream Queens. Most, if not all of these, should make it into the next RHOSP 13 zstream release. This issue is to document the relevant backports and enhancements to the product.

These are the primary patches and the type of enhancement they provide. Below them are the related dependency patches.

* tripleo-common
    * Don’t fail tripleo-bootstrap on package installs = Stability
        * https://review.openstack.org/#/c/636707/
    * Skip deployments with no uuid = Stability
        * https://review.openstack.org/#/c/637277/
    * Run NetworkDeployment as async task = Speed
        * https://review.openstack.org/#/c/636978/
* python-tripleoclient
    * Disable SSH password authentication during tripleo-admin enablement = Stability and security
        * https://review.openstack.org/#/c/633028/
    * openstack overcloud failures (4 patches) = Logging
        * https://review.openstack.org/#/c/639820/
        * https://review.openstack.org/#/c/619129/
        * https://review.openstack.org/#/c/642919/
        * https://review.openstack.org/#/c/619133/
    * Pass stack name to enable_ssh_admin_workflow (7 patches) = Stability
        * https://review.openstack.org/#/c/637636/
        * https://review.openstack.org/#/c/608465/
        * https://review.openstack.org/#/c/611855/
        * https://review.openstack.org/#/c/640179/
        * https://review.openstack.org/#/c/641090/
        * https://review.openstack.org/#/c/643060/
        * https://review.openstack.org/#/c/640125/
    * overcloud plan deployment status (2 patches) = Logging
        * https://review.openstack.org/#/c/639819/
        * https://review.openstack.org/#/c/639803/
* tripleo-heat-templates
    * Remove “when failed” from debug task names = Logging
        * https://review.openstack.org/#/c/638507/
    * Tag step plays = Usability
        * https://review.openstack.org/#/c/638503/
    * Tag tasks in in common tasks = Usability
        * https://review.openstack.org/#/c/641769/
    * Convert with_dict tasks to use loop and be less chatty = Logging
        * https://review.openstack.org/#/c/607920/

Comment 1 Bob Fournier 2019-03-17 14:23:44 UTC
All of these patches are handled by the DF DFG (not HardProv) so moving there.

Comment 4 Luke Short 2019-03-22 14:08:31 UTC
The "Pass stack name to enable_ssh_admin_workflow" patchset is being excluded since it is not upstream yet and would not land in time for 13z6.

Comment 11 Luke Short 2019-03-26 17:11:27 UTC
Unique Bugzilla reports have been opened up for each component:

* openstack-tripleo-common: (this report, 1688461)
* openstack-tripleo-heat-templates: https://bugzilla.redhat.com/show_bug.cgi?id=1692872
* python-tripleoclient: https://bugzilla.redhat.com/show_bug.cgi?id=1692932

Comment 18 Victor Voronkov 2019-04-17 13:22:09 UTC
Compose 2019-04-10.1 doesn't contain the relevant packages

Comment 19 Victor Voronkov 2019-04-18 09:28:20 UTC
config-download found in 2019-04-10.1

Comment 27 Victor Voronkov 2019-06-17 18:47:07 UTC
Verified on compose 2019-06-13.2

After re-depoy OC with config-download:
...
Ansible passed.
Overcloud configuration completed.
Started Mistral Workflow tripleo.deployment.v1.get_horizon_url. Execution ID: 83eca8ff-934e-4242-a54d-52523a8e178c
Overcloud Endpoint: https://10.0.0.101:13000/
Overcloud Horizon Dashboard URL: https://10.0.0.101:443/dashboard
Overcloud rc file: /home/stack/overcloudrc
Overcloud Deployed


openstack overcloud status

Started Mistral Workflow tripleo.deployment.v1.get_deployment_status. Execution ID: 7ee5083a-3724-44d1-9475-b6bd20c15c05

+-----------+---------------------+---------------------+-------------------+
| Plan Name |       Created       |       Updated       | Deployment Status |
+-----------+---------------------+---------------------+-------------------+
| overcloud | 2019-06-17 04:50:42 | 2019-06-17 04:50:42 |   DEPLOY_SUCCESS  |
+-----------+---------------------+---------------------+-------------------+

###################### config-download dir created under /tmp

sudo ls -l /tmp/ansible-mistral-actiongYUvFY
total 92
-rw-r--r--. 1 mistral mistral   604 Jun 17 00:50 ansible.cfg
-rw-r--r--. 1 mistral mistral     2 Jun 17 00:50 ansible-errors.json
-rw-r--r--. 1 mistral mistral 67014 Jun 17 00:50 ansible.log
-rwxr-x---. 1 mistral mistral   539 Jun 17 00:50 ansible-playbook-command.sh
drwx------. 2 mistral mistral     6 Jun 17 01:20 ansible-ssh
-rw-r--r--. 1 mistral mistral   128 Jun 17 00:50 inventory.yaml
-rw-r--r--. 1 mistral mistral  1227 Jun 17 00:50 playbook.yaml
-rw-------. 1 mistral mistral  3243 Jun 17 00:50 ssh_private_key

######## providing bogus NTP server to test failures

(undercloud) [stack@undercloud-0 ~]$ openstack overcloud status

Started Mistral Workflow tripleo.deployment.v1.get_deployment_status. Execution ID: 660dd7d3-39ab-4ea8-870d-7a0787bd48e0

+-----------+---------------------+---------------------+-------------------+
| Plan Name |       Created       |       Updated       | Deployment Status |
+-----------+---------------------+---------------------+-------------------+
| overcloud | 2019-06-17 11:11:05 | 2019-06-17 11:11:05 |   DEPLOY_FAILED   |
+-----------+---------------------+---------------------+-------------------+

(undercloud) [stack@undercloud-0 ~]$ openstack overcloud failures


Started Mistral Workflow tripleo.deployment.v1.get_deployment_failures. Execution ID: c70f7088-16c8-43e7-9459-4d2666e642ed

|-> Failures for host: controller-0
|--> Task: Output for sync deployment ControllerHostPrepDeployment
|---> _ansible_no_log: false
|---> _ansible_verbose_always: true
|---> changed: false
|---> failed_when_result: true
|---> msg: [
    {
        "stderr": [
            "[2019-06-17 07:13:20,765] (heat-config) [DEBUG] Running /usr/libexec/heat-config/hooks/ansible < /var/lib/heat-config/deployed/a9eceaef-a91a-414e-b70f-3575d4734eb2.json",
            "TASK [Ensure system is NTP time synced] ****************************************",
            "fatal: [localhost]: FAILED! => {\"changed\": true, \"cmd\": [\"ntpdate\", \"-u\", \"qwerty.dlt\"], \"delta\": \"0:00:00.007986\", \"end\": \"2019-06-17 07:13:50.946743\", \"msg\": \"
non-zero return code\", \"rc\": 1, \"start\": \"2019-06-17 07:13:50.938757\", \"stderr\": \"Error resolving qwerty.dlt: Name or service not known (-2)\\n17 Jun 07:13:50 ntpdate[18296]: Can't find
 host qwerty.dlt: Name or service not known (-2)\\n17 Jun 07:13:50 ntpdate[18296]: no servers can be used, exiting\", \"stderr_lines\": [\"Error resolving qwerty.dlt: Name or service not known (-
2)\", \"17 Jun 07:13:50 ntpdate[18296]: Can't find host qwerty.dlt: Name or service not known (-2)\", \"17 Jun 07:13:50 ntpdate[18296]: no servers can be used, exiting\"], \"stdout\": \"\", \"std
out_lines\": []}",
            "\tto retry, use: --limit @/var/lib/heat-config/heat-config-ansible/32408a85-2eea-406f-aa1c-42ba37e422ee_playbook.retry",
            "",
            "PLAY RECAP *********************************************************************",
            "localhost                  : ok=53   changed=46   unreachable=0    failed=1   ",
            "",
            "",
            "[2019-06-17 07:13:51,015] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-ansible/32408a85-2eea-406f-aa1c-42ba37e422ee_playbook.yaml. [2]",
            "",
            "",
            "[2019-06-17 07:13:51,019] (heat-config) [INFO] Completed /usr/libexec/heat-config/hooks/ansible",
            "[2019-06-17 07:13:51,020] (heat-config) [DEBUG] Running heat-config-notify /var/lib/heat-config/deployed/32408a85-2eea-406f-aa1c-42ba37e422ee.json < /var/lib/heat-config/deployed/324
08a85-2eea-406f-aa1c-42ba37e422ee.notify.json",
            "[2019-06-17 07:13:51,694] (heat-config) [INFO] ",
            "[2019-06-17 07:13:51,695] (heat-config) [DEBUG] "
        ]
    },
    {
        "status_code": "2"
    }
]

Comment 29 errata-xmlrpc 2019-07-10 13:01:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2019:1742