Bug 1573120 - [UPGRADES] ERROR error running the validation groups ['post-upgrade']
Summary: [UPGRADES] ERROR error running the validation groups ['post-upgrade']
Keywords:
Status: CLOSED DUPLICATE of bug 1651136
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-validations
Version: 13.0 (Queens)
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: ---
: ---
Assignee: Gaël Chamoulaud
QA Contact: nlevinki
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-04-30 08:39 UTC by Yurii Prokulevych
Modified: 2019-08-26 11:23 UTC (History)
22 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-08-26 11:23:33 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
OpenStack gerrit 619112 0 'None' MERGED Get the stack name if possible before running the validations group 2020-11-19 09:29:04 UTC

Description Yurii Prokulevych 2018-04-30 08:39:34 UTC
Description of problem:
-----------------------
After undercloud upgrade 'post-upgrade' validation group is called and failed
...
2018-04-30 04:15:32,883 ERROR: ERROR error running the validation groups ['post-upgrade']   {"stderr": "[DEPRECATION WARNING]: DEFAULT_SUDO_FLAGS option, In favor of become which is a\n generic framework . This feature will be removed in version 2.8. Deprecation \nwarnings can be disabled by setting deprecation_warnings=False in ansible.cfg.\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with script\nplugin: Inventory script (/bin/tripleo-ansible-inventory) had an execution\nerror: /usr/lib/python2.7/site-\npackages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning:\nCertificate for 192.168.24.2 has no `subjectAltName`, falling back to check for\na `commonName` for now. This feature is being removed by major browsers and\ndeprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for\ndetails.)   SubjectAltNameWarning /usr/lib/python2.7/site-\npackages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning:\nCertificate for 192.168.24.2 has no `subjectAltName`, falling back to check for\na `commonName` for now. This feature is being removed by major browsers and\ndeprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for\ndetails.)   SubjectAltNameWarning Error creating inventory: <html><body><h1>503\nService Unavailable</h1> No server is available to handle this request.\n</body></html>\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with yaml plugin:\nSyntax Error while loading YAML.   The error appears to have been in '/bin\n/tripleo-ansible-inventory': line 65, column 20, but may be elsewhere in the\nfile depending on the exact syntax problem.  The offending line appears to be:\ndef _parse_config():                    ^ here  exception type: <class\n'yaml.scanner.ScannerError'> exception: mapping values are not allowed in this\ncontext   in \"<unicode string>\", line 65, column 20\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with ini plugin:\n/bin/tripleo-ansible-inventory:22: Expected key=value host variable assignment,\ngot: __future__\n [WARNING]: Unable to parse /bin/tripleo-ansible-inventory as an inventory\nsource\n [WARNING]: No inventory was parsed, only implicit localhost is available\n [WARNING]: provided hosts list is empty, only localhost is available. Note\nthat the implicit localhost does not match 'all'\n [WARNING]: Could not match supplied host pattern, ignoring: undercloud\n", "stdout": "Warning! The validation did not run on any host.\n"} {"stderr": "[DEPRECATION WARNING]: DEFAULT_SUDO_FLAGS option, In favor of become which is a\n generic framework . This feature will be removed in version 2.8. Deprecation \nwarnings can be disabled by setting deprecation_warnings=False in ansible.cfg.\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with script\nplugin: Inventory script (/bin/tripleo-ansible-inventory) had an execution\nerror: /usr/lib/python2.7/site-\npackages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning:\nCertificate for 192.168.24.2 has no `subjectAltName`, falling back to check for\na `commonName` for now. This feature is being removed by major browsers and\ndeprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for\ndetails.)   SubjectAltNameWarning /usr/lib/python2.7/site-\npackages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning:\nCertificate for 192.168.24.2 has no `subjectAltName`, falling back to check for\na `commonName` for now. This feature is being removed by major browsers and\ndeprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for\ndetails.)   SubjectAltNameWarning Error creating inventory: <html><body><h1>503\nService Unavailable</h1> No server is available to handle this request.\n</body></html>\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with yaml plugin:\nSyntax Error while loading YAML.   The error appears to have been in '/bin\n/tripleo-ansible-inventory': line 65, column 20, but may be elsewhere in the\nfile depending on the exact syntax problem.  The offending line appears to be:\ndef _parse_config():                    ^ here  exception type: <class\n'yaml.scanner.ScannerError'> exception: mapping values are not allowed in this\ncontext   in \"<unicode string>\", line 65, column 20\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with ini plugin:\n/bin/tripleo-ansible-inventory:22: Expected key=value host variable assignment,\ngot: __future__\n [WARNING]: Unable to parse /bin/tripleo-ansible-inventory as an inventory\nsource\n [WARNING]: No inventory was parsed, only implicit localhost is available\n [WARNING]: provided hosts list is empty, only localhost is available. Note\nthat the implicit localhost does not match 'all'\n [WARNING]: Could not match supplied host pattern, ignoring: undercloud\n", "stdout": "Warning! The validation did not run on any host.\n"} {"stderr": "[DEPRECATION WARNING]: DEFAULT_SUDO_FLAGS option, In favor of become which is a\n generic framework . This feature will be removed in version 2.8. Deprecation \nwarnings can be disabled by setting deprecation_warnings=False in ansible.cfg.\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with script\nplugin: Inventory script (/bin/tripleo-ansible-inventory) had an execution\nerror: /usr/lib/python2.7/site-\npackages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning:\nCertificate for 192.168.24.2 has no `subjectAltName`, falling back to check for\na `commonName` for now. This feature is being removed by major browsers and\ndeprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for\ndetails.)   SubjectAltNameWarning /usr/lib/python2.7/site-\npackages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning:\nCertificate for 192.168.24.2 has no `subjectAltName`, falling back to check for\na `commonName` for now. This feature is being removed by major browsers and\ndeprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for\ndetails.)   SubjectAltNameWarning Error creating inventory: <html><body><h1>503\nService Unavailable</h1> No server is available to handle this request.\n</body></html>\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with yaml plugin:\nSyntax Error while loading YAML.   The error appears to have been in '/bin\n/tripleo-ansible-inventory': line 65, column 20, but may be elsewhere in the\nfile depending on the exact syntax problem.  The offending line appears to be:\ndef _parse_config():                    ^ here  exception type: <class\n'yaml.scanner.ScannerError'> exception: mapping values are not allowed in this\ncontext   in \"<unicode string>\", line 65, column 20\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with ini plugin:\n/bin/tripleo-ansible-inventory:22: Expected key=value host variable assignment,\ngot: __future__\n [WARNING]: Unable to parse /bin/tripleo-ansible-inventory as an inventory\nsource\n [WARNING]: No inventory was parsed, only implicit localhost is available\n [WARNING]: provided hosts list is empty, only localhost is available. Note\nthat the implicit localhost does not match 'all'\n [WARNING]: Could not match supplied host pattern, ignoring: undercloud\n", "stdout": "Warning! The validation did not run on any host.\n"} Mistral execution ID: bb3f1b74-30ee-4581-9f0a-1e798d26e1e3
2018-04-30 04:15:32,886 INFO: 
#############################################################################
Undercloud upgrade complete.

mistral execution-get bb3f1b74-30ee-4581-9f0a-1e798d26e1e3
+--------------------+--------------------------------------+
| Field              | Value                                |
+--------------------+--------------------------------------+
| ID                 | bb3f1b74-30ee-4581-9f0a-1e798d26e1e3 |
| Workflow ID        | f7d9a3ad-9357-447d-a14c-afc55667537f |
| Workflow name      | tripleo.validations.v1.run_groups    |
| Workflow namespace |                                      |
| Description        |                                      |
| Task Execution ID  | <none>                               |
| State              | ERROR                                |
| State info         | None                                 |
| Created at         | 2018-04-30 08:14:20                  |
| Updated at         | 2018-04-30 08:15:27                  |
+--------------------+--------------------------------------+

mistral execution-get-output bb3f1b74-30ee-4581-9f0a-1e798d26e1e3
{
    "status": "FAILED", 
    "validations": [
        {
            "id": "openstack-endpoints", 
            "name": "Check connectivity to various OpenStack services", 
            "description": "This validation gets the PublicVip address from the deployment and tries to access Horizon and get a Keystone token.\n", 
            "groups": [
                "post-deployment", 
                "pre-upgrade", 
                "post-upgrade"
            ], 
            "metadata": {}
        }, 
        {
            "id": "stack-health", 
            "name": "Stack Health Check", 
            "description": "Check if all stack resources are in a *_COMPLETE state before starting an upgrade.\n", 
            "groups": [
                "pre-upgrade", 
                "post-upgrade"
            ], 
            "metadata": {}
        }, 
        {
            "id": "undercloud-service-status", 
            "name": "Verify undercloud services state before running update or upgrade", 
            "description": "Check undercloud status before running a stack update - especially minor update and major upgrade.\n", 
            "groups": [
                "post-upgrade", 
                "pre-upgrade"
            ], 
            "metadata": {}
        }
    ], 
    "result": null
}

mistral task-list -f yaml bb3f1b74-30ee-4581-9f0a-1e798d26e1e3
- Created at: '2018-04-30 08:14:20'
  Execution ID: bb3f1b74-30ee-4581-9f0a-1e798d26e1e3
  ID: 61647470-e979-4beb-951c-d485b870b7fc
  Name: find_validations
  State: SUCCESS
  State info: null
  Updated at: '2018-04-30 08:14:20'
  Workflow name: tripleo.validations.v1.run_groups
  Workflow namespace: ''
- Created at: '2018-04-30 08:14:20'
  Execution ID: bb3f1b74-30ee-4581-9f0a-1e798d26e1e3
  ID: 9b173b2f-e93b-41f3-bec5-e4a067dde8a1
  Name: notify_running
  State: SUCCESS
  State info: null
  Updated at: '2018-04-30 08:14:21'
  Workflow name: tripleo.validations.v1.run_groups
  Workflow namespace: ''
- Created at: '2018-04-30 08:14:21'
  Execution ID: bb3f1b74-30ee-4581-9f0a-1e798d26e1e3
  ID: 5c9b0668-6de1-4ab9-88bf-aca44c2c1393
  Name: run_validation_group
  State: ERROR
  State info: One or more actions had f...
  Updated at: '2018-04-30 08:15:26'
  Workflow name: tripleo.validations.v1.run_groups
  Workflow namespace: ''
- Created at: '2018-04-30 08:15:26'
  Execution ID: bb3f1b74-30ee-4581-9f0a-1e798d26e1e3
  ID: 621422a0-0ea5-4b9d-815c-b11f50ac629b
  Name: send_message
  State: SUCCESS
  State info: null
  Updated at: '2018-04-30 08:15:27'
  Workflow name: tripleo.validations.v1.run_groups
  Workflow namespace: ''
- Created at: '2018-04-30 08:15:26'
  Execution ID: bb3f1b74-30ee-4581-9f0a-1e798d26e1e3
  ID: 86212cb3-9ab2-4618-9e37-a3b9ed12e9fb
  Name: set_status_failed
  State: SUCCESS
  State info: null
  Updated at: '2018-04-30 08:15:26'
  Workflow name: tripleo.validations.v1.run_groups
  Workflow namespace: ''

mistral task-get-result 5c9b0668-6de1-4ab9-88bf-aca44c2c1393
[
    {
        "status": "FAILED", 
        "result": null, 
        "stderr": "[DEPRECATION WARNING]: DEFAULT_SUDO_FLAGS option, In favor of become which is a\n generic framework . This feature will be removed in version 2.8. Deprecation \nwarnings can be disabled by setting deprecation_warnings=False in ansible.cfg.\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with script\nplugin: Inventory script (/bin/tripleo-ansible-inventory) had an execution\nerror: /usr/lib/python2.7/site-\npackages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning:\nCertificate for 192.168.24.2 has no `subjectAltName`, falling back to check for\na `commonName` for now. This feature is being removed by major browsers and\ndeprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for\ndetails.)   SubjectAltNameWarning /usr/lib/python2.7/site-\npackages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning:\nCertificate for 192.168.24.2 has no `subjectAltName`, falling back to check for\na `commonName` for now. This feature is being removed by major browsers and\ndeprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for\ndetails.)   SubjectAltNameWarning Error creating inventory: <html><body><h1>503\nService Unavailable</h1> No server is available to handle this request.\n</body></html>\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with yaml plugin:\nSyntax Error while loading YAML.   The error appears to have been in '/bin\n/tripleo-ansible-inventory': line 65, column 20, but may be elsewhere in the\nfile depending on the exact syntax problem.  The offending line appears to be:\ndef _parse_config():                    ^ here  exception type: <class\n'yaml.scanner.ScannerError'> exception: mapping values are not allowed in this\ncontext   in \"<unicode string>\", line 65, column 20\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with ini plugin:\n/bin/tripleo-ansible-inventory:22: Expected key=value host variable assignment,\ngot: __future__\n [WARNING]: Unable to parse /bin/tripleo-ansible-inventory as an inventory\nsource\n [WARNING]: No inventory was parsed, only implicit localhost is available\n [WARNING]: provided hosts list is empty, only localhost is available. Note\nthat the implicit localhost does not match 'all'\n [WARNING]: Could not match supplied host pattern, ignoring: undercloud\n", 
        "stdout": "Warning! The validation did not run on any host.\n"
    }, 
    {
        "status": "FAILED", 
        "result": null, 
        "stderr": "[DEPRECATION WARNING]: DEFAULT_SUDO_FLAGS option, In favor of become which is a\n generic framework . This feature will be removed in version 2.8. Deprecation \nwarnings can be disabled by setting deprecation_warnings=False in ansible.cfg.\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with script\nplugin: Inventory script (/bin/tripleo-ansible-inventory) had an execution\nerror: /usr/lib/python2.7/site-\npackages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning:\nCertificate for 192.168.24.2 has no `subjectAltName`, falling back to check for\na `commonName` for now. This feature is being removed by major browsers and\ndeprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for\ndetails.)   SubjectAltNameWarning /usr/lib/python2.7/site-\npackages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning:\nCertificate for 192.168.24.2 has no `subjectAltName`, falling back to check for\na `commonName` for now. This feature is being removed by major browsers and\ndeprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for\ndetails.)   SubjectAltNameWarning Error creating inventory: <html><body><h1>503\nService Unavailable</h1> No server is available to handle this request.\n</body></html>\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with yaml plugin:\nSyntax Error while loading YAML.   The error appears to have been in '/bin\n/tripleo-ansible-inventory': line 65, column 20, but may be elsewhere in the\nfile depending on the exact syntax problem.  The offending line appears to be:\ndef _parse_config():                    ^ here  exception type: <class\n'yaml.scanner.ScannerError'> exception: mapping values are not allowed in this\ncontext   in \"<unicode string>\", line 65, column 20\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with ini plugin:\n/bin/tripleo-ansible-inventory:22: Expected key=value host variable assignment,\ngot: __future__\n [WARNING]: Unable to parse /bin/tripleo-ansible-inventory as an inventory\nsource\n [WARNING]: No inventory was parsed, only implicit localhost is available\n [WARNING]: provided hosts list is empty, only localhost is available. Note\nthat the implicit localhost does not match 'all'\n [WARNING]: Could not match supplied host pattern, ignoring: undercloud\n", 
        "stdout": "Warning! The validation did not run on any host.\n"
    }, 

    {
        "status": "FAILED", 
        "result": null, 
        "stderr": "[DEPRECATION WARNING]: DEFAULT_SUDO_FLAGS option, In favor of become which is a\n generic framework . This feature will be removed in version 2.8. Deprecation \nwarnings can be disabled by setting deprecation_warnings=False in ansible.cfg.\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with script\nplugin: Inventory script (/bin/tripleo-ansible-inventory) had an execution\nerror: /usr/lib/python2.7/site-\npackages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning:\nCertificate for 192.168.24.2 has no `subjectAltName`, falling back to check for\na `commonName` for now. This feature is being removed by major browsers and\ndeprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for\ndetails.)   SubjectAltNameWarning /usr/lib/python2.7/site-\npackages/requests/packages/urllib3/connection.py:344: SubjectAltNameWarning:\nCertificate for 192.168.24.2 has no `subjectAltName`, falling back to check for\na `commonName` for now. This feature is being removed by major browsers and\ndeprecated by RFC 2818. (See https://github.com/shazow/urllib3/issues/497 for\ndetails.)   SubjectAltNameWarning Error creating inventory: <html><body><h1>503\nService Unavailable</h1> No server is available to handle this request.\n</body></html>\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with yaml plugin:\nSyntax Error while loading YAML.   The error appears to have been in '/bin\n/tripleo-ansible-inventory': line 65, column 20, but may be elsewhere in the\nfile depending on the exact syntax problem.  The offending line appears to be:\ndef _parse_config():                    ^ here  exception type: <class\n'yaml.scanner.ScannerError'> exception: mapping values are not allowed in this\ncontext   in \"<unicode string>\", line 65, column 20\n [WARNING]:  * Failed to parse /bin/tripleo-ansible-inventory with ini plugin:\n/bin/tripleo-ansible-inventory:22: Expected key=value host variable assignment,\ngot: __future__\n [WARNING]: Unable to parse /bin/tripleo-ansible-inventory as an inventory\nsource\n [WARNING]: No inventory was parsed, only implicit localhost is available\n [WARNING]: provided hosts list is empty, only localhost is available. Note\nthat the implicit localhost does not match 'all'\n [WARNING]: Could not match supplied host pattern, ignoring: undercloud\n", 
        "stdout": "Warning! The validation did not run on any host.\n"
    }
]

Re-running some of playbooks manually succeeds:
ansible-playbook -i /bin/tripleo-ansible-inventory \
  /usr/share/openstack-tripleo-validations/validations/stack-health.yaml
...
ok: [localhost] => (item={'resource_name': u'MessagingAllNodesDeployment', 'links': [{u'href': u'https://192.168.24.2:13004/v1/5f627eed6a914ea28661457fb9f41165/stacks/overcloud/376b4b8a-f487-43d8-936d-93aa887faf58/resources/MessagingAllNodesDeployment', u'rel': u'self'}, {u'href': u'https://192.168.24.2:13004/v1/5f627eed6a914ea28661457fb9f41165/stacks/overcloud/376b4b8a-f487-43d8-936d-93aa887faf58', u'rel': u'stack'}, {u'href': u'https://192.168.24.2:13004/v1/5f627eed6a914ea28661457fb9f41165/stacks/overcloud-MessagingAllNodesDeployment-qz45qnlqfof5/6cd8b5cc-940f-4d6a-b0f1-86983baf5ab7', u'rel': u'nested'}], 'logical_resource_id': u'MessagingAllNodesDeployment', 'creation_time': u'2018-04-30T06:42:30Z', 'resource_status': u'CREATE_COMPLETE', 'updated_time': u'2018-04-30T06:42:30Z', 'required_by': [u'AllNodesDeploySteps', u'MessagingAllNodesValidationDeployment', u'UpdateWorkflow'], 'resource_status_reason': u'state changed', 'physical_resource_id': u'6cd8b5cc-940f-4d6a-b0f1-86983baf5ab7', 'resource_type': u'OS::TripleO::AllNodesDeployment'}) => {
    "changed": false, 
    "item": {
        "creation_time": "2018-04-30T06:42:30Z", 
        "links": [
            {
                "href": "https://192.168.24.2:13004/v1/5f627eed6a914ea28661457fb9f41165/stacks/overcloud/376b4b8a-f487-43d8-936d-93aa887faf58/resources/MessagingAllNodesDeployment", 
                "rel": "self"
            }, 
            {
                "href": "https://192.168.24.2:13004/v1/5f627eed6a914ea28661457fb9f41165/stacks/overcloud/376b4b8a-f487-43d8-936d-93aa887faf58", 
                "rel": "stack"
            }, 
            {
                "href": "https://192.168.24.2:13004/v1/5f627eed6a914ea28661457fb9f41165/stacks/overcloud-MessagingAllNodesDeployment-qz45qnlqfof5/6cd8b5cc-940f-4d6a-b0f1-86983baf5ab7", 
                "rel": "nested"
            }
        ], 
        "logical_resource_id": "MessagingAllNodesDeployment", 
        "physical_resource_id": "6cd8b5cc-940f-4d6a-b0f1-86983baf5ab7", 
        "required_by": [
            "AllNodesDeploySteps", 
            "MessagingAllNodesValidationDeployment", 
            "UpdateWorkflow"
        ], 
        "resource_name": "MessagingAllNodesDeployment", 
        "resource_status": "CREATE_COMPLETE", 
        "resource_status_reason": "state changed", 
        "resource_type": "OS::TripleO::AllNodesDeployment", 
        "updated_time": "2018-04-30T06:42:30Z"
    }, 
    "msg": "All assertions passed"
}

PLAY RECAP ********************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0   


Version-Release number of selected component (if applicable):
-------------------------------------------------------------
openstack-tripleo-validations-8.4.1-1.el7ost.noarch
openstack-tripleo-heat-templates-8.0.2-4.el7ost.noarch
python-tripleoclient-9.2.1-3.el7ost.noarch

Steps to Reproduce:
-------------------
1. Upgrade UC to RHOS-13(2018-04-26.3)

Expected results:
-----------------
post-ugprade validation succeeds

Comment 8 Jon Schlueter 2018-05-22 13:24:09 UTC
Per Assessment of bug, not a blocker, it a failure caused by validations changing from one version to another.  Needs addressed but not blocking the release.

Comment 9 Jose Luis Franco 2018-05-22 15:33:40 UTC
I've been debugging the issue and the cause seems to be that some of the services are still not up before running the validations, mainly heat_api. As far as I know, heat is being used to obtain the inventory in https://github.com/openstack/tripleo-validations/blob/master/scripts/tripleo-ansible-inventory#L130 . As the service is not yet available, it returns 503 Service Unavailabe from the exception message: https://github.com/openstack/tripleo-validations/blob/master/scripts/tripleo-ansible-inventory#L153

I tried adding a 5 minute delay just before running the validations to wait for the services to be all up, and the error stopped appearing:

2018-05-21 19:06:12,739 INFO: Configuring Mistral workbooks
2018-05-21 19:06:56,181 INFO: Mistral workbooks configured successfully
2018-05-21 19:06:56,648 INFO: Not creating default plan "overcloud" because it already exists.
2018-05-21 19:06:56,648 INFO: Configuring an hourly cron trigger for tripleo-ui logging
2018-05-21 19:07:00,857 INFO: Added _member_ role to admin user
2018-05-21 19:07:02,065 INFO: Migrating stack "59a8e1d6-b9e1-48d9-8127-9f1cd1716a99" to convergence engine
2018-05-21 19:08:07,885 INFO: Finished migrating stack "59a8e1d6-b9e1-48d9-8127-9f1cd1716a99"
2018-05-21 19:08:07,928 INFO: Waiting for 5 minutes for services to be up
2018-05-21 19:13:08,232 INFO: Starting and waiting for validation groups ['post-upgrade'] 
2018-05-21 19:13:50,073 INFO: 
#############################################################################
Undercloud upgrade complete.

The file containing this installation's passwords is at
/home/stack/undercloud-passwords.conf.

There is also a stackrc file at /home/stack/stackrc.

These files are needed to interact with the OpenStack services, and should be
secured.

#############################################################################


Broadcast message from systemd-journald@undercloud (Mon 2018-05-21 19:13:58 UTC):

haproxy[10336]: proxy nova_metadata has no server available!


Message from syslogd@undercloud at May 21 19:13:58 ...
 haproxy[10336]:proxy nova_metadata has no server available!


I may be wrong, but some busy waiting mechanism to wait for some time for the services before running the validations could solve the issue.

Comment 26 Jose Luis Franco 2019-02-18 15:08:30 UTC
So, checking the patch linked in the BZ (https://review.openstack.org/#/c/619112/11), I believe it would solve the problem you're seeing. As it will try to list the existing stack names and retrieve the swift container with the name of the stack. I can see that the patch wasn't merged downstream yet, only upstream, that's why you got the very same error.

Anyway, if after executing the mistral workflow passing the right plan you get a SUCCEEDED task execution then you can move on to the overcloud upgrade or minor update step you were planning to perform. That will ensure that the environment is in a correct state.

Comment 34 AMOL LONARE 2019-06-24 05:30:46 UTC
Do we have an ETA for this BZ?

Comment 40 Jose Luis Franco 2019-08-26 11:23:33 UTC
This bugzilla is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1651136, which has being address via gerrit patches https://code.engineering.redhat.com/gerrit/169593 and https://code.engineering.redhat.com/gerrit/173435. Merged in instack-undercloud-8.4.7-7.el7ost. 

Closing this bug as duplicate.

*** This bug has been marked as a duplicate of bug 1651136 ***


Note You need to log in before you can comment on or make changes to this bug.