Bug 1631793
| Summary: | running a deployment with '--run-validations' fails and the error message is hardly readable. | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Alexander Chuzhoy <sasha> |
| Component: | openstack-tripleo-validations | Assignee: | Florian Fuchs <flfuchs> |
| Status: | CLOSED WONTFIX | QA Contact: | nlevinki <nlevinki> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 14.0 (Rocky) | CC: | beth.white, bfournie, emacchi, gchamoul, jbuchta, jjoyce, jschluet, mburns, slinaber, tvignaud |
| Target Milestone: | --- | Keywords: | Triaged, ZStream |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-01-24 17:54:16 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I see a similar failure with OSP-13. I have a config that successfully deploys, I delete the stack and attempt a redeployment using the "--run-validations" flag. It fails with the output below, I can't tell which validation has failed:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 134, in run
ret_val = super(OpenStackShell, self).run(argv)
File "/usr/lib/python2.7/site-packages/cliff/app.py", line 279, in run
result = self.run_subcommand(remainder)
File "/usr/lib/python2.7/site-packages/osc_lib/shell.py", line 169, in run_subcommand
ret_value = super(OpenStackShell, self).run_subcommand(argv)
File "/usr/lib/python2.7/site-packages/cliff/app.py", line 400, in run_subcommand
result = cmd.run(parsed_args)
File "/usr/lib/python2.7/site-packages/tripleoclient/command.py", line 25, in run
super(Command, self).run(parsed_args)
File "/usr/lib/python2.7/site-packages/osc_lib/command/command.py", line 41, in run
return super(Command, self).run(parsed_args)
File "/usr/lib/python2.7/site-packages/cliff/command.py", line 184, in run
return_code = self.take_action(parsed_args) or 0
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 966, in take_action
self._deploy_tripleo_heat_templates_tmpdir(stack, parsed_args)
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 423, in _deploy_tripleo_heat_templates_tmpdir
new_tht_root, tht_root)
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 516, in _deploy_tripleo_heat_templates
parsed_args.plan_environment_file)
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 531, in _try_overcloud_deploy_with_compat_yaml
plan_env_file)
File "/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py", line 299, in _heat_deploy
skip_deploy_identifier=skip_deploy_identifier)
File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 63, in deploy_and_wait
deploy(clients, **workflow_input)
File "/usr/lib/python2.7/site-packages/tripleoclient/workflows/deployment.py", line 46, in deploy
assert payload['status'] == "SUCCESS", pprint.pformat(payload)
AssertionError: {u'execution': {u'created_at': u'2018-09-24 13:51:29',
u'id': u'6279781b-59c1-4ec1-b9d3-0931a45d725a',
u'input': {u'group_names': [u'pre-deployment'],
u'plan': u'overcloud',
u'queue_name': u'tripleo'},
u'name': u'tripleo.validations.v1.run_groups',
u'params': {u'index': 0,
u'namespace': u'',
u'root_execution_id': u'ff439fe6-f86e-40fc-a3cd-736566fea975',
u'task_execution_id': u'f10253b6-11be-4087-bfa9-b1d2cda67de4'},
u'spec': {u'input': [{u'group_names': []},
{u'plan': u'overcloud'},
{u'queue_name': u'tripleo'}],
u'name': u'run_groups',
u'tags': [u'tripleo-common-managed'],
u'tasks': {u'find_validations': {u'action': u'tripleo.validations.list_validations groups=<% $.group_names %>',
u'name': u'find_validations',
u'on-success': u'notify_running',
u'publish': {u'validations': u'<% task().result %>'},
u'type': u'direct',
u'version': u'2.0'},
u'notify_running': {u'action': u'zaqar.queue_post',
u'input': {u'messages': {u'body': {u'payload': {u'execution': u'<% execution() %>',
u'group_names': u'<% $.group_names %>',
u'plan': u'<% $.plan %>',
u'status': u'RUNNING',
u'validation_names': u'<% $.validations.id %>'},
u'type': u'tripleo.validations.v1.run_validations'}},
u'queue_name': u'<% $.queue_name %>'},
u'name': u'notify_running',
u'on-complete': u'run_validation_group',
u'retry': u'count=5 delay=1',
u'type': u'direct',
u'version': u'2.0'},
u'run_validation_group': {u'name': u'run_validation_group',
u'on-error': u'set_status_failed',
u'on-success': u'send_message',
u'publish': {u'status': u'SUCCESS'},
u'type': u'direct',
u'version': u'2.0',
u'with-items': u'validation in <% $.validations.id %>',
u'workflow': u'tripleo.validations.v1.run_validation validation_name=<% $.validation %> plan=<% $.plan %> queue_name=<% $.queue_name %>'},
u'send_message': {u'action': u'zaqar.queue_post',
u'input': {u'messages': {u'body': {u'payload': {u'execution': u'<% execution() %>',
u'group_names': u'<% $.group_names %>',
u'plan': u'<% $.plan %>',
u'status': u"<% $.get('status', 'SUCCESS') %>",
u'validation_names': u'<% $.validations.id %>'},
u'type': u'tripleo.validations.v1.run_groups'}},
u'queue_name': u'<% $.queue_name %>'},
u'name': u'send_message',
u'on-success': [{u'fail': u'<% $.get(\'status\') = "FAILED" %>'}],
u'retry': u'count=5 delay=1',
u'type': u'direct',
u'version': u'2.0'},
u'set_status_failed': {u'name': u'set_status_failed',
u'on-complete': u'send_message',
u'publish': {u'status': u'FAILED'},
u'type': u'direct',
u'version': u'2.0'}},
u'version': u'2.0'},
u'updated_at': u'2018-09-24 13:51:29'},
u'group_names': [u'pre-deployment'],
u'plan': u'overcloud',
u'status': u'RUNNING',
u'validation_names': [u'512e',
u'ceilometerdb-size',
u'ceph-ansible-installed',
u'deployment-images',
u'dhcp-provisioning',
u'dns',
u'network-environment',
u'switch-vlans',
u'undercloud-debug',
u'undercloud-process-count']}
END return value: 1
(undercloud) [stack@host01 ~]$
This happens every time in same way, I can get an sosreport if needed.
After talking with Brad about this, moving to UI as it seems specific to validations. |
running a deployment with '--run-validations' fails and the error message is hardly readable. Environment: openstack-tripleo-heat-templates-9.0.0-0.20180831204457.17bb71e.0rc1.el7ost.noarch instack-undercloud-9.3.1-0.20180831000258.e464799.el7ost.noarch openstack-tripleo-common-9.3.1-0.20180831204016.bb0582a.el7ost.noarch openstack-tripleo-common-containers-9.3.1-0.20180831204016.bb0582a.el7ost.noarch Ran the following deployment command: openstack overcloud deploy \ --timeout 100 \ --run-validations \ --templates /usr/share/openstack-tripleo-heat-templates \ --stack overcloud \ --libvirt-type kvm \ --ntp-server clock.redhat.com \ -e /home/stack/virt/internal.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/virt/network/network-environment.yaml \ -e /home/stack/virt/enable-tls.yaml \ -e /home/stack/virt/inject-trust-anchor.yaml \ -e /home/stack/virt/public_vip.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \ -e /home/stack/virt/hostnames.yml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \ -e /home/stack/virt/debug.yaml \ -e /home/stack/virt/nodes_data.yaml \ --environment-file /usr/share/openstack-tripleo-heat-templates/environments/services/ironic.yaml \ -e /home/stack/virt/ironic_osp14.yaml \ -e /home/stack/virt/docker-images.yam Result: Initializing overcloud plan deployment None None None None None None None None None None None None None None None {u'status': u'FAILED', u'validations': [{u'id': u'512e', u'name': u'Advanced Format 512e Support', u'description': u'Detect whether the undercloud disks use Advanced Format. If they do, the overcloud images may fail to upload to Glance.\n', u'groups': [u'prep', u'pre-deployment'], u'metadata': {}}, {u'id': u'ceilometerdb-size', u'name': u'Events and Metrics Database Size Check', u'description': u"The undercloud's metric and events databases can grow to a substantial size if metering_time_to_live and event_time_to_live is set to a negative value (infinite limit).\nThis validation checks each setting and fails if variables are set to a negative value or if they have no custom setting (their value is -1 by default).\n", u'groups': [u'pre-deployment'], u'metadata': {}}, {u'id': u'ceph-ansible-installed', u'name': u'Check if ceph-ansible is installed on the undercloud', u'description': u"Prints a message if ceph-ansible isn't installed\n", u'groups': [u'pre-deployment'], u'metadata': {}}, {u'id': u'collect-flavors-and-verify-profiles', u'name': u'Collect and verify role flavors', u'description': u'This validation checks the flavors assigned to roles exist and have the correct capabilities set.\n', u'groups': [u'pre-deployment', u'pre-upgrade'], u'metadata': {}}, {u'id': u'default-node-count', u'name': u'Verify hypervisor statistics', u'description': u'This validation checks that the nodes and hypervisor statistics add up.\n', u'groups': [u'pre-deployment'], u'metadata': {}}, {u'id': u'deployment-images', u'name': u'Verify existence of deployment images', u'description': u'This validation checks that images bm-deploy-kernel and bm-deploy-ramdisk exist before deploying the overcloud, and that only one exists by that name.\n', u'groups': [u'pre-deployment', u'pre-upgrade'], u'metadata': {}}, {u'id': u'dhcp-provisioning', u'name': u'DHCP on the Provisioning Network', u'description': u"An unexpected DHCP server on the provisioning network can cause problems with deploying the Ironic nodes.\nThis validation checks for DHCP responses on the undercloud's provisioning interface (eth1 by default) and fails if there are any.\n", u'groups': [u'pre-deployment'], u'metadata': {}}, {u'id': u'dns', u'name': u'Verify DNS', u'description': u'Verify that the DNS resolution works\n', u'groups': [u'pre-deployment'], u'metadata': {}}, {u'id': u'ironic-boot-configuration', u'name': u'Check Ironic boot configuration', u'description': u'Check if baremetal boot configuration is correct.\n', u'groups': [u'pre-deployment', u'pre-upgrade'], u'metadata': {}}, {u'id': u'network-environment', u'name': u'Validate the Heat environment file for network configuration', u'description': u"This validates the network environment and nic-config files that specify the overcloud network configuration and are stored in the current plan's Swift container.\nThe deployers are expected to write these files themselves as described in the Network Isolation guide:\nhttp://tripleo.org/advanced_deployment/network_isolation.html\n", u'groups': [u'pre-deployment'], u'metadata': {}}, {u'id': u'switch-vlans', u'name': u'Compare switch port VLANs to VLANs in nic config', u'description': u"LLDP data received during introspection contains the configured VLANs for each switch port attached to the nodes interfaces. Compare the VLAN IDs set on the switch port to those configured in nic config files. Since the mapping of roles to nodes isn't known prior to deployment, this check can only check VLANs across all switch ports, not on a particular switch port.\n", u'groups': [u'pre-deployment'], u'metadata': {}}, {u'id': u'undercloud-debug', u'name': u'Undercloud Services Debug Check', u'description': u"The undercloud's openstack services should _not_ have debug enabled. This will check if debug is enabled on undercloud services. If debug is enabled, the root filesystem can fill up quickly, and is not a good thing.\n", u'groups': [u'pre-deployment'], u'metadata': {}}, {u'id': u'undercloud-heat-purge-deleted', u'name': u'Verify heat-manage purge_deleted is enabled in crontab', u'description': u'Without a purge_deleted crontab enabled, the heat database can grow very large. This validation checks that the purge_deleted crontab has been set up.\n', u'groups': [u'pre-upgrade', u'pre-deployment'], u'metadata': {}}, {u'id': u'undercloud-process-count', u'name': u'Check the number of OpenStack processes on undercloud', u'description': u'The default settings for OpenStack is to run one process (heat-engine, keystone, etc.) per CPU core. On a machine with a lot of cores this is both unnecessary and can consume a significant amount of RAM, leading to crashes due to OOMKiller.\n', u'groups': [u'pre-deployment'], u'metadata': {}}], u'result': u'Failure caused by error in tasks: send_message\n\n send_message [task_ex_id=0928cde1-a08f-4b9c-8969-9cbc2cdbd5a9] -> Workflow failed due to message status\n [wf_ex_id=4226d47a-a9c8-41e6-be57-87ecc22602e0, idx=0]: Workflow failed due to message status\n'} Unexpected status FAILED for tripleo.deployment.v1.deploy_plan Note that the same command proceeded to deployment when no --run-validations switch was used. With that said, the output from the validations is hardly readable.