Bug 1892323
Summary: | Validations in FFU 13->16.1: Not possible to run validations before 'openstack undercloud upgrade' | |||
---|---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Jose Luis Franco <jfrancoa> | |
Component: | python-tripleoclient | Assignee: | Cédric Jeanneret <cjeanner> | |
Status: | CLOSED ERRATA | QA Contact: | David Rosenfeld <drosenfe> | |
Severity: | high | Docs Contact: | ||
Priority: | high | |||
Version: | 16.1 (Train) | CC: | cjeanner, gchamoul, hbrock, jschluet, jslagle, mburns, mschuppe, slinaber, spower | |
Target Milestone: | z4 | Keywords: | Triaged | |
Target Release: | 16.1 (Train on RHEL 8.2) | |||
Hardware: | Unspecified | |||
OS: | Unspecified | |||
Whiteboard: | ||||
Fixed In Version: | python-tripleoclient-12.3.2-1.20200914164931.el8ost | Doc Type: | If docs needed, set a value | |
Doc Text: | Story Points: | --- | ||
Clone Of: | ||||
: | 1898633 (view as bug list) | Environment: | ||
Last Closed: | 2021-03-17 15:33:21 UTC | Type: | Bug | |
Regression: | --- | Mount Type: | --- | |
Documentation: | --- | CRM: | ||
Verified Versions: | Category: | --- | ||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
Cloudforms Team: | --- | Target Upstream Version: | ||
Embargoed: | ||||
Bug Depends On: | 1912879 | |||
Bug Blocks: | 1898482, 1898633 |
Description
Jose Luis Franco
2020-10-28 13:44:40 UTC
Just as a side note, when triggering the "openstack undercloud upgrade" command the inflight validations for the Undercloud do get triggered: (undercloud) [stack@undercloud-0 ~]$ openstack undercloud upgrade -y 2>&1 | tee undercloud_upgrade.log ^[[B^[[B^[[B^[[B^[[BUpdating necessary packages: python3-tripleoclient openstack-tripleo-common openstack-tripleo-heat-templates openstack-tripleo-validations tripleo-ansib$ e Updating Subscription Management repositories. Unable to read consumer identity /usr/lib/python3.6/site-packages/dateutil/parser/_parser.py:70: UnicodeWarning: decode() called on unicode string, see https://bugzilla.redhat.com/show_bug.cgi?id=1693751 instream = instream.decode() This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Last metadata expiration check: 1:18:42 ago on Wed 28 Oct 2020 08:40:18 AM EDT. Dependencies resolved. ============================================================================================== Package Arch Version Repository Size ============================================================================================== Upgrading: openstack-tripleo-heat-templates noarch 11.3.2-1.20200914170156.el8ost rhelosp-16.1 654 k openstack-tripleo-validations noarch 11.3.2-1.20200914170825.el8ost rhelosp-16.1 161 k tripleo-ansible noarch 0.5.1-1.20200914163925.el8ost rhelosp-16.1 357 k Transaction Summary ============================================================================================== Upgrade 3 Packages Total download size: 1.1 M Downloading Packages: (1/3): tripleo-ansible-0.5.1-1.20200914163925.e 652 kB/s | 357 kB 00:00 (2/3): openstack-tripleo-validations-11.3.2-1.2 293 kB/s | 161 kB 00:00 (3/3): openstack-tripleo-heat-templates-11.3.2- 1.1 MB/s | 654 kB 00:00 -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- [43/1955] Total 2.0 MB/s | 1.1 MB 00:00 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Upgrading : tripleo-ansible-0.5.1-1.20200914163925.el8ost.noarch 1/6 Upgrading : openstack-tripleo-heat-templates-11.3.2-1.2020091417 2/6 Upgrading : openstack-tripleo-validations-11.3.2-1.2020091417082 3/6 Cleanup : openstack-tripleo-heat-templates-11.3.2-0.2020061608 4/6 Cleanup : tripleo-ansible-0.5.1-0.20200611113659.34b8fcc.el8os 5/6 Cleanup : openstack-tripleo-validations-11.3.2-0.2020061111525 6/6 Running scriptlet: openstack-tripleo-validations-11.3.2-0.2020061111525 6/6 Verifying : openstack-tripleo-heat-templates-11.3.2-1.2020091417 1/6 Verifying : openstack-tripleo-heat-templates-11.3.2-0.2020061608 2/6 Verifying : openstack-tripleo-validations-11.3.2-1.2020091417082 3/6 Verifying : openstack-tripleo-validations-11.3.2-0.2020061111525 4/6 Verifying : tripleo-ansible-0.5.1-1.20200914163925.el8ost.noarch 5/6 Verifying : tripleo-ansible-0.5.1-0.20200611113659.34b8fcc.el8os 6/6 Installed products updated. Upgraded: openstack-tripleo-heat-templates-11.3.2-1.20200914170156.el8ost.noarch openstack-tripleo-validations-11.3.2-1.20200914170825.el8ost.noarch tripleo-ansible-0.5.1-1.20200914163925.el8ost.noarch Complete! Using /tmp/undercloud-disk-space-pre-upgrade.yamli9qda5hxansible.cfg as config file [WARNING]: log file at /usr/share/ansible/validation-playbooks/ansible.log is not writeable and we cannot create it, aborting Success! The validation passed for all hosts: * undercloud Invalid local_interface specified. eth0 is not available. An error occurred during configuration validation, please check your host configuration and try again. Error message: Invalid local_interface specified. eth0 is not available. So there seems to be some difference between inflight and command triggered. And if you wonder that the source of the problem was the lack of tripleo-ansible or tripleo-validations, it didn't make a difference: (undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator run --static-inventory inventory.yaml --validation 512e Failed to discover available identity versions when contacting https://192.168.24.2:13000/. Attempting to parse version from URL. Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to https://192.168.24.2:13000: HTTPSConnectionPool(host='192.168.24.2', port=13000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f5382b45f98>: Failed to establish a new connection: [Errno 113] No route to host',)) (undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator list Failed to discover available identity versions when contacting https://192.168.24.2:13000/. Attempting to parse version from URL. Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Unable to establish connection to https://192.168.24.2:13000: HTTPSConnectionPool(host='192.168.24.2', port=13000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f8dd82a4198>: Failed to establish a new connection: [Errno 113] No route to host',)) These commands were run after the first openstack undercloud upgrade command. (In reply to Jose Luis Franco from comment #1) > Just as a side note, when triggering the "openstack undercloud upgrade" > command the inflight validations for the Undercloud do get triggered: > > (undercloud) [stack@undercloud-0 ~]$ openstack undercloud upgrade -y 2>&1 | > tee undercloud_upgrade.log > > ^[[B^[[B^[[B^[[B^[[BUpdating necessary packages: python3-tripleoclient > openstack-tripleo-common openstack-tripleo-heat-templates > openstack-tripleo-validations tripleo-ansib$ > e > > Updating Subscription Management repositories. > > Unable to read consumer identity > > /usr/lib/python3.6/site-packages/dateutil/parser/_parser.py:70: > UnicodeWarning: decode() called on unicode string, see > https://bugzilla.redhat.com/show_bug.cgi?id=1693751 > instream = instream.decode() > > > > This system is not registered to Red Hat Subscription Management. You can > use subscription-manager to register. > Last metadata expiration check: 1:18:42 ago on Wed 28 Oct 2020 08:40:18 AM > EDT. > Dependencies resolved. > > ============================================================================= > ================= > Package Arch Version > Repository Size > ============================================================================= > ================= > Upgrading: > openstack-tripleo-heat-templates noarch 11.3.2-1.20200914170156.el8ost > rhelosp-16.1 654 k > openstack-tripleo-validations noarch 11.3.2-1.20200914170825.el8ost > rhelosp-16.1 161 k > tripleo-ansible noarch 0.5.1-1.20200914163925.el8ost > rhelosp-16.1 357 k > > Transaction Summary > ============================================================================= > ================= > Upgrade 3 Packages > > Total download size: 1.1 M > Downloading Packages: > (1/3): tripleo-ansible-0.5.1-1.20200914163925.e 652 kB/s | 357 kB 00:00 > (2/3): openstack-tripleo-validations-11.3.2-1.2 293 kB/s | 161 kB 00:00 > (3/3): openstack-tripleo-heat-templates-11.3.2- 1.1 MB/s | 654 kB 00:00 > ----------------------------------------------------------------------------- > --- > ----------------------------------------------------------------------------- > --- > [43/1955] > Total 2.0 MB/s | 1.1 MB 00:00 > Running transaction check > Transaction check succeeded. > Running transaction test > Transaction test succeeded. > Running transaction > Preparing : > 1/1 > Upgrading : tripleo-ansible-0.5.1-1.20200914163925.el8ost.noarch > 1/6 > Upgrading : openstack-tripleo-heat-templates-11.3.2-1.2020091417 > 2/6 > Upgrading : openstack-tripleo-validations-11.3.2-1.2020091417082 > 3/6 > Cleanup : openstack-tripleo-heat-templates-11.3.2-0.2020061608 > 4/6 > Cleanup : tripleo-ansible-0.5.1-0.20200611113659.34b8fcc.el8os > 5/6 > Cleanup : openstack-tripleo-validations-11.3.2-0.2020061111525 > 6/6 > Running scriptlet: openstack-tripleo-validations-11.3.2-0.2020061111525 > 6/6 > Verifying : openstack-tripleo-heat-templates-11.3.2-1.2020091417 > 1/6 > Verifying : openstack-tripleo-heat-templates-11.3.2-0.2020061608 > 2/6 > Verifying : openstack-tripleo-validations-11.3.2-1.2020091417082 > 3/6 > Verifying : openstack-tripleo-validations-11.3.2-0.2020061111525 > 4/6 > Verifying : tripleo-ansible-0.5.1-1.20200914163925.el8ost.noarch > 5/6 > Verifying : tripleo-ansible-0.5.1-0.20200611113659.34b8fcc.el8os > 6/6 > Installed products updated. > > Upgraded: > openstack-tripleo-heat-templates-11.3.2-1.20200914170156.el8ost.noarch > openstack-tripleo-validations-11.3.2-1.20200914170825.el8ost.noarch > tripleo-ansible-0.5.1-1.20200914163925.el8ost.noarch > > Complete! > > Using /tmp/undercloud-disk-space-pre-upgrade.yamli9qda5hxansible.cfg as > config file > [WARNING]: log file at /usr/share/ansible/validation-playbooks/ansible.log > is not writeable and we cannot create it, aborting > > Success! The validation passed for all hosts: > * undercloud > Invalid local_interface specified. eth0 is not available. > An error occurred during configuration validation, please check your host > configuration and try again. > Error message: Invalid local_interface specified. eth0 is not available. > > So there seems to be some difference between inflight and command triggered. > And if you wonder that the source of the problem was the lack of > tripleo-ansible or tripleo-validations, it didn't make a difference: > > (undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator run > --static-inventory inventory.yaml --validation 512e > > Failed to discover available identity versions when contacting > https://192.168.24.2:13000/. Attempting to parse version from URL. > > Could not find versioned identity endpoints when attempting to authenticate. > Please check that your auth_url is correct. Unable to establish connection > to https://192.168.24.2:13000: HTTPSConnectionPool(host='192.168.24.2', > port=13000): Max retries exceeded with url: / (Caused by > NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at > 0x7f5382b45f98>: Failed to establish a new connection: [Errno 113] No route > to host',)) > > > > (undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator list > Failed to discover available identity versions when contacting > https://192.168.24.2:13000/. Attempting to parse version from URL. > Could not find versioned identity endpoints when attempting to authenticate. > Please check that your auth_url is correct. Unable to establish connection > to https://192.168.24.2:13000: HTTPSConnectionPool(host='192.168.24.2', > port=13000): Max retries exceeded with url: / (Caused by > NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at > 0x7f8dd82a4198>: Failed to establish a new connection: [Errno 113] No route > to host',)) > > These commands were run after the first openstack undercloud upgrade command. This comment made me think, that instead of fixing all of this, why don't I include this FFU validations as part of the inflight openstack undercloud upgrade validations? They seem to work fine, as it caught that I had a wrong interface configured, and they could block the upgrade which is more efficient than having a separate step. I will investigate this area and if it's possible, we could close this bugzilla as won't fix. Hello there, Thank you Jose for reporting this issue, and for all the details. After a (really) quick check and hack on my current env, it seems we might "just" add "auth_required = False" to some, if not all, validator subcommands. I'd at least add it for the List and Show things - for the run, it's a bit more complicated due to the fact that we, actually, might need an authentication. Maybe as a parameter. Need to check that specific point, but I can at least start a review in order to disable for the "read-only" things. Cheers, C. So, I did apply the proposed patch in my environment and it did bypass the authentication part. Now I'm able to list validations and groups, however when I try to run any validation/group I get the following: (undercloud) [stack@undercloud-0 ~]$ vi inventory.yaml (undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator run --static-inventory inventory.yaml --group pre-upgrade Error while creating Ansible artifacts log file.Please check the access rights for {} Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/validations_libs/utils.py", line 45, in create_artifacts_dir os.makedirs(log_dir) File "/usr/lib64/python3.6/os.py", line 210, in makedirs makedirs(head, mode, exist_ok) File "/usr/lib64/python3.6/os.py", line 210, in makedirs makedirs(head, mode, exist_ok) File "/usr/lib64/python3.6/os.py", line 220, in makedirs mkdir(name, mode) PermissionError: [Errno 13] Permission denied: '/var/log/validations' Exception occured while running the command Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/validations_libs/utils.py", line 45, in create_artifacts_dir os.makedirs(log_dir) File "/usr/lib64/python3.6/os.py", line 210, in makedirs makedirs(head, mode, exist_ok) File "/usr/lib64/python3.6/os.py", line 210, in makedirs makedirs(head, mode, exist_ok) File "/usr/lib64/python3.6/os.py", line 220, in makedirs mkdir(name, mode) PermissionError: [Errno 13] Permission denied: '/var/log/validations' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_validator.py", line 398, in take_action self._run_validator_run(parsed_args) File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_validator.py", line 369, in _run_validator_run quiet=parsed_args.quiet) File "/usr/lib/python3.6/site-packages/validations_libs/validation_actions.py", line 100, in run_validations prefix=os.path.basename(playbook)) File "/usr/lib/python3.6/site-packages/validations_libs/utils.py", line 48, in create_artifacts_dir LOG.exception("Error while creating Ansible artifacts log file." AttributeError: 'NoneType' object has no attribute 'format' 'NoneType' object has no attribute 'format' (undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator run --static-inventory inventory.yaml --validation 512e Error while creating Ansible artifacts log file.Please check the access rights for {} Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/validations_libs/utils.py", line 45, in create_artifacts_dir os.makedirs(log_dir) File "/usr/lib64/python3.6/os.py", line 210, in makedirs makedirs(head, mode, exist_ok) File "/usr/lib64/python3.6/os.py", line 210, in makedirs makedirs(head, mode, exist_ok) File "/usr/lib64/python3.6/os.py", line 220, in makedirs mkdir(name, mode) PermissionError: [Errno 13] Permission denied: '/var/log/validations' Exception occured while running the command Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/validations_libs/utils.py", line 45, in create_artifacts_dir os.makedirs(log_dir) File "/usr/lib64/python3.6/os.py", line 210, in makedirs makedirs(head, mode, exist_ok) File "/usr/lib64/python3.6/os.py", line 210, in makedirs makedirs(head, mode, exist_ok) File "/usr/lib64/python3.6/os.py", line 220, in makedirs mkdir(name, mode) PermissionError: [Errno 13] Permission denied: '/var/log/validations' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_validator.py", line 398, in take_action self._run_validator_run(parsed_args) File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_validator.py", line 369, in _run_validator_run quiet=parsed_args.quiet) File "/usr/lib/python3.6/site-packages/validations_libs/validation_actions.py", line 100, in run_validations prefix=os.path.basename(playbook)) File "/usr/lib/python3.6/site-packages/validations_libs/utils.py", line 48, in create_artifacts_dir LOG.exception("Error while creating Ansible artifacts log file." AttributeError: 'NoneType' object has no attribute 'format' 'NoneType' object has no attribute 'format' (undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator show history Exception occured while running the command Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run return super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/display.py", line 116, in run column_names, data = self.take_action(parsed_args) File "/usr/lib/python3.6/site-packages/tripleoclient/v1/tripleo_validator.py", line 460, in take_action return actions.show_history(parsed_args.validation) File "/usr/lib/python3.6/site-packages/validations_libs/validation_actions.py", line 164, in show_history if validation_id else vlogs.get_all_logfiles(extension)) File "/usr/lib/python3.6/site-packages/validations_libs/validation_logs.py", line 209, in get_all_logfiles return [join(self.logs_path, f) for f in os.listdir(self.logs_path) if FileNotFoundError: [Errno 2] No such file or directory: '/var/log/validations/' [Errno 2] No such file or directory: '/var/log/validations/' (undercloud) [stack@undercloud-0 ~]$ openstack tripleo validator show parameter {} @Cédric , should we address this issue in this BZ, as they are related? Or should I open another one? Thanks That's for another BZ - the main issue was the authentication here. Verification of this BZ is blocked by: https://bugzilla.redhat.com/show_bug.cgi?id=1912879 Validations can now be run without authentication: openstack tripleo validator run --static-inventory inventory.yaml --group pre-upgrade Running Validations without authentication. +--------------------------------------+-------------------------------------+--------+--------------------------+----------------+-------------------+-------------+ | UUID | Validations | Status | Host_Group | Status_by_Host | Unreachable_Hosts | Duration | +--------------------------------------+-------------------------------------+--------+--------------------------+----------------+-------------------+-------------+ | baef61fe-792d-452e-843f-df6c0f3ec206 | check-disk-space-pre-upgrade | PASSED | all | inventory.yaml | inventory.yaml | 0:00:16.565 | | 48a24b4e-302f-4db5-b77e-58b71c4f2df8 | check-ftype | PASSED | all | inventory.yaml | inventory.yaml | 0:00:04.297 | | 98431c75-cee0-4e14-93d8-f27f39685937 | check-latest-packages-version | PASSED | all | inventory.yaml | inventory.yaml | 0:00:04.442 | | 485e9c4b-66b6-403e-9df3-11f604081638 | check-ram | PASSED | all | inventory.yaml | inventory.yaml | 0:00:04.449 | | a4d416bb-63a4-4383-9bbe-67af80cf8b3c | service-status | PASSED | all | inventory.yaml | inventory.yaml | 0:00:04.641 | | 8a00c87a-1eb6-4c63-907f-2ddd2fddb5e4 | validate-selinux | PASSED | all | inventory.yaml | inventory.yaml | 0:00:04.279 | | ed2f35eb-e519-45fc-99ef-21f36f4f6062 | collect-flavors-and-verify-profiles | PASSED | undercloud | | | | | 5c4f5c21-26f3-46df-82b8-c1acf675d442 | container-status | PASSED | undercloud, allovercloud | | | | | a328e0bc-2da9-4f78-a276-b09c4ca76149 | openstack-endpoints | PASSED | undercloud | | | | | c92afb4d-5954-40c2-b538-4b409b17cf03 | containerized-undercloud-docker | PASSED | undercloud | | | | | fa9fbb07-e05e-4991-b3b1-db769e63dd2c | repos | PASSED | undercloud, allovercloud | | | | | 281f5f6a-99e8-441e-8876-14c76c13ff71 | image-serve | PASSED | undercloud | | | | | b7fbf2bc-47fe-410b-b22b-d43be6cb9b00 | undercloud-service-status | PASSED | undercloud | | | | | a92e3850-c296-4a0d-abec-dab8f69344ed | ironic-boot-configuration | PASSED | undercloud | | | | | 9fd986ab-9969-4309-a276-26d94b5a2074 | stack-health | PASSED | undercloud | | | | | de6d15b9-de86-4462-8c8f-c732f351dbd9 | node-health | PASSED | undercloud | | | | | fb82b219-0ec9-4a5e-92ac-5e91c39ffce0 | nova-status | PASSED | nova_api | | | | | 957c0411-ecfc-4e08-8f90-04c283a2873e | undercloud-disk-space-pre-upgrade | PASSED | undercloud | | | | | f40eca52-e795-4597-9680-1d3188f605ce | undercloud-heat-purge-deleted | PASSED | undercloud | | | | +--------------------------------------+-------------------------------------+--------+--------------------------+----------------+-------------------+-------------+ Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.4 director bug fix advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:0817 |