How does this procedure work for validations designed to be run prior to undercloud deployment, e.g. the 'prep' group? Reported by: nweinber1 https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.0/html/director_installation_and_usage/using-the-validation-framework#annotations:49a57c72-b361-4cd4-91dd-46149871c2ea
This is still "a topic" for the VF - there are some concerns about how to run prep validations. Mainly, two spirits: - "do not deploy|install anything, and run prep validations using plain ansible-playbook" CLI and things - "prep are meant to be launched after you get a working undercloud" (mainly because of the CLI integration, which requires authentication for the validation on a global scale) Both are correct on their own, but also wrong: for instance, a customer wants to run a prep validation in order to ensure their director/undercloud node is sufficient... We still need to figure out the right thing to do - I'll push this as a topic for my weekly with the squad and try to come with something we're OK to support.
Hello there, I would say that, for 16.0 (and probably up to 17, unless we're able to provide some z-stream for 16.x.y release), we must have a deployed undercloud in order to run the validations. We currently don't have human power nor time enough to actually solve this issue - it would require some more discussions and doc, and we must provide a better UI than "ansible-playbook [many options]". We might provide a temporary KCS explaining how to actually run the "ansible-playbook" thing, but this shouldn't be in the official doc for the validations. Would that be OK? This situation is NOT definitive and will evolve, we're just unable to provide something that will satisfy both CU and Support right now :(. Cheers, C.
Hey Cedric, This is fine with me; so long as the docs are consistent with the functionality I will defer to your expertise about what is/is not realistic to implement at this stage :) Best, Nathan
Agree with Nathan, if the functionality is out there for our users, our docs should be a source of truth Thanks, Eliad
Hello, Quick update: we should be able to run validations before the undercloud is deployed using a new script we'll be providing soon - I'll need to check again with my team, but we should be able to ship it in osp-16.2 already. A backport *might* be possible for osp-16.1, but it will require some more discussions, as it includes a change in the packages being installed. Not 100% sure this is something we want to introduce in a z-stream. I'll keep you updated for osp-16.2 changes. Cheers, C.
Thanks Cedric, looking forward to use it. Agree wrt to backport, not sure this is worth the risk/effort.
Update: - moving to 16.1 since 16.0 is soon EOL, and still a valid question for 16.1 - the fact we moved to the new validations-libs and validations-common packages for the framework should help a lot, since we provide a script in there - we're actively working on the doc (once we fix all the current open issues) - expect some major update for z4 (z3 is more for stabilisation). Cheers, C.
@cjeanner -- Did the script land for 16.1?
Hey Dan, I just checked, it is: [RedHat-8.2 - root@undercloud ~]# which validation.py /usr/bin/validation.py [RedHat-8.2 - root@undercloud ~]# rpm -qf /usr/bin/validation.py validations-common-1.1.2-1.20200914180306.el8ost.noarch [RedHat-8.2 - root@undercloud ~]# validation.py --help usage: validation.py [-h] [--inventory INVENTORY] [--extra-vars EXTRA_VARS [EXTRA_VARS ...]] [--validation <validation_id>[,<validation_id>,...]] [--group <group>[,<group>,...]] [--quiet] [--validation-dir VALIDATION_DIR] [--ansible-base-dir ANSIBLE_BASE_DIR] [--output-log OUTPUT_LOG] {run,list,show} Run, show or list Validations. positional arguments: {run,list,show} Validation Action optional arguments: -h, --help show this help message and exit --inventory INVENTORY, -i INVENTORY Path of the Ansible inventory. --extra-vars EXTRA_VARS [EXTRA_VARS ...] Extra ansible variables --validation <validation_id>[,<validation_id>,...], -v <validation_id>[,<validation_id>,...] Run specific validations, if more than one validation is required separate the names with commas: --validation check-ftype,512e | --validation 512e --group <group>[,<group>,...], -g <group>[,<group>,...] Run specific group validations, if more than one group is required separate the group names with commas: --group pre-upgrade,prep | --group openshift-on- openstack --quiet Run Ansible in silent mode. --validation-dir VALIDATION_DIR Path where the validation playbooks is located. --ansible-base-dir ANSIBLE_BASE_DIR Path where the ansible roles, library and plugins is located. --output-log OUTPUT_LOG Path where the run result will be stored Example: ./validation run --validation check-ftype,512e For instance, `validation.py --group pre-deployment run' seems to work pretty well: +--------------------------------------+-------------------------------------+--------+--------------+----------------+-------------------+-------------+ | UUID | Validations | Status | Host_Group | Status_by_Host | Unreachable_Hosts | Duration | +--------------------------------------+-------------------------------------+--------+--------------+----------------+-------------------+-------------+ | 890457c1-4392-439d-9a60-c7ab183ccdca | 512e | PASSED | all | localhost | | 0:00:01.869 | | 5051753a-df7e-47cd-816b-db96080e5715 | switch-vlans | PASSED | undercloud | | | | | 55377061-e150-49e2-98c5-995db7f8851f | dns | PASSED | all | localhost | | 0:00:00.572 | | 3d8d07cb-8658-4b22-b1f5-10356e49b875 | service-status | PASSED | all | localhost | | 0:00:05.628 | | 1e1423fd-9c63-47a6-9e23-76cba066aa44 | tls-everywhere-pre-deployment | PASSED | undercloud | | | | | dc45812c-f34d-4c89-bf59-ee02594c8fd9 | validate-selinux | PASSED | all | localhost | | 0:00:01.942 | | 3bd38dc1-22c2-4740-af4e-7d31f8605cd9 | ceilometerdb-size | PASSED | undercloud | | | | | 6c98f334-547c-433e-9fc8-eacf21de5803 | undercloud-debug | PASSED | undercloud | | | | | 82903ede-e273-4910-b8c2-6da96ac03f08 | ceph-ansible-installed | PASSED | undercloud | | | | | 51de8995-fbdc-40ae-92c8-d0f56e9484f7 | ceph-dependencies-installed | PASSED | allovercloud | | | | | 4e496d40-0d92-46b7-9db8-d0d36c053306 | collect-flavors-and-verify-profiles | PASSED | undercloud | | | | | 2d81fb04-1ab8-4143-b9ef-b00200456a9b | default-node-count | PASSED | undercloud | | | | | da5e29d2-8c0a-4db2-8d41-374ebb908520 | dhcp-provisioning | PASSED | undercloud | | | | | 27fa0f9f-08f7-4633-b430-14be9b8ffac3 | ironic-boot-configuration | PASSED | undercloud | | | | | e14e142c-6ad1-4bf5-ac49-1f90bf29dc75 | network-environment | PASSED | undercloud | | | | | 86a7d046-5378-4417-9a84-30aec3ffac0a | node-disks | PASSED | undercloud | | | | | 5fe82b1a-7941-4abd-9451-0c64a80f40ed | undercloud-heat-purge-deleted | PASSED | undercloud | | | | | 1bd2ea31-3849-489a-bd37-24d8e0c4c30c | undercloud-process-count | PASSED | undercloud | | | | +--------------------------------------+-------------------------------------+--------+--------------+----------------+-------------------+-------------+ Though I have a small doubt about the actual inventory used in this run... Asking @mbultel and @gchamoul for confirmation - pretty sure we'll need to tweak a minimal inventory in order to ensure we match "undercloud" and "allovercloud" groups... Cheers, C.
Hi Cédric, Dan, We already cover that case in the Upstream Doc[1]. This should be reworked especially because the CALLBACK referenced in the doc is now hosted in validations-common. Honestly, I won't use the validation.py script here because: 1- this a the script we use for the CI, 2- and IMO this script should have been included in a rpm! [1] - https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/post_deployment/validations/ansible.html
Hi folks, So I tried out validation.py and got it to run the prep validations without undercloud installed. However, on 16.1, the results of those validations don't appear. So this is my inventory file: undercloud ansible_host=10.65.176.99 Running as root user (and had to ssh-copy-id root's public key to itself to get it to work). The validations run but the output isn't in a table (which I'm guessing is an upcoming version). Instead it seems like standard ansible.log output: Monday 14 June 2021 06:08:44 +0000 (0:00:00.041) 0:00:00.041 *********** Monday 14 June 2021 06:08:45 +0000 (0:00:00.511) 0:00:00.552 *********** Monday 14 June 2021 06:08:45 +0000 (0:00:00.110) 0:00:00.662 *********** =============================================================================== advanced_format_512e_support : List the available drives ---------------- 0.51s advanced_format_512e_support : Detect whether the drive uses Advanced Format --- 0.11s Monday 14 June 2021 06:08:45 +0000 (0:00:00.041) 0:00:00.041 *********** Monday 14 June 2021 06:08:48 +0000 (0:00:02.199) 0:00:02.241 *********** Monday 14 June 2021 06:08:48 +0000 (0:00:00.059) 0:00:02.301 *********** =============================================================================== Gathering Facts --------------------------------------------------------- 2.20s check_cpu : Verify the number of CPU cores ------------------------------ 0.06s Monday 14 June 2021 06:08:49 +0000 (0:00:00.042) 0:00:00.042 *********** Monday 14 June 2021 06:08:49 +0000 (0:00:00.057) 0:00:00.099 *********** Monday 14 June 2021 06:08:50 +0000 (0:00:01.328) 0:00:01.428 *********** Monday 14 June 2021 06:08:50 +0000 (0:00:00.057) 0:00:01.485 *********** Monday 14 June 2021 06:08:50 +0000 (0:00:00.120) 0:00:01.606 *********** Monday 14 June 2021 06:08:51 +0000 (0:00:00.867) 0:00:02.474 *********** Etc, etc, etc... So it doesn't really display whether it's a pass or fail. And when I run the show command, I get an empty table: # validation.py show +------+-------------+--------+--------------+----------+ | UUID | Validations | Status | Execution at | Duration | +------+-------------+--------+--------------+----------+ +------+-------------+--------+--------------+----------+ The artifacts directory contains data and each of the subdirectories contain a status file. We could document this method, but it seems a bit of an awkward way for the customer to check the status file of each validation. A table would seem more user friendly. But just want to check, am I missing something with the show command? Is there a specific location the artifacts should be stored?
Hey Dan, I deploy a recent 16.1 to see the state of the VF here. I run the pre-deployment group, without any Undercloud deployed there, this is the result of my command [1]: So simply a: validation.py run --group pre-deployment With no inventory file provided. [1] [stack@undercloud-0 ~]$ validation.py run --group pre-deployment Friday 18 June 2021 09:16:55 -0400 (0:00:00.053) 0:00:00.053 *********** Friday 18 June 2021 09:16:56 -0400 (0:00:00.621) 0:00:00.675 *********** Friday 18 June 2021 09:16:56 -0400 (0:00:00.111) 0:00:00.786 *********** =============================================================================== advanced_format_512e_support : List the available drives ---------------- 0.62s advanced_format_512e_support : Detect whether the drive uses Advanced Format --- 0.11s [WARNING]: Could not match supplied host pattern, ignoring: undercloud Friday 18 June 2021 09:16:58 -0400 (0:00:00.058) 0:00:00.058 *********** Friday 18 June 2021 09:16:58 -0400 (0:00:00.513) 0:00:00.572 *********** =============================================================================== dns : Ensure DNS resolution works --------------------------------------- 0.51s Friday 18 June 2021 09:16:59 -0400 (0:00:00.062) 0:00:00.062 *********** Friday 18 June 2021 09:16:59 -0400 (0:00:00.189) 0:00:00.252 *********** Friday 18 June 2021 09:17:01 -0400 (0:00:01.702) 0:00:01.954 *********** Friday 18 June 2021 09:17:01 -0400 (0:00:00.160) 0:00:02.114 *********** Friday 18 June 2021 09:17:02 -0400 (0:00:00.568) 0:00:02.683 *********** Friday 18 June 2021 09:17:02 -0400 (0:00:00.572) 0:00:03.255 *********** Friday 18 June 2021 09:17:03 -0400 (0:00:00.224) 0:00:03.480 *********** Friday 18 June 2021 09:17:03 -0400 (0:00:00.165) 0:00:03.645 *********** Friday 18 June 2021 09:17:03 -0400 (0:00:00.162) 0:00:03.808 *********** Friday 18 June 2021 09:17:03 -0400 (0:00:00.211) 0:00:04.020 *********** Friday 18 June 2021 09:17:03 -0400 (0:00:00.315) 0:00:04.336 *********** Friday 18 June 2021 09:17:04 -0400 (0:00:00.137) 0:00:04.474 *********** =============================================================================== service_status : Gather package facts ----------------------------------- 1.70s service_status : Get failed containers for podman ----------------------- 0.57s service_status : Do we have podman -------------------------------------- 0.57s service_status : Get failed services from Systemd ----------------------- 0.32s service_status : Fail if we detect failed podman container -------------- 0.23s service_status : include_tasks ------------------------------------------ 0.21s service_status : include_tasks ------------------------------------------ 0.19s service_status : Get failed containers from docker ---------------------- 0.17s service_status : Fail if we detect failed docker container -------------- 0.16s service_status : Is docker running -------------------------------------- 0.16s service_status : Fails if we find failed systemd units ------------------ 0.14s Friday 18 June 2021 09:17:04 -0400 (0:00:00.096) 0:00:00.096 *********** Friday 18 June 2021 09:17:05 -0400 (0:00:00.471) 0:00:00.568 *********** Friday 18 June 2021 09:17:05 -0400 (0:00:00.059) 0:00:00.627 *********** Friday 18 June 2021 09:17:05 -0400 (0:00:00.060) 0:00:00.688 *********** Friday 18 June 2021 09:17:05 -0400 (0:00:00.057) 0:00:00.745 *********** Friday 18 June 2021 09:17:05 -0400 (0:00:00.058) 0:00:00.804 *********** Friday 18 June 2021 09:17:06 -0400 (0:00:00.360) 0:00:01.164 *********** Friday 18 June 2021 09:17:06 -0400 (0:00:00.265) 0:00:01.430 *********** Friday 18 June 2021 09:17:06 -0400 (0:00:00.059) 0:00:01.489 *********** Friday 18 June 2021 09:17:06 -0400 (0:00:00.058) 0:00:01.548 *********** Friday 18 June 2021 09:17:06 -0400 (0:00:00.058) 0:00:01.607 *********** Friday 18 June 2021 09:17:06 -0400 (0:00:00.058) 0:00:01.665 *********** Friday 18 June 2021 09:17:06 -0400 (0:00:00.058) 0:00:01.724 *********** Friday 18 June 2021 09:17:06 -0400 (0:00:00.057) 0:00:01.782 *********** Friday 18 June 2021 09:17:06 -0400 (0:00:00.058) 0:00:01.840 *********** Friday 18 June 2021 09:17:06 -0400 (0:00:00.058) 0:00:01.899 *********** Friday 18 June 2021 09:17:06 -0400 (0:00:00.059) 0:00:01.958 *********** =============================================================================== validate_selinux : Ensure /var/log/audit/audit.log does exist ----------- 0.47s validate_selinux : Fetch denials from auditlog -------------------------- 0.36s validate_selinux : Get stat for denials.log ----------------------------- 0.27s validate_selinux : Load skip list from provided file -------------------- 0.06s validate_selinux : Everything is fine ----------------------------------- 0.06s validate_selinux : Output information if everything is fine ------------- 0.06s validate_selinux : Fail if /var/log/audit/audit.log does not exit ------- 0.06s validate_selinux : Output information in case we do not fail ------------ 0.06s validate_selinux : Create skip list ------------------------------------- 0.06s validate_selinux : Get stat for filtered denials ------------------------ 0.06s validate_selinux : No skip_list ----------------------------------------- 0.06s validate_selinux : Filter out denials ----------------------------------- 0.06s validate_selinux : Load skip list variables (undercloud or overcloud) --- 0.06s validate_selinux : Fail if we found untracked denials ------------------- 0.06s validate_selinux : Gather subset of facts ------------------------------- 0.06s validate_selinux : debug ------------------------------------------------ 0.06s [WARNING]: Could not match supplied host pattern, ignoring: undercloud [WARNING]: Could not match supplied host pattern, ignoring: undercloud [WARNING]: Could not match supplied host pattern, ignoring: undercloud [WARNING]: Could not match supplied host pattern, ignoring: allovercloud [WARNING]: Could not match supplied host pattern, ignoring: undercloud [WARNING]: Could not match supplied host pattern, ignoring: undercloud [WARNING]: Could not match supplied host pattern, ignoring: undercloud [WARNING]: Could not match supplied host pattern, ignoring: undercloud [WARNING]: Could not match supplied host pattern, ignoring: undercloud [WARNING]: Could not match supplied host pattern, ignoring: undercloud [WARNING]: Could not match supplied host pattern, ignoring: undercloud [WARNING]: Could not match supplied host pattern, ignoring: undercloud [WARNING]: Could not match supplied host pattern, ignoring: undercloud +--------------------------------------+-------------------------------------+--------+--------------+----------------+-------------------+-------------+ | UUID | Validations | Status | Host_Group | Status_by_Host | Unreachable_Hosts | Duration | +--------------------------------------+-------------------------------------+--------+--------------+----------------+-------------------+-------------+ | d1add578-6b05-4c97-af86-84baabe2747d | 512e | PASSED | all | localhost | | 0:00:00.741 | | c9e9be79-67de-4c73-b04f-c0a0663125f0 | switch-vlans | PASSED | undercloud | | | | | da3ef74d-39bc-418b-9d4e-b6a1c0a32d00 | dns | PASSED | all | localhost | | 0:00:00.524 | | 74cc4d0c-0769-4199-9d58-15d172296815 | service-status | PASSED | all | localhost | | 0:00:04.397 | | ac9cd8f6-4e7b-4456-bdb5-206334b98ec0 | validate-selinux | PASSED | all | localhost | | 0:00:01.908 | | 922408e4-fa24-45d4-8b95-9b5c3b28f3f5 | undercloud-heat-purge-deleted | PASSED | undercloud | | | | | 623be412-7f3f-42fc-8fda-9c150670c483 | ceilometerdb-size | PASSED | undercloud | | | | | 6b899ce4-61e4-4073-8fb4-dddc9239aa3d | ceph-ansible-installed | PASSED | undercloud | | | | | 37533107-983e-4a47-ad49-a152cc52c9ef | ceph-dependencies-installed | PASSED | allovercloud | | | | | a1073255-6b6a-41ca-b47b-cfbc1b8e51e3 | undercloud-process-count | PASSED | undercloud | | | | | 03bf005d-7573-4c1d-8d3a-49b63f0660a3 | collect-flavors-and-verify-profiles | PASSED | undercloud | | | | | 7f2bde27-6c39-4f34-9a72-6900f1fcd411 | default-node-count | PASSED | undercloud | | | | | f84bf558-b1b0-43e2-922c-b817ebd358a2 | dhcp-provisioning | PASSED | undercloud | | | | | 8d83e731-796d-43ac-b79d-690a0bd4934a | ironic-boot-configuration | PASSED | undercloud | | | | | 5e240f9f-ecf3-4c02-8d92-d9c8930c04ca | network-environment | PASSED | undercloud | | | | | 62ef6387-8b39-4589-b394-d1034ea528f6 | node-disks | PASSED | undercloud | | | | | eb79e439-b764-4ca7-8315-14ebfcc23201 | tls-everywhere-pre-deployment | PASSED | undercloud | | | | | a41e3416-182b-4507-a8d5-c08bda17dff9 | undercloud-debug | PASSED | undercloud | | | | +--------------------------------------+-------------------------------------+--------+--------------+----------------+-------------------+-------------+
*** Bug 1898482 has been marked as a duplicate of this bug. ***
Hello there, Is there anything more to do on this topic? Or shall we close it in some way? Cheers, C.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days