Bug 2055136

Summary: virt module is not changed to the correct stream during host upgrade
Product: Red Hat Enterprise Virtualization Manager Reporter: Petr Kubica <pkubica>
Component: ovirt-engineAssignee: Dana <delfassy>
Status: CLOSED ERRATA QA Contact: Petr Kubica <pkubica>
Severity: high Docs Contact:
Priority: high    
Version: 4.4.10CC: bugs, delfassy, emarcus, lsvaty, mavital, michal.skrivanek, mkalinin, mperina, pbrilla
Target Milestone: ovirt-4.5.0-1Keywords: Automation, AutomationBlocker, ZStream
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: ovirt-engine-4.5.0.6 Doc Type: Release Note
Doc Text:
With this release, the virt DNF module version is correctly set according to the RHEL version of the host during the host upgrade flow.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2022-05-26 16:23:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Infra RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Petr Kubica 2022-02-16 10:52:32 UTC 
Description of problem:
Module virt was not changed from 8.2 (or 8.3) to av during host upgrade
(basically this doesn't work any more: https://bugzilla.redhat.com/show_bug.cgi?id=1877679)

Looks like tasks in ovirt-host-deploy-facts especially related to av are skipped (/usr/share/ovirt-engine/ansible-runner-service-project/project/ovirt-host-upgrade.yml - /usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-host-deploy-facts/tasks/av-setup.yml) which are performed before failing task 

it is failing on:
- name: Install ovirt-host package if it isn't installed
  yum:
    name: ovirt-host
    state: latest
    update_cache: yes
    lock_timeout: 300
  tags:
    - updatecheck
    - skip_ansible_lint # E403

in /usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-host-upgrade/tasks/main.yml:8

The upgrade fails on dependency issue: (ansible-runner-service.log)
2022-02-15 18:18:07,777 - runner_service.services.playbook - DEBUG - cb_event_handler event_data={'uuid': 'ae6b6253-b7c4-4407-a815-723cc413843f', 'counter': 98, 'stdout': '', 'start_line': 92, 'end_line': 92, 'runner_ident': '8cbb4646-8e8b-11ec-8f7e-001a4a014700', 'event': 'runner_on_start', 'pid': 13838, 'created': '2022-02-15T18:18:07.776237', 
'parent_uuid': '001a4a01-4700-51a3-76c0-00000000001d', 'event_data': {'playbook': 'ovirt-host-upgrade.yml', 'playbook_uuid': '271a64e9-e0ec-4b23-9c87-c5afeb2c7f1d', 'play': 'all', 'play_uuid': '001a4a01-4700-51a3-76c0-00000000000b', 'play_pattern': 'all', 'task': "Install ovirt-host package if it isn't installed", 'task_uuid': '001a4a01-4700-51a3
-76c0-00000000001d', 'task_action': 'yum', 'task_args': '', 'task_path': '/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-host-upgrade/tasks/main.yml:8', 'role': 'ovirt-host-upgrade', 'host': '<FQDN_OF_THE_HOST>', 'uuid': 'ae6b6253-b7c4-4407-a815-723cc413843f'}}

2022-02-15 18:18:17,411 - runner_service.services.playbook - DEBUG - cb_event_handler event_data={'uuid': '9038e101-2030-4058-8a3c-0c495ca1a6ee', 'counter': 99, 'stdout': 'fatal: [<FQDN_OF_THE_HOST>]: FAILED! => {"changed": false, "failures": [], "msg": "Depsolve Error occured: \\n Problem: package ovirt-host-dependencies-4.4.9-2.el8ev.x86_64 requires vdsm >= 4.40.90, but none of the providers can be
 installed\\n  - package ovirt-host-4.4.9-2.el8ev.x86_64 requires ovirt-host-dependencies = 4.4.9-2.el8ev, but none of the providers can be installed\\n  - package vdsm-4.40.90.4-1.el8ev.x86_64 requires libvirt-daemon-kvm >= 7.6.0-4.module+el8.5.0+12786+c4633d9a, but none of the providers can be installed\\n  - package vdsm-4.40.90.2-1.el8ev.x86_64 requires libvirt-daemon-kvm >= 7.6.0-4.module+el8.5.0+12786+c4633d9a, but n
one of the providers can be installed\\n  - package vdsm-4.40.90.5-1.el8ev.x86_64 requires libvirt-daemon-kvm >= 7.6.0-4.module+el8.5.0+12786+c4633d9a, but none of the providers can be installed\\n  - package vdsm-4.40.100.2-1.el8ev.x86_64 requires libvirt-daemon-kvm >= 7.6.0-4.module+el8.5.0+12786+c4633d9a, but none of the providers can be installed\\n  - cannot install the best update candidate for package ovirt-host-4.4
.1-4.el8ev.x86_64\\n  - package libvirt-daemon-kvm-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 is filtered out by modular filtering", "rc": 1, "results": []}', 'start_line': 92, 'end_line': 93, 'runner_ident': '8cbb4646-8e8b-11ec-8f7e-001a4a014700', 'event': 'runner_on_failed', 'pid': 13838, 'created': '2022-02-15T18:18:17.410133', 'parent_uuid': '001a4a01-4700-51a3-76c0-00000000001d', 'event_data': {'playbook': 'ovirt-ho
st-upgrade.yml', 'playbook_uuid': '271a64e9-e0ec-4b23-9c87-c5afeb2c7f1d', 'play': 'all', 'play_uuid': '001a4a01-4700-51a3-76c0-00000000000b', 'play_pattern': 'all', 'task': "Install ovirt-host package if it isn't installed", 'task_uuid': '001a4a01-4700-51a3-76c0-00000000001d', 'task_action': 'yum', 'task_args': '', 'task_path': '/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-host-upgrade/tasks/m
ain.yml:8', 'role': 'ovirt-host-upgrade', 'host': '<FQDN_OF_THE_HOST>', 'remote_addr': '<FQDN_OF_THE_HOST>', 'res': {'msg': 'Depsolve Error occured: \n Problem: package ovirt-host-dependencies-4.4.9-2.el8ev.x86_64 requires vdsm >= 4.40.90, but none of the providers can be installed\n  - package ovirt-host-4.4.9-2.el8ev.x86_64 requires ovirt-host-dependencies = 4.4.9-2.el8ev, bu
t none of the providers can be installed\n  - package vdsm-4.40.90.4-1.el8ev.x86_64 requires libvirt-daemon-kvm >= 7.6.0-4.module+el8.5.0+12786+c4633d9a, but none of the providers can be installed\n  - package vdsm-4.40.90.2-1.el8ev.x86_64 requires libvirt-daemon-kvm >= 7.6.0-4.module+el8.5.0+12786+c4633d9a, but none of the providers can be installed\n  - package vdsm-4.40.90.5-1.el8ev.x86_64 requires libvirt-daemon-kvm >=
 7.6.0-4.module+el8.5.0+12786+c4633d9a, but none of the providers can be installed\n  - package vdsm-4.40.100.2-1.el8ev.x86_64 requires libvirt-daemon-kvm >= 7.6.0-4.module+el8.5.0+12786+c4633d9a, but none of the providers can be installed\n  - cannot install the best update candidate for package ovirt-host-4.4.1-4.el8ev.x86_64\n  - package libvirt-daemon-kvm-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 is filtered out by 
modular filtering', 'failures': [], 'results': [], 'rc': 1, 'invocation': {'module_args': {'name': ['ovirt-host'], 'state': 'latest', 'update_cache': True, 'lock_timeout': 300, 'allow_downgrade': False, 'autoremove': False, 'bugfix': False, 'disable_gpg_check': False, 'disable_plugin': [], 'disablerepo': [], 'download_only': False, 'enable_plugin': [], 'enablerepo': [], 'exclude': [], 'installroot': '/', 'install_repoquery
': True, 'install_weak_deps': True, 'security': False, 'skip_broken': False, 'update_only': False, 'validate_certs': True, 'conf_file': None, 'disable_excludes': None, 'download_dir': None, 'list': None, 'releasever': None}}, '_ansible_no_log': False, 'changed': False}, 'start': '2022-02-15T18:18:07.776150', 'end': '2022-02-15T18:18:17.409825', 'duration': 9.633675, 'ignore_errors': None, 'event_loop': None, 'uuid': '9038e
101-2030-4058-8a3c-0c495ca1a6ee'}}

After fail:
[root@sat-client-01 ~]# yum module list --enabled
...
virt 8.2 [e] common Virtualization module
...

Version-Release number of selected component (if applicable):
upgrade from 8.2 or 8.3 RHEL
with engine:
ovirt-engine-4.4.10.6-0.1.el8ev.noarch

How reproducible:
always

Steps to Reproduce:
1. Have a host with virt:8.2 or virt:8.3
2. upgrade these hosts to latest 4.4.10
Comment 1 Michal Skrivanek 2022-02-17 09:17:02 UTC 
AV change need to happen before upgrading ovirt-host, or it shouldn't be trying o upgrade it to latest
Comment 2 Martin Perina 2022-02-17 10:55:58 UTC 
(In reply to Michal Skrivanek from comment #1)
> AV change need to happen before upgrading ovirt-host, or it shouldn't be
> trying o upgrade it to latest

Not true, virt module version is change during the upgrade from RHEL 8.m to RHEL 8.n

As discussed on the grooming today, repositories are not correct, that's why upgrade fails
Comment 3 Petr Kubica 2022-02-17 11:10:45 UTC 
Repositories are correct.

Used standard build (which was used a lot of times for installing host)

Also I used the same repositories for upgrading from 
RHEL 8.4 - virt:av - RHV 4.4.8  to  RHEL 8.5 virt:av RHV 4.4.10 and there it worked

Non-working scenarios:
RHEL 8.2 - virt:8.2 - RHV 4.4.2
RHEL 8.3 - virt:8.3 - RHV 4.4.5
Comment 6 Pavol Brilla 2022-04-27 08:22:55 UTC 
upgrade from 4.4(8.5) to 4.5(8.6) is also failing:

  "stdout" : "fatal: [FQDN]: FAILED! => {\"msg\": \"The conditional check 'ansible_facts.packages['r
edhat-release'].version is version(latest_rh_rel, '!=')' failed. The error was: error while evaluating conditional (ansible_facts.pa
ckages['redhat-release'].version is version(latest_rh_rel, '!=')): 'list object' has no attribute 'version'\\n\\nThe error appears t
o be in '/usr/share/ovirt-engine/ansible-runner-service-project/project/roles/ovirt-host-upgrade/tasks/advanced-virtualization.yml':
 line 16, column 11, but may\\nbe elsewhere in the file depending on the exact syntax problem.\\n\\nThe offending line appears to be
:\\n\\n    - block:\\n        - name: Reset configuration of advanced virtualization module\\n          ^ here\\n\"}",
Comment 7 Pavol Brilla 2022-04-27 08:28:14 UTC 
testing from latest live environment (4.4.10) to latest internal build containing ovirt-engine-setup-4.5.0.4-0.1.el8ev.noarch.rpm


upgrade of engine itself looks fine and was successful, I will do some more runs to be sure
host upgrade from engine GUI - Check for Updgrades > Upgrade is failing, above message was found in ovirt-host-mgmt-ansible...
Comment 8 Dana 2022-04-27 09:06:21 UTC 
I saw in host upgrade log that all tasks related to av-setup are skipped:
2022-02-15 18:17:48 UTC - TASK [ovirt-host-deploy-facts : Reset configuration of advanced virtualization module] ***
2022-02-15 18:17:48 UTC - TASK [ovirt-host-deploy-facts : Find relevant advanced virtualization module version] ***
2022-02-15 18:17:48 UTC - TASK [ovirt-host-deploy-facts : Enable advanced virtualization module] *********

I checked the conditions for execution av-setup tasks, and found that:
    - not vdsm_preinstalled|bool 
is not met:

2022-02-15 18:17:48 UTC - TASK [ovirt-host-deploy-facts : Check if vdsm is preinstalled] *****************
2022-02-15 18:17:48 UTC - ok: [sat-client-02.rhev.lab.eng.brq.redhat.com]
      "res" : {
        "changed" : false,
        "ansible_facts" : {
          "vdsm_preinstalled" : true
        },

in Fetch installed packages:
"vdsm" : [ {
              "name" : "vdsm",
              "version" : "4.40.50.10",
              "release" : "1.el8ev",
              "epoch" : null,
              "arch" : "x86_64",
              "source" : "rpm"
            } ],

so this is the reason the version is not changed.
Does it really matter is vdsm is preinstalled or can we just remove this condition?
Comment 9 Michal Skrivanek 2022-04-27 09:14:45 UTC 
in which log, the one from Feb? that's irrelevant. Please fix the _current_ problem, and current change from 4.2/4.3/4.4 to 4.5's virt:rhel

Pavol, please attach the log
Comment 11 Pavol Brilla 2022-04-27 09:44:23 UTC 
I attached log from upgrade:
ovirt-host-mgmt-ansible-20220427111347-FQDN-d4fdda58-bf4e-4dd9-8576-42f7507bc65e.log

I tried to generate sos report or ovirt-log-collctor but in both cases size was too big to attach.

I contacted also Dana through chat, and provided her with access to environment, so she can also have full picture of env.


Current issue which I have is upgrade 4.4.10 > 4.5, ( attempts for upgrade from 4.4.2 or other previous version will come latter )
Comment 12 Dana 2022-04-27 10:01:57 UTC 
(In reply to Michal Skrivanek from comment #9)
> in which log, the one from Feb? that's irrelevant. Please fix the _current_
> problem, and current change from 4.2/4.3/4.4 to 4.5's virt:rhel
> 
> Pavol, please attach the log

It was in the log from February, but that's the issue (I checked it earlier today). I'm creating a fix for it now
Comment 13 Dana 2022-04-27 12:42:09 UTC 
The patch I added includes fixes both for the original bug & the issue Pavol found
Comment 16 Marina Kalinin 2022-05-05 21:41:41 UTC 
Note: I see the docs are already updated with SP1 instructions:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html-single/upgrade_guide/index#Manually_Updating_Hosts_minor_updates
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html-single/installing_red_hat_virtualization_as_a_self-hosted_engine_using_the_command_line/index#Enabling_the_Red_Hat_Enterprise_Linux_Host_Repositories_SHE_deployment_host

So we are good here.
Comment 17 Petr Kubica 2022-05-09 10:32:52 UTC 
Verified in ovirt-engine-4.5.0.6-0.7.el8ev.noarch

Tested host upgrade from 4.4.4 (RHEL 8.3) to 4.5 (RHEL 8.6)
Module stream was changed from virt:8.3 to virt:rhel during the upgrade
Comment 22 errata-xmlrpc 2022-05-26 16:23:53 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: RHV Manager (ovirt-engine) [ovirt-4.5.0] security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:4711