Bug 1730538 - Engine compares incorrect versions when deciding to attach Tools ISO to Windows VMs
Summary: Engine compares incorrect versions when deciding to attach Tools ISO to Windo...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: rhvm-setup-plugins
Version: 4.3.4
Hardware: x86_64
OS: Linux
high
medium
Target Milestone: ovirt-4.3.7
: 4.3.7
Assignee: Liran Rotenberg
QA Contact: Petr Matyáš
URL:
Whiteboard:
Depends On:
Blocks: 1768393 1768387
TreeView+ depends on / blocked
 
Reported: 2019-07-17 05:17 UTC by Germano Veit Michel
Modified: 2019-12-12 10:37 UTC (History)
13 users (show)

Fixed In Version: rhvm-setup-plugins-4.3.5-1,ovirt-engine-4.3.7.1
Doc Type: Bug Fix
Doc Text:
The Windows Guest Tools ISO was automatically attached to a Windows virtual machine even when it was an older version. In this release the ISO version is checked in the data domains and the ISO domains, and the ISO is automatically attached only if it is a newer version than the existing one.
Clone Of:
Environment:
Last Closed: 2019-12-12 10:36:34 UTC
oVirt Team: Virt
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 4288091 None None None 2019-07-17 05:44:50 UTC
Red Hat Product Errata RHBA-2019:4229 None None None 2019-12-12 10:37:02 UTC
oVirt gerrit 103887 'None' MERGED core: add auto attaching from data domains 2020-01-27 12:48:16 UTC

Description Germano Veit Michel 2019-07-17 05:17:38 UTC
Description of problem:

Something seems to be off here, let me explain...

The logic in the engine is to automatically attach the Guest Tools ISO to Windows VMs that have outdated tools, or need it installed. This logic compares 2 things to decide if the tools need to be updated:

1) the ISO version, i.e. 4.2.9 (extracted from the iso filename, RHV-toolsSetup_4.2_9.iso)
2) RHEV-Agent version from the vm app_list, this comes from a list of valid substrings in vdc_option AgentAppName list.

So basically RHV-toolsSetup_X.Y.Z.iso vs RHEV-Agent X.Y.Z.

Problem is the X.Y.Z version in the ISO does not match the X.Y.Z version of Agent the exact same ISO ships(?!), so the validation is wrong, as if the VM needs updating the Tools.

Take RHV-toolsSetup_4.2_8.iso, it ships with RHEV-Agent 4.2.2. This will always make the engine think the VM needs upgrading, because 4.2.8 > 4.2.2.

$ cat RHV-WGT-4.2-8/Manifest/iso_manifest.txt 
qemu-ga-msi-4.42-3
rhev-agent-msi-4.42-2
rhev-apt-4.42-1
rhev-balloon-msi-4.42-2
rhev-block-msi-4.42-2
rhev-network-msi-4.42-2
rhev-serial-msi-4.42-2
rhev-spice-agent-msi-4.42-1
rhev-spice-guest-msi-4.42-1
rhev-spice-wddm-guest-msi-4.42-1
rhev-sso-msi-4.42-1
rhev-viorng-msi-4.42-2
rhev-vioscsi-msi-4.42-2
rhev-wrapper-4.42-8

Looking in 4.3, I see the same problem. RHV-toolsSetup_4.3_8.iso ships with RHEV-Agent 4.3.3?
$ egrep 'rhev-wrapper|rhev-agent' RHV-WGT-4.3-8/Manifest/iso_manifest.txt 
rhev-agent-msi-4.43-3
rhev-wrapper-4.43-8

And back in RHV-toolsSetup_4.2_5.iso on another VM of mine I see RHEV-Agent 4.2.1....

See this one (4.2_9 ISO, from customer):
engine=> select app_list from vms where vm_name = 'vm.example.com';                                                                                                                          
RHEV-Serial64 4.2.3,
RHEV-Agent64 4.2.2,     <---- Agent is 4.2.2
RHEV-Tools 4.2.9,       <---- Tools is 4.2.9
RHEV-Network64 4.2.3,
RHEV-Spice-Agent64 4.2.1,
RHEV-SCSI64 4.2.3,
RHEV-RNG64 4.2.3,
RHEV-Block64 4.2.3,
RHEV-Balloon64 4.2.3,

Looks like the ISO version from filename seems to match RHEV-Tools version in app_list. But the engine looks at RHEV-Agent to determine if CD should be attached or not because that is what is set in vdc_options:

From vdc_options, engine=# select option_value from vdc_options where option_name = 'AgentAppName';
                                                option_value                                                 
-------------------------------------------------------------------------------------------------------------
 rhev-agent,rhevm-guest-agent,rhevm-guest-agent-common,RHEV-Agent,ovirt-guest-agent,ovirt-guest-agent-common


Shouldn't the engine look at RHEV-Tools instead of RHEV-Agent? Or should the ISO ship with matching version of Agent and ISO Version?

Version-Release number of selected component (if applicable):
4.2, 4.3

How reproducible:
Seen on 4.2.8 (customer) and 4.3.3 (labs)

Steps to Reproduce:
1. Have a windows VM, with tools installed and tools iso on ISO Domain, check disparity of ISO version from filename, RHEV-Agent and RHEV-Tools

Actual results:
Windows VM runs with ISO attached

Comment 1 Germano Veit Michel 2019-07-17 05:28:27 UTC
This is particularly annoying at this point, because we have a couple qemu bugs where if the NFS ISO domain becomes unavailable it can hang or even crash the VM.

https://bugzilla.redhat.com/show_bug.cgi?id=1603104
https://access.redhat.com/solutions/4057341
https://bugzilla.redhat.com/show_bug.cgi?id=1704990

Comment 4 Yedidyah Bar David 2019-07-22 07:22:30 UTC
Germano, I think I fully agree with your analysis, but wonder how come it's reported only now. Do you recall if this ever worked? I looked a bit and can't find anything that broke the behavior - so either it was broken forever or I do not understand something. For the record, the last time we changed AgentAppName was in bug 1460863 (4.2.0) / bug 1464339 (4.1.4).

Comment 5 Germano Veit Michel 2019-07-23 01:17:44 UTC
Hi Didi.

(In reply to Yedidyah Bar David from comment #4)
> Germano, I think I fully agree with your analysis, but wonder how come it's
> reported only now. Do you recall if this ever worked? I looked a bit and
> can't find anything that broke the behavior - so either it was broken
> forever or I do not understand something. 

Good question. If we look at much older versions of RHEV-Tools it seems to have worked in the past, because the RHEV-Agent version was in sync with the ISO version, see on 4.0.7:
$ egrep 'rhev-agent|rhev-wrapper' RHEV-WGT-4.0-7/Manifest/iso_manifest.txt 
rhev-agent-msi-4.40-7
rhev-wrapper-4.40-7

So ISO = 4.0.7 and Agent = 4.0.7. The current code would have worked.

I did not check all versions, maybe 4.0.7 was just lucky.

Thinking about it, I recall a few cases where customers just had a bunch of ISOs attached without knowing it. The problem has most likely been there for quite some time, but it only shows up in case there is a second problem (i.e. bug that kills/hangs VMs if ISO is down etc...), a problem like this is easy to fly under the radar as its harmless by itself.

Comment 6 Yedidyah Bar David 2019-07-24 07:04:53 UTC
(In reply to Germano Veit Michel from comment #5)
> Hi Didi.
> 
> (In reply to Yedidyah Bar David from comment #4)
> > Germano, I think I fully agree with your analysis, but wonder how come it's
> > reported only now. Do you recall if this ever worked? I looked a bit and
> > can't find anything that broke the behavior - so either it was broken
> > forever or I do not understand something. 
> 
> Good question. If we look at much older versions of RHEV-Tools it seems to
> have worked in the past, because the RHEV-Agent version was in sync with the
> ISO version, see on 4.0.7:
> $ egrep 'rhev-agent|rhev-wrapper' RHEV-WGT-4.0-7/Manifest/iso_manifest.txt 
> rhev-agent-msi-4.40-7
> rhev-wrapper-4.40-7
> 
> So ISO = 4.0.7 and Agent = 4.0.7. The current code would have worked.
> 
> I did not check all versions, maybe 4.0.7 was just lucky.
> 
> Thinking about it, I recall a few cases where customers just had a bunch of
> ISOs attached without knowing it. The problem has most likely been there for
> quite some time, but it only shows up in case there is a second problem
> (i.e. bug that kills/hangs VMs if ISO is down etc...), a problem like this
> is easy to fly under the radar as its harmless by itself.

Lev, does this make sense? Did we use to have identical version of ISO and of
Agent and got it broken at some point? Or is there something else I am missing?

In other words: Is it safe to change AgentAppName to be only 'RHEV-Tools'?

Thanks.

Comment 7 Yedidyah Bar David 2019-07-24 07:26:53 UTC
BTW, I assume that this applies only to WGT ISO installed on the engine machine, where engine-setup created a local NFS ISO domain. A local ISO domain is deprecated since 4.0, bug 1332813, and impossible to create via engine-setup since 4.2 (unless you try manually changing the answer file, which is not supported). If I got it right, current bug applies only to systems set up before that. If so, I suggest to recommend that such systems have the engine-local ISO domain removed. Makes sense?

Comment 8 Germano Veit Michel 2019-07-25 01:29:07 UTC
(In reply to Yedidyah Bar David from comment #7)
> BTW, I assume that this applies only to WGT ISO installed on the engine
> machine, where engine-setup created a local NFS ISO domain. A local ISO
> domain is deprecated since 4.0, bug 1332813, and impossible to create via
> engine-setup since 4.2 (unless you try manually changing the answer file,
> which is not supported). If I got it right, current bug applies only to
> systems set up before that. If so, I suggest to recommend that such systems
> have the engine-local ISO domain removed. Makes sense?

Hey Didi,

I'm not sure if I understand your question correctly...

The ISOs I used to reproduce were simply downloaded from the customer portal[1] and placed on a ISO Domain in a fresh RHV 4.3 setup. I don't think uploading the ISO to a normal ISO Domain, a Data Domain or having it in the engine-local ISO domain would make any difference in relation to the application/drivers versions it contains.

Did I miss your question? :)

[1] https://access.redhat.com/downloads/content/415/ver=4.3/rhel---7/4.3/x86_64/product-software

Comment 9 Yedidyah Bar David 2019-07-25 06:34:04 UTC
Sorry, Germano. Please ignore my comment 7, then :-(

Restoring needinfo on Lev for my question in comment 6.

Comment 10 Lev Veyde 2019-07-25 14:21:05 UTC
(In reply to Yedidyah Bar David from comment #6)
> (In reply to Germano Veit Michel from comment #5)
> > Hi Didi.
> > 
> > (In reply to Yedidyah Bar David from comment #4)
> > > Germano, I think I fully agree with your analysis, but wonder how come it's
> > > reported only now. Do you recall if this ever worked? I looked a bit and
> > > can't find anything that broke the behavior - so either it was broken
> > > forever or I do not understand something. 
> > 
> > Good question. If we look at much older versions of RHEV-Tools it seems to
> > have worked in the past, because the RHEV-Agent version was in sync with the
> > ISO version, see on 4.0.7:
> > $ egrep 'rhev-agent|rhev-wrapper' RHEV-WGT-4.0-7/Manifest/iso_manifest.txt 
> > rhev-agent-msi-4.40-7
> > rhev-wrapper-4.40-7
> > 
> > So ISO = 4.0.7 and Agent = 4.0.7. The current code would have worked.
> > 
> > I did not check all versions, maybe 4.0.7 was just lucky.
> > 
> > Thinking about it, I recall a few cases where customers just had a bunch of
> > ISOs attached without knowing it. The problem has most likely been there for
> > quite some time, but it only shows up in case there is a second problem
> > (i.e. bug that kills/hangs VMs if ISO is down etc...), a problem like this
> > is easy to fly under the radar as its harmless by itself.
> 
> Lev, does this make sense? Did we use to have identical version of ISO and of
> Agent and got it broken at some point? Or is there something else I am
> missing?
> 
> In other words: Is it safe to change AgentAppName to be only 'RHEV-Tools'?
> 
> Thanks.

No Didi, versions shouldn't match, and actually rarely do match.

I'm not sure why RHEV-Agent version is checked at all.

Comment 11 Germano Veit Michel 2019-07-25 22:22:42 UTC
(In reply to Lev Veyde from comment #10)
> No Didi, versions shouldn't match, and actually rarely do match.
> 
> I'm not sure why RHEV-Agent version is checked at all.

Hi Lev. And would it be correct to compare RHEV-Tools version with the ISO filename version string? Is it supposed to match?
I checked a few and it seems to be reliable.

Comment 12 Lev Veyde 2019-07-25 22:43:04 UTC
(In reply to Germano Veit Michel from comment #11)
> (In reply to Lev Veyde from comment #10)
> > No Didi, versions shouldn't match, and actually rarely do match.
> > 
> > I'm not sure why RHEV-Agent version is checked at all.
> 
> Hi Lev. And would it be correct to compare RHEV-Tools version with the ISO
> filename version string? Is it supposed to match?
> I checked a few and it seems to be reliable.

Hi Germano,

Yes, it should match.
Please note a small difference in the format (X.Y_Z vs X.Y.Z, i.e. RHV-toolsSetup_4.3_5.iso contains RHV Windows Guest Tools version 4.3.5).

Comment 13 Germano Veit Michel 2019-07-25 23:03:54 UTC
Thanks Lev!

So Didi, is comment #12 the solution on 4.3.z? 

I ask because we have a customer that the ISO domain goes down and it triggers qemu bugs on the VMs with the ISOs attached. Having this problem auto attaching ISOs to windows VMs is making things worse.

Comment 16 Yedidyah Bar David 2019-08-20 08:44:39 UTC
(In reply to Germano Veit Michel from comment #13)
> Thanks Lev!
> 
> So Didi, is comment #12 the solution on 4.3.z? 
> 
> I ask because we have a customer that the ISO domain goes down and it
> triggers qemu bugs on the VMs with the ISOs attached. Having this problem
> auto attaching ISOs to windows VMs is making things worse.

I think it is, but will let Gal look at this, as I am not sure.

Comment 17 Michal Skrivanek 2019-08-20 10:35:44 UTC
note that this is also being solved in bug 1722519

Comment 18 Yedidyah Bar David 2019-08-20 10:44:16 UTC
(In reply to Michal Skrivanek from comment #17)
> note that this is also being solved in bug 1722519

I think these are 3 different issues:

1. Current bug - versions of what to compare when deciding if to attach an iso

2. bug 1722519 - make this logic work also with data domains

3. No bug yet - Make this auto-attachment optional. IMO we should also do this one. As a user, I'd find it annoying that a system attaches iso images to my VMs when it thinks I might want it to, without me being about to prevent that.

Comment 19 Yedidyah Bar David 2019-08-20 10:45:45 UTC
(In reply to Yedidyah Bar David from comment #18)
> 3. No bug yet - Make this auto-attachment optional. IMO we should also do
> this one. As a user, I'd find it annoying that a system attaches iso images
> to my VMs when it thinks I might want it to, without me being about to
> prevent that.

That is, even without considering comment 1. With it, even more so.

Comment 20 Daniel Gur 2019-08-28 13:14:22 UTC
sync2jira

Comment 21 Daniel Gur 2019-08-28 13:19:24 UTC
sync2jira

Comment 23 Petr Matyáš 2019-10-31 16:26:14 UTC
What was really done in this bug?

I have ISO domain with newer WGT ISO than installed and I'm getting a warning that 'New guest tools are available' but no ISO is attached automatically.

So is this the only expected behaviour?

The ISO domain is regular ISO domain not local nor data domain.
And I have RHV-toolsSetup_4.2_9.iso installed and RHV-toolsSetup_4.3_10.iso available on ovirt-engine-4.3.7.0-0.1.el7.noarch.

Comment 24 Liran Rotenberg 2019-11-01 12:40:28 UTC
(In reply to Petr Matyáš from comment #23)
> What was really done in this bug?
> 
1. You will have auto-attaching of WGT ISO both from ISO domain and from data domain(NFS), can be uploaded via ovirt-imageio.
2. If you have the newest version available on the guest, the ISO won't be attached.
3. If there is a newer ISO, it will be auto-attached to the VM.

> I have ISO domain with newer WGT ISO than installed and I'm getting a
> warning that 'New guest tools are available' but no ISO is attached
> automatically.
> 
> So is this the only expected behaviour?
It is, the status bug is tracked in another BZ, BZ1720795.
> 
> The ISO domain is regular ISO domain not local nor data domain.
> And I have RHV-toolsSetup_4.2_9.iso installed and RHV-toolsSetup_4.3_10.iso
> available on ovirt-engine-4.3.7.0-0.1.el7.noarch.

Comment 25 Petr Matyáš 2019-11-01 12:56:12 UTC
(In reply to Liran Rotenberg from comment #24)
> (In reply to Petr Matyáš from comment #23)
> 3. If there is a newer ISO, it will be auto-attached to the VM.

> > I have ISO domain with newer WGT ISO than installed and I'm getting a
> > warning that 'New guest tools are available' but no ISO is attached
> > automatically.
> 
> > So is this the only expected behaviour?
> It is, the status bug is tracked in another BZ, BZ1720795.

I don't understand what you mean, is the auto-attaching done in this bug or in BZ1720795?
Because in no situation I tried the auto attaching is executed now.
I have 4.2 WGT installed and 4.3 WGT available and there is no auto attaching, all of these are on an ISO domain.

Comment 26 Liran Rotenberg 2019-11-01 17:35:21 UTC
(In reply to Petr Matyáš from comment #25)
> (In reply to Liran Rotenberg from comment #24)
> > (In reply to Petr Matyáš from comment #23)
> > 3. If there is a newer ISO, it will be auto-attached to the VM.
> 
> > > I have ISO domain with newer WGT ISO than installed and I'm getting a
> > > warning that 'New guest tools are available' but no ISO is attached
> > > automatically.
> > 
> > > So is this the only expected behaviour?
> > It is, the status bug is tracked in another BZ, BZ1720795.
> 
> I don't understand what you mean, is the auto-attaching done in this bug or
> in BZ1720795?
> Because in no situation I tried the auto attaching is executed now.
Auto attaching is done in this bug.
> I have 4.2 WGT installed and 4.3 WGT available and there is no auto
> attaching, all of these are on an ISO domain.

This is supose to work. Can you report what is the application version reported on the VM of WGT?
Also, what is the rhvm-setup-plugins version, and what is the GuestToolsSetupIsoPrefix variable in the engine's DB(in vdc_options table.. query: select * from vdc_options where option_name = 'GuestToolsSetupIsoPrefix';)?

Comment 27 Petr Matyáš 2019-11-04 08:28:43 UTC
Right now it reports RHEV-Tools 4.2.9 and RHEV-Agent64 4.2.2 and other apps of corresponding 4.2 version.

rhvm-setup-plugins-4.3.4-1.el7ev.noarch

engine=# select * from vdc_options where option_name = 'GuestToolsSetupIsoPrefix';
 option_id |       option_name        |  option_value   | version |  default_value
-----------+--------------------------+-----------------+---------+-----------------
       127 | GuestToolsSetupIsoPrefix | RHV-toolsSetup_ | general | RHV-toolsSetup_
(1 row)

Comment 28 Liran Rotenberg 2019-11-04 08:36:51 UTC
This is the wrong version of rhvm-setup-plugins, the fix is in 4.3.5.1.
The option value should be '.*rhe?v-toolssetup_'.

Sandro/Didi, any reason deploying engine doesn't consume the right version of rhvm-setup-plugins?

Comment 29 Petr Matyáš 2019-11-04 09:52:22 UTC
Actually we have it, although not in the build I was testing but in the next one (please don't move bugs to ON_QA too soon).

Verified on rhvm-setup-plugins-4.3.5-1.el7ev.noarch

If you have any older version of WGT ISO installed engine will automatically attach newest ISO from ISO domain.

There is no event for this auto attachment and also it is not visible in change cd dialogue but I will create a new bug for this.

Comment 33 errata-xmlrpc 2019-12-12 10:36:34 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4229


Note You need to log in before you can comment on or make changes to this bug.