Hide Forgot
Description of problem: Something seems to be off here, let me explain... The logic in the engine is to automatically attach the Guest Tools ISO to Windows VMs that have outdated tools, or need it installed. This logic compares 2 things to decide if the tools need to be updated: 1) the ISO version, i.e. 4.2.9 (extracted from the iso filename, RHV-toolsSetup_4.2_9.iso) 2) RHEV-Agent version from the vm app_list, this comes from a list of valid substrings in vdc_option AgentAppName list. So basically RHV-toolsSetup_X.Y.Z.iso vs RHEV-Agent X.Y.Z. Problem is the X.Y.Z version in the ISO does not match the X.Y.Z version of Agent the exact same ISO ships(?!), so the validation is wrong, as if the VM needs updating the Tools. Take RHV-toolsSetup_4.2_8.iso, it ships with RHEV-Agent 4.2.2. This will always make the engine think the VM needs upgrading, because 4.2.8 > 4.2.2. $ cat RHV-WGT-4.2-8/Manifest/iso_manifest.txt qemu-ga-msi-4.42-3 rhev-agent-msi-4.42-2 rhev-apt-4.42-1 rhev-balloon-msi-4.42-2 rhev-block-msi-4.42-2 rhev-network-msi-4.42-2 rhev-serial-msi-4.42-2 rhev-spice-agent-msi-4.42-1 rhev-spice-guest-msi-4.42-1 rhev-spice-wddm-guest-msi-4.42-1 rhev-sso-msi-4.42-1 rhev-viorng-msi-4.42-2 rhev-vioscsi-msi-4.42-2 rhev-wrapper-4.42-8 Looking in 4.3, I see the same problem. RHV-toolsSetup_4.3_8.iso ships with RHEV-Agent 4.3.3? $ egrep 'rhev-wrapper|rhev-agent' RHV-WGT-4.3-8/Manifest/iso_manifest.txt rhev-agent-msi-4.43-3 rhev-wrapper-4.43-8 And back in RHV-toolsSetup_4.2_5.iso on another VM of mine I see RHEV-Agent 4.2.1.... See this one (4.2_9 ISO, from customer): engine=> select app_list from vms where vm_name = 'vm.example.com'; RHEV-Serial64 4.2.3, RHEV-Agent64 4.2.2, <---- Agent is 4.2.2 RHEV-Tools 4.2.9, <---- Tools is 4.2.9 RHEV-Network64 4.2.3, RHEV-Spice-Agent64 4.2.1, RHEV-SCSI64 4.2.3, RHEV-RNG64 4.2.3, RHEV-Block64 4.2.3, RHEV-Balloon64 4.2.3, Looks like the ISO version from filename seems to match RHEV-Tools version in app_list. But the engine looks at RHEV-Agent to determine if CD should be attached or not because that is what is set in vdc_options: From vdc_options, engine=# select option_value from vdc_options where option_name = 'AgentAppName'; option_value ------------------------------------------------------------------------------------------------------------- rhev-agent,rhevm-guest-agent,rhevm-guest-agent-common,RHEV-Agent,ovirt-guest-agent,ovirt-guest-agent-common Shouldn't the engine look at RHEV-Tools instead of RHEV-Agent? Or should the ISO ship with matching version of Agent and ISO Version? Version-Release number of selected component (if applicable): 4.2, 4.3 How reproducible: Seen on 4.2.8 (customer) and 4.3.3 (labs) Steps to Reproduce: 1. Have a windows VM, with tools installed and tools iso on ISO Domain, check disparity of ISO version from filename, RHEV-Agent and RHEV-Tools Actual results: Windows VM runs with ISO attached
This is particularly annoying at this point, because we have a couple qemu bugs where if the NFS ISO domain becomes unavailable it can hang or even crash the VM. https://bugzilla.redhat.com/show_bug.cgi?id=1603104 https://access.redhat.com/solutions/4057341 https://bugzilla.redhat.com/show_bug.cgi?id=1704990
Germano, I think I fully agree with your analysis, but wonder how come it's reported only now. Do you recall if this ever worked? I looked a bit and can't find anything that broke the behavior - so either it was broken forever or I do not understand something. For the record, the last time we changed AgentAppName was in bug 1460863 (4.2.0) / bug 1464339 (4.1.4).
Hi Didi. (In reply to Yedidyah Bar David from comment #4) > Germano, I think I fully agree with your analysis, but wonder how come it's > reported only now. Do you recall if this ever worked? I looked a bit and > can't find anything that broke the behavior - so either it was broken > forever or I do not understand something. Good question. If we look at much older versions of RHEV-Tools it seems to have worked in the past, because the RHEV-Agent version was in sync with the ISO version, see on 4.0.7: $ egrep 'rhev-agent|rhev-wrapper' RHEV-WGT-4.0-7/Manifest/iso_manifest.txt rhev-agent-msi-4.40-7 rhev-wrapper-4.40-7 So ISO = 4.0.7 and Agent = 4.0.7. The current code would have worked. I did not check all versions, maybe 4.0.7 was just lucky. Thinking about it, I recall a few cases where customers just had a bunch of ISOs attached without knowing it. The problem has most likely been there for quite some time, but it only shows up in case there is a second problem (i.e. bug that kills/hangs VMs if ISO is down etc...), a problem like this is easy to fly under the radar as its harmless by itself.
(In reply to Germano Veit Michel from comment #5) > Hi Didi. > > (In reply to Yedidyah Bar David from comment #4) > > Germano, I think I fully agree with your analysis, but wonder how come it's > > reported only now. Do you recall if this ever worked? I looked a bit and > > can't find anything that broke the behavior - so either it was broken > > forever or I do not understand something. > > Good question. If we look at much older versions of RHEV-Tools it seems to > have worked in the past, because the RHEV-Agent version was in sync with the > ISO version, see on 4.0.7: > $ egrep 'rhev-agent|rhev-wrapper' RHEV-WGT-4.0-7/Manifest/iso_manifest.txt > rhev-agent-msi-4.40-7 > rhev-wrapper-4.40-7 > > So ISO = 4.0.7 and Agent = 4.0.7. The current code would have worked. > > I did not check all versions, maybe 4.0.7 was just lucky. > > Thinking about it, I recall a few cases where customers just had a bunch of > ISOs attached without knowing it. The problem has most likely been there for > quite some time, but it only shows up in case there is a second problem > (i.e. bug that kills/hangs VMs if ISO is down etc...), a problem like this > is easy to fly under the radar as its harmless by itself. Lev, does this make sense? Did we use to have identical version of ISO and of Agent and got it broken at some point? Or is there something else I am missing? In other words: Is it safe to change AgentAppName to be only 'RHEV-Tools'? Thanks.
BTW, I assume that this applies only to WGT ISO installed on the engine machine, where engine-setup created a local NFS ISO domain. A local ISO domain is deprecated since 4.0, bug 1332813, and impossible to create via engine-setup since 4.2 (unless you try manually changing the answer file, which is not supported). If I got it right, current bug applies only to systems set up before that. If so, I suggest to recommend that such systems have the engine-local ISO domain removed. Makes sense?
(In reply to Yedidyah Bar David from comment #7) > BTW, I assume that this applies only to WGT ISO installed on the engine > machine, where engine-setup created a local NFS ISO domain. A local ISO > domain is deprecated since 4.0, bug 1332813, and impossible to create via > engine-setup since 4.2 (unless you try manually changing the answer file, > which is not supported). If I got it right, current bug applies only to > systems set up before that. If so, I suggest to recommend that such systems > have the engine-local ISO domain removed. Makes sense? Hey Didi, I'm not sure if I understand your question correctly... The ISOs I used to reproduce were simply downloaded from the customer portal[1] and placed on a ISO Domain in a fresh RHV 4.3 setup. I don't think uploading the ISO to a normal ISO Domain, a Data Domain or having it in the engine-local ISO domain would make any difference in relation to the application/drivers versions it contains. Did I miss your question? :) [1] https://access.redhat.com/downloads/content/415/ver=4.3/rhel---7/4.3/x86_64/product-software
Sorry, Germano. Please ignore my comment 7, then :-( Restoring needinfo on Lev for my question in comment 6.
(In reply to Yedidyah Bar David from comment #6) > (In reply to Germano Veit Michel from comment #5) > > Hi Didi. > > > > (In reply to Yedidyah Bar David from comment #4) > > > Germano, I think I fully agree with your analysis, but wonder how come it's > > > reported only now. Do you recall if this ever worked? I looked a bit and > > > can't find anything that broke the behavior - so either it was broken > > > forever or I do not understand something. > > > > Good question. If we look at much older versions of RHEV-Tools it seems to > > have worked in the past, because the RHEV-Agent version was in sync with the > > ISO version, see on 4.0.7: > > $ egrep 'rhev-agent|rhev-wrapper' RHEV-WGT-4.0-7/Manifest/iso_manifest.txt > > rhev-agent-msi-4.40-7 > > rhev-wrapper-4.40-7 > > > > So ISO = 4.0.7 and Agent = 4.0.7. The current code would have worked. > > > > I did not check all versions, maybe 4.0.7 was just lucky. > > > > Thinking about it, I recall a few cases where customers just had a bunch of > > ISOs attached without knowing it. The problem has most likely been there for > > quite some time, but it only shows up in case there is a second problem > > (i.e. bug that kills/hangs VMs if ISO is down etc...), a problem like this > > is easy to fly under the radar as its harmless by itself. > > Lev, does this make sense? Did we use to have identical version of ISO and of > Agent and got it broken at some point? Or is there something else I am > missing? > > In other words: Is it safe to change AgentAppName to be only 'RHEV-Tools'? > > Thanks. No Didi, versions shouldn't match, and actually rarely do match. I'm not sure why RHEV-Agent version is checked at all.
(In reply to Lev Veyde from comment #10) > No Didi, versions shouldn't match, and actually rarely do match. > > I'm not sure why RHEV-Agent version is checked at all. Hi Lev. And would it be correct to compare RHEV-Tools version with the ISO filename version string? Is it supposed to match? I checked a few and it seems to be reliable.
(In reply to Germano Veit Michel from comment #11) > (In reply to Lev Veyde from comment #10) > > No Didi, versions shouldn't match, and actually rarely do match. > > > > I'm not sure why RHEV-Agent version is checked at all. > > Hi Lev. And would it be correct to compare RHEV-Tools version with the ISO > filename version string? Is it supposed to match? > I checked a few and it seems to be reliable. Hi Germano, Yes, it should match. Please note a small difference in the format (X.Y_Z vs X.Y.Z, i.e. RHV-toolsSetup_4.3_5.iso contains RHV Windows Guest Tools version 4.3.5).
Thanks Lev! So Didi, is comment #12 the solution on 4.3.z? I ask because we have a customer that the ISO domain goes down and it triggers qemu bugs on the VMs with the ISOs attached. Having this problem auto attaching ISOs to windows VMs is making things worse.
(In reply to Germano Veit Michel from comment #13) > Thanks Lev! > > So Didi, is comment #12 the solution on 4.3.z? > > I ask because we have a customer that the ISO domain goes down and it > triggers qemu bugs on the VMs with the ISOs attached. Having this problem > auto attaching ISOs to windows VMs is making things worse. I think it is, but will let Gal look at this, as I am not sure.
note that this is also being solved in bug 1722519
(In reply to Michal Skrivanek from comment #17) > note that this is also being solved in bug 1722519 I think these are 3 different issues: 1. Current bug - versions of what to compare when deciding if to attach an iso 2. bug 1722519 - make this logic work also with data domains 3. No bug yet - Make this auto-attachment optional. IMO we should also do this one. As a user, I'd find it annoying that a system attaches iso images to my VMs when it thinks I might want it to, without me being about to prevent that.
(In reply to Yedidyah Bar David from comment #18) > 3. No bug yet - Make this auto-attachment optional. IMO we should also do > this one. As a user, I'd find it annoying that a system attaches iso images > to my VMs when it thinks I might want it to, without me being about to > prevent that. That is, even without considering comment 1. With it, even more so.
sync2jira
What was really done in this bug? I have ISO domain with newer WGT ISO than installed and I'm getting a warning that 'New guest tools are available' but no ISO is attached automatically. So is this the only expected behaviour? The ISO domain is regular ISO domain not local nor data domain. And I have RHV-toolsSetup_4.2_9.iso installed and RHV-toolsSetup_4.3_10.iso available on ovirt-engine-4.3.7.0-0.1.el7.noarch.
(In reply to Petr Matyáš from comment #23) > What was really done in this bug? > 1. You will have auto-attaching of WGT ISO both from ISO domain and from data domain(NFS), can be uploaded via ovirt-imageio. 2. If you have the newest version available on the guest, the ISO won't be attached. 3. If there is a newer ISO, it will be auto-attached to the VM. > I have ISO domain with newer WGT ISO than installed and I'm getting a > warning that 'New guest tools are available' but no ISO is attached > automatically. > > So is this the only expected behaviour? It is, the status bug is tracked in another BZ, BZ1720795. > > The ISO domain is regular ISO domain not local nor data domain. > And I have RHV-toolsSetup_4.2_9.iso installed and RHV-toolsSetup_4.3_10.iso > available on ovirt-engine-4.3.7.0-0.1.el7.noarch.
(In reply to Liran Rotenberg from comment #24) > (In reply to Petr Matyáš from comment #23) > 3. If there is a newer ISO, it will be auto-attached to the VM. > > I have ISO domain with newer WGT ISO than installed and I'm getting a > > warning that 'New guest tools are available' but no ISO is attached > > automatically. > > > So is this the only expected behaviour? > It is, the status bug is tracked in another BZ, BZ1720795. I don't understand what you mean, is the auto-attaching done in this bug or in BZ1720795? Because in no situation I tried the auto attaching is executed now. I have 4.2 WGT installed and 4.3 WGT available and there is no auto attaching, all of these are on an ISO domain.
(In reply to Petr Matyáš from comment #25) > (In reply to Liran Rotenberg from comment #24) > > (In reply to Petr Matyáš from comment #23) > > 3. If there is a newer ISO, it will be auto-attached to the VM. > > > > I have ISO domain with newer WGT ISO than installed and I'm getting a > > > warning that 'New guest tools are available' but no ISO is attached > > > automatically. > > > > > So is this the only expected behaviour? > > It is, the status bug is tracked in another BZ, BZ1720795. > > I don't understand what you mean, is the auto-attaching done in this bug or > in BZ1720795? > Because in no situation I tried the auto attaching is executed now. Auto attaching is done in this bug. > I have 4.2 WGT installed and 4.3 WGT available and there is no auto > attaching, all of these are on an ISO domain. This is supose to work. Can you report what is the application version reported on the VM of WGT? Also, what is the rhvm-setup-plugins version, and what is the GuestToolsSetupIsoPrefix variable in the engine's DB(in vdc_options table.. query: select * from vdc_options where option_name = 'GuestToolsSetupIsoPrefix';)?
Right now it reports RHEV-Tools 4.2.9 and RHEV-Agent64 4.2.2 and other apps of corresponding 4.2 version. rhvm-setup-plugins-4.3.4-1.el7ev.noarch engine=# select * from vdc_options where option_name = 'GuestToolsSetupIsoPrefix'; option_id | option_name | option_value | version | default_value -----------+--------------------------+-----------------+---------+----------------- 127 | GuestToolsSetupIsoPrefix | RHV-toolsSetup_ | general | RHV-toolsSetup_ (1 row)
This is the wrong version of rhvm-setup-plugins, the fix is in 4.3.5.1. The option value should be '.*rhe?v-toolssetup_'. Sandro/Didi, any reason deploying engine doesn't consume the right version of rhvm-setup-plugins?
Actually we have it, although not in the build I was testing but in the next one (please don't move bugs to ON_QA too soon). Verified on rhvm-setup-plugins-4.3.5-1.el7ev.noarch If you have any older version of WGT ISO installed engine will automatically attach newest ISO from ISO domain. There is no event for this auto attachment and also it is not visible in change cd dialogue but I will create a new bug for this.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:4229