Bug 1730538

This is particularly annoying at this point, because we have a couple qemu bugs where if the NFS ISO domain becomes unavailable it can hang or even crash the VM.

https://bugzilla.redhat.com/show_bug.cgi?id=1603104
https://access.redhat.com/solutions/4057341
https://bugzilla.redhat.com/show_bug.cgi?id=1704990

Germano, I think I fully agree with your analysis, but wonder how come it's reported only now. Do you recall if this ever worked? I looked a bit and can't find anything that broke the behavior - so either it was broken forever or I do not understand something. For the record, the last time we changed AgentAppName was in bug 1460863 (4.2.0) / bug 1464339 (4.1.4).

Hi Didi.

(In reply to Yedidyah Bar David from comment #4)
> Germano, I think I fully agree with your analysis, but wonder how come it's
> reported only now. Do you recall if this ever worked? I looked a bit and
> can't find anything that broke the behavior - so either it was broken
> forever or I do not understand something. 

Good question. If we look at much older versions of RHEV-Tools it seems to have worked in the past, because the RHEV-Agent version was in sync with the ISO version, see on 4.0.7:
$ egrep 'rhev-agent|rhev-wrapper' RHEV-WGT-4.0-7/Manifest/iso_manifest.txt 
rhev-agent-msi-4.40-7
rhev-wrapper-4.40-7

So ISO = 4.0.7 and Agent = 4.0.7. The current code would have worked.

I did not check all versions, maybe 4.0.7 was just lucky.

Thinking about it, I recall a few cases where customers just had a bunch of ISOs attached without knowing it. The problem has most likely been there for quite some time, but it only shows up in case there is a second problem (i.e. bug that kills/hangs VMs if ISO is down etc...), a problem like this is easy to fly under the radar as its harmless by itself.

(In reply to Germano Veit Michel from comment #5)
> Hi Didi.
> 
> (In reply to Yedidyah Bar David from comment #4)
> > Germano, I think I fully agree with your analysis, but wonder how come it's
> > reported only now. Do you recall if this ever worked? I looked a bit and
> > can't find anything that broke the behavior - so either it was broken
> > forever or I do not understand something. 
> 
> Good question. If we look at much older versions of RHEV-Tools it seems to
> have worked in the past, because the RHEV-Agent version was in sync with the
> ISO version, see on 4.0.7:
> $ egrep 'rhev-agent|rhev-wrapper' RHEV-WGT-4.0-7/Manifest/iso_manifest.txt 
> rhev-agent-msi-4.40-7
> rhev-wrapper-4.40-7
> 
> So ISO = 4.0.7 and Agent = 4.0.7. The current code would have worked.
> 
> I did not check all versions, maybe 4.0.7 was just lucky.
> 
> Thinking about it, I recall a few cases where customers just had a bunch of
> ISOs attached without knowing it. The problem has most likely been there for
> quite some time, but it only shows up in case there is a second problem
> (i.e. bug that kills/hangs VMs if ISO is down etc...), a problem like this
> is easy to fly under the radar as its harmless by itself.

Lev, does this make sense? Did we use to have identical version of ISO and of
Agent and got it broken at some point? Or is there something else I am missing?

In other words: Is it safe to change AgentAppName to be only 'RHEV-Tools'?

Thanks.

BTW, I assume that this applies only to WGT ISO installed on the engine machine, where engine-setup created a local NFS ISO domain. A local ISO domain is deprecated since 4.0, bug 1332813, and impossible to create via engine-setup since 4.2 (unless you try manually changing the answer file, which is not supported). If I got it right, current bug applies only to systems set up before that. If so, I suggest to recommend that such systems have the engine-local ISO domain removed. Makes sense?

(In reply to Yedidyah Bar David from comment #7)
> BTW, I assume that this applies only to WGT ISO installed on the engine
> machine, where engine-setup created a local NFS ISO domain. A local ISO
> domain is deprecated since 4.0, bug 1332813, and impossible to create via
> engine-setup since 4.2 (unless you try manually changing the answer file,
> which is not supported). If I got it right, current bug applies only to
> systems set up before that. If so, I suggest to recommend that such systems
> have the engine-local ISO domain removed. Makes sense?

Hey Didi,

I'm not sure if I understand your question correctly...

The ISOs I used to reproduce were simply downloaded from the customer portal[1] and placed on a ISO Domain in a fresh RHV 4.3 setup. I don't think uploading the ISO to a normal ISO Domain, a Data Domain or having it in the engine-local ISO domain would make any difference in relation to the application/drivers versions it contains.

Did I miss your question? :)

[1] https://access.redhat.com/downloads/content/415/ver=4.3/rhel---7/4.3/x86_64/product-software

Sorry, Germano. Please ignore my comment 7, then :-(

Restoring needinfo on Lev for my question in comment 6.

(In reply to Yedidyah Bar David from comment #6)
> (In reply to Germano Veit Michel from comment #5)
> > Hi Didi.
> > 
> > (In reply to Yedidyah Bar David from comment #4)
> > > Germano, I think I fully agree with your analysis, but wonder how come it's
> > > reported only now. Do you recall if this ever worked? I looked a bit and
> > > can't find anything that broke the behavior - so either it was broken
> > > forever or I do not understand something. 
> > 
> > Good question. If we look at much older versions of RHEV-Tools it seems to
> > have worked in the past, because the RHEV-Agent version was in sync with the
> > ISO version, see on 4.0.7:
> > $ egrep 'rhev-agent|rhev-wrapper' RHEV-WGT-4.0-7/Manifest/iso_manifest.txt 
> > rhev-agent-msi-4.40-7
> > rhev-wrapper-4.40-7
> > 
> > So ISO = 4.0.7 and Agent = 4.0.7. The current code would have worked.
> > 
> > I did not check all versions, maybe 4.0.7 was just lucky.
> > 
> > Thinking about it, I recall a few cases where customers just had a bunch of
> > ISOs attached without knowing it. The problem has most likely been there for
> > quite some time, but it only shows up in case there is a second problem
> > (i.e. bug that kills/hangs VMs if ISO is down etc...), a problem like this
> > is easy to fly under the radar as its harmless by itself.
> 
> Lev, does this make sense? Did we use to have identical version of ISO and of
> Agent and got it broken at some point? Or is there something else I am
> missing?
> 
> In other words: Is it safe to change AgentAppName to be only 'RHEV-Tools'?
> 
> Thanks.

No Didi, versions shouldn't match, and actually rarely do match.

I'm not sure why RHEV-Agent version is checked at all.

(In reply to Lev Veyde from comment #10)
> No Didi, versions shouldn't match, and actually rarely do match.
> 
> I'm not sure why RHEV-Agent version is checked at all.

Hi Lev. And would it be correct to compare RHEV-Tools version with the ISO filename version string? Is it supposed to match?
I checked a few and it seems to be reliable.

(In reply to Germano Veit Michel from comment #11)
> (In reply to Lev Veyde from comment #10)
> > No Didi, versions shouldn't match, and actually rarely do match.
> > 
> > I'm not sure why RHEV-Agent version is checked at all.
> 
> Hi Lev. And would it be correct to compare RHEV-Tools version with the ISO
> filename version string? Is it supposed to match?
> I checked a few and it seems to be reliable.

Hi Germano,

Yes, it should match.
Please note a small difference in the format (X.Y_Z vs X.Y.Z, i.e. RHV-toolsSetup_4.3_5.iso contains RHV Windows Guest Tools version 4.3.5).

Thanks Lev!

So Didi, is comment #12 the solution on 4.3.z? 

I ask because we have a customer that the ISO domain goes down and it triggers qemu bugs on the VMs with the ISOs attached. Having this problem auto attaching ISOs to windows VMs is making things worse.

(In reply to Germano Veit Michel from comment #13)
> Thanks Lev!
> 
> So Didi, is comment #12 the solution on 4.3.z? 
> 
> I ask because we have a customer that the ISO domain goes down and it
> triggers qemu bugs on the VMs with the ISOs attached. Having this problem
> auto attaching ISOs to windows VMs is making things worse.

I think it is, but will let Gal look at this, as I am not sure.

note that this is also being solved in bug 1722519

(In reply to Michal Skrivanek from comment #17)
> note that this is also being solved in bug 1722519

I think these are 3 different issues:

1. Current bug - versions of what to compare when deciding if to attach an iso

2. bug 1722519 - make this logic work also with data domains

3. No bug yet - Make this auto-attachment optional. IMO we should also do this one. As a user, I'd find it annoying that a system attaches iso images to my VMs when it thinks I might want it to, without me being about to prevent that.

(In reply to Yedidyah Bar David from comment #18)
> 3. No bug yet - Make this auto-attachment optional. IMO we should also do
> this one. As a user, I'd find it annoying that a system attaches iso images
> to my VMs when it thinks I might want it to, without me being about to
> prevent that.

That is, even without considering comment 1. With it, even more so.

sync2jira

sync2jira

What was really done in this bug?

I have ISO domain with newer WGT ISO than installed and I'm getting a warning that 'New guest tools are available' but no ISO is attached automatically.

So is this the only expected behaviour?

The ISO domain is regular ISO domain not local nor data domain.
And I have RHV-toolsSetup_4.2_9.iso installed and RHV-toolsSetup_4.3_10.iso available on ovirt-engine-4.3.7.0-0.1.el7.noarch.

(In reply to Petr Matyáš from comment #23)
> What was really done in this bug?
> 
1. You will have auto-attaching of WGT ISO both from ISO domain and from data domain(NFS), can be uploaded via ovirt-imageio.
2. If you have the newest version available on the guest, the ISO won't be attached.
3. If there is a newer ISO, it will be auto-attached to the VM.

> I have ISO domain with newer WGT ISO than installed and I'm getting a
> warning that 'New guest tools are available' but no ISO is attached
> automatically.
> 
> So is this the only expected behaviour?
It is, the status bug is tracked in another BZ, BZ1720795.
> 
> The ISO domain is regular ISO domain not local nor data domain.
> And I have RHV-toolsSetup_4.2_9.iso installed and RHV-toolsSetup_4.3_10.iso
> available on ovirt-engine-4.3.7.0-0.1.el7.noarch.

(In reply to Liran Rotenberg from comment #24)
> (In reply to Petr Matyáš from comment #23)
> 3. If there is a newer ISO, it will be auto-attached to the VM.

> > I have ISO domain with newer WGT ISO than installed and I'm getting a
> > warning that 'New guest tools are available' but no ISO is attached
> > automatically.
> 
> > So is this the only expected behaviour?
> It is, the status bug is tracked in another BZ, BZ1720795.

I don't understand what you mean, is the auto-attaching done in this bug or in BZ1720795?
Because in no situation I tried the auto attaching is executed now.
I have 4.2 WGT installed and 4.3 WGT available and there is no auto attaching, all of these are on an ISO domain.

(In reply to Petr Matyáš from comment #25)
> (In reply to Liran Rotenberg from comment #24)
> > (In reply to Petr Matyáš from comment #23)
> > 3. If there is a newer ISO, it will be auto-attached to the VM.
> 
> > > I have ISO domain with newer WGT ISO than installed and I'm getting a
> > > warning that 'New guest tools are available' but no ISO is attached
> > > automatically.
> > 
> > > So is this the only expected behaviour?
> > It is, the status bug is tracked in another BZ, BZ1720795.
> 
> I don't understand what you mean, is the auto-attaching done in this bug or
> in BZ1720795?
> Because in no situation I tried the auto attaching is executed now.
Auto attaching is done in this bug.
> I have 4.2 WGT installed and 4.3 WGT available and there is no auto
> attaching, all of these are on an ISO domain.

This is supose to work. Can you report what is the application version reported on the VM of WGT?
Also, what is the rhvm-setup-plugins version, and what is the GuestToolsSetupIsoPrefix variable in the engine's DB(in vdc_options table.. query: select * from vdc_options where option_name = 'GuestToolsSetupIsoPrefix';)?

Right now it reports RHEV-Tools 4.2.9 and RHEV-Agent64 4.2.2 and other apps of corresponding 4.2 version.

rhvm-setup-plugins-4.3.4-1.el7ev.noarch

engine=# select * from vdc_options where option_name = 'GuestToolsSetupIsoPrefix';
 option_id |       option_name        |  option_value   | version |  default_value
-----------+--------------------------+-----------------+---------+-----------------
       127 | GuestToolsSetupIsoPrefix | RHV-toolsSetup_ | general | RHV-toolsSetup_
(1 row)

This is the wrong version of rhvm-setup-plugins, the fix is in 4.3.5.1.
The option value should be '.*rhe?v-toolssetup_'.

Sandro/Didi, any reason deploying engine doesn't consume the right version of rhvm-setup-plugins?

Actually we have it, although not in the build I was testing but in the next one (please don't move bugs to ON_QA too soon).

Verified on rhvm-setup-plugins-4.3.5-1.el7ev.noarch

If you have any older version of WGT ISO installed engine will automatically attach newest ISO from ISO domain.

There is no event for this auto attachment and also it is not visible in change cd dialogue but I will create a new bug for this.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:4229