Tomas, isn't this a result of your integration with qemu guest agent? Can you please filter-out the local addresses (they are uninteresting to the end user)?
Any idea why 15.0.0.2 is being reported?

This bug report has Keywords: Regression or TestBlocker.
Since no regressions or test blockers are allowed between releases, it is also being identified as a blocker for this release. Please resolve ASAP.

(In reply to Dan Kenigsberg from comment #1)
> Tomas, isn't this a result of your integration with qemu guest agent? Can
> you please filter-out the local addresses (they are uninteresting to the end
> user)?
> Any idea why 15.0.0.2 is being reported?

Dan, 15.0.0.2 is my ovn IP and it's OK, but it has ::1 in the end which is wrong!
The same will be 10.35.128.0::1

Yes, this is probably the result of QEMU-GA integration. But lets think of it as a feature. I am not entirely convinced this should be filtered on VDSM level, nor am I convinced this is always uninteresting to the end user. The fact that we don't have a suitable UI for such information today does not mean it will always stay that way. If engine does not want to show localhost/link local addresses etc. in IP column why don't filter it on engine?

Do you run ovirt-guest-agent then?

(In reply to Michal Skrivanek from comment #5)
> Do you run ovirt-guest-agent then?

Of course i do, if i wasn't running, i couldn't see this IP's on my engine UI  - 
ovirt-guest-agent-common-1.0.14-3.el7ev.noarch

(In reply to Michael Burman from comment #6)
> (In reply to Michal Skrivanek from comment #5)
> > Do you run ovirt-guest-agent then?
> 
> Of course i do, if i wasn't running, i couldn't see this IP's on my engine
> UI  - 
> ovirt-guest-agent-common-1.0.14-3.el7ev.noarch

Actually, as of the recent changes by Tomas, if ovirt-guest-agent is off, Vdsm collects the addresses from qemu-guest-agent. So you are expected to see IPs even if ovirt-guest-agent is off.

The bug here is that qemu-guest-agent reports boring local addresses, as well as the odd-looking 15.0.0.2::1.

Tomas, what is this 15.0.0.2::1 address, and why is it reported by qemu-guest-agent?

(In reply to Dan Kenigsberg from comment #7)
> Tomas, what is this 15.0.0.2::1 address, and why is it reported by
> qemu-guest-agent?

Looking at the video there's no "15.0.0.2::1" address. There is "15.0.0.2" and "::1" which is probably all correct. You see both IPv4 and IPv6 addresses, and AFAIK there is no localhost filtering in qemu-ga like we have in ovirt-ga.

Michael, please attach output of "ip" and ovirt-guest-agent logs

(In reply to Michal Skrivanek from comment #8)
> (In reply to Dan Kenigsberg from comment #7)
> > Tomas, what is this 15.0.0.2::1 address, and why is it reported by
> > qemu-guest-agent?
> 
> Looking at the video there's no "15.0.0.2::1" address. There is "15.0.0.2"
> and "::1" which is probably all correct. You see both IPv4 and IPv6
> addresses, and AFAIK there is no localhost filtering in qemu-ga like we have
> in ovirt-ga.
> 
> Michael, please attach output of "ip" and ovirt-guest-agent logs

or rather vdsm. ovirt-ga data have precedence over qemu-ga, but it is polled differently so it may happen you see qemu-ga data temporarily.

(In reply to Michal Skrivanek from comment #8)
> (In reply to Dan Kenigsberg from comment #7)
> > Tomas, what is this 15.0.0.2::1 address, and why is it reported by
> > qemu-guest-agent?
> 
> Looking at the video there's no "15.0.0.2::1" address. There is "15.0.0.2"
> and "::1" which is probably all correct. You see both IPv4 and IPv6
> addresses, and AFAIK there is no localhost filtering in qemu-ga like we have
> in ovirt-ga.
> 
> Michael, please attach output of "ip" and ovirt-guest-agent logs

In the video there is 15:0:0:2::1 indeed and not "15.0.0.2" and "::1
this is not correct, ::1 added to all IPv4 addresses. This is sown in the video as well.

Also affecting our automation

(In reply to Michael Burman from comment #10)

> In the video there is 15:0:0:2::1 indeed and not "15.0.0.2" and "::1
> this is not correct, ::1 added to all IPv4 addresses. This is sown in the
> video as well.

no, nothing is "added" to anything. There's just "::1" as a localhost IPv6 address. Are you looking at a different video perhaps?:)

(In reply to Raz Tamir from comment #11)
> Also affecting our automation

hm, how exactly? IIUC the report (and the video) the ovirt-guest-agent report takes over once it properly starts. So if you're checking certain strings to be there you still get them, and in fact at the same time as before. It's just that sooner than that you can see qemu-ga report which includes all interfaces potentially different from what ovirt-ga reports.
If you have tests checking for "IP address not being reported when ovirt-guest-agent is not running" then they need to be fixed instead.

(In reply to Michal Skrivanek from comment #13)
> (In reply to Raz Tamir from comment #11)
> > Also affecting our automation
> 
> hm, how exactly? IIUC the report (and the video) the ovirt-guest-agent
> report takes over once it properly starts. So if you're checking certain
> strings to be there you still get them, and in fact at the same time as
> before. It's just that sooner than that you can see qemu-ga report which
> includes all interfaces potentially different from what ovirt-ga reports.
> If you have tests checking for "IP address not being reported when
> ovirt-guest-agent is not running" then they need to be fixed instead.

I think we should remove the report of IPs from qemu-guest-agent (in VDSM?) until this issue is fully and successfully resolved.

(In reply to Yaniv Kaul from comment #14)

> I think we should remove the report of IPs from qemu-guest-agent (in VDSM?)
> until this issue is fully and successfully resolved.

why? what's wrong with reporting all addresses?

(In reply to Michal Skrivanek from comment #15)
> (In reply to Yaniv Kaul from comment #14)
> 
> > I think we should remove the report of IPs from qemu-guest-agent (in VDSM?)
> > until this issue is fully and successfully resolved.
> 
> why? what's wrong with reporting all addresses?

It's buggy. There's no need to report 127.0.0.1, for example.
It's buggy - Engine can't handle the % thing (Windows only).

The whole feature of moving from o-g-a to q-g-a was not properly done - it was not in the scope initially for 4.2, the design was not there, other teams were not involved, etc.
Let's avoid such features next time.
I believe it also landed somewhat late.

(In reply to Michal Skrivanek from comment #12)
> (In reply to Michael Burman from comment #10)
> 
> > In the video there is 15:0:0:2::1 indeed and not "15.0.0.2" and "::1
> > this is not correct, ::1 added to all IPv4 addresses. This is sown in the
> > video as well.
> 
> no, nothing is "added" to anything. There's just "::1" as a localhost IPv6
> address. Are you looking at a different video perhaps?:)

I understand it's the IPv6 local net, but in the UI it's look part of the IPv4 address that's all.

(In reply to Yaniv Kaul from comment #16)
> (In reply to Michal Skrivanek from comment #15)
> > (In reply to Yaniv Kaul from comment #14)
> > 
> > > I think we should remove the report of IPs from qemu-guest-agent (in VDSM?)
> > > until this issue is fully and successfully resolved.
> > 
> > why? what's wrong with reporting all addresses?
> 
> It's buggy. There's no need to report 127.0.0.1, for example.

but that's not incorrect. That is a valid interface with a a valid address

> It's buggy - Engine can't handle the % thing (Windows only).

it's not windows only, it's a valid address and indeed the bug is on engine side that the database field designed for IP addresses can not hold a valid IPv6 address.


> The whole feature of moving from o-g-a to q-g-a was not properly done - it
> was not in the scope initially for 4.2, the design was not there, other
> teams were not involved, etc.
> Let's avoid such features next time.
> I believe it also landed somewhat late.

I disagree on all above

(In reply to Michal Skrivanek from comment #13)
> (In reply to Raz Tamir from comment #11)
> > Also affecting our automation
> 
> hm, how exactly? IIUC the report (and the video) the ovirt-guest-agent
> report takes over once it properly starts. So if you're checking certain
> strings to be there you still get them, and in fact at the same time as
> before. It's just that sooner than that you can see qemu-ga report which
> includes all interfaces potentially different from what ovirt-ga reports.
> If you have tests checking for "IP address not being reported when
> ovirt-guest-agent is not running" then they need to be fixed instead.

When we execute vdsm-client VM.getStats we get 

'netIfaces': [{'name': 'lo', 'hw': '00:00:00:00:00:00', 'inet': ['127.0.0.1'], 'inet6': ['::1']}, {'name': 'eth0', 'hw': '00:1a:4a:16:88:1d', 'inet': ['10.46.17.29'], 'inet6': ['2620:52:0:2e10:21a:4aff:fe16:881d', 'fe80::21a:4aff:fe16:881d']}]

It is a quick fix in our code to exclude the 127.* but just want to make sure you are aware that this issue is not a UI only as stated in the summary

either way, disabling is not difficult. It can be done either globally (in vdsm.conf for the whole qemu-guest-agent reporting) or only the network piece can be disabled in code

(In reply to Raz Tamir from comment #19)
> (In reply to Michal Skrivanek from comment #13)
> > (In reply to Raz Tamir from comment #11)
> > > Also affecting our automation
> > 
> > hm, how exactly? IIUC the report (and the video) the ovirt-guest-agent
> > report takes over once it properly starts. So if you're checking certain
> > strings to be there you still get them, and in fact at the same time as
> > before. It's just that sooner than that you can see qemu-ga report which
> > includes all interfaces potentially different from what ovirt-ga reports.
> > If you have tests checking for "IP address not being reported when
> > ovirt-guest-agent is not running" then they need to be fixed instead.
> 
> When we execute vdsm-client VM.getStats we get 
> 
> 'netIfaces': [{'name': 'lo', 'hw': '00:00:00:00:00:00', 'inet':
> ['127.0.0.1'], 'inet6': ['::1']}, {'name': 'eth0', 'hw':
> '00:1a:4a:16:88:1d', 'inet': ['10.46.17.29'], 'inet6':
> ['2620:52:0:2e10:21a:4aff:fe16:881d', 'fe80::21a:4aff:fe16:881d']}]

right. So that's a complete list which looks alright to me. But once ovirt-guest-agent sends its report you should get the same one
Looking at one of the VMs from the video I see:
"netIfaces": [{"name": "eth0","inet6": ["fe80::200:ff:fe00:20"],"inet": ["10.35.130.57"],"hw": "00:00:00:00:00:20"}]


> It is a quick fix in our code to exclude the 127.* but just want to make
> sure you are aware that this issue is not a UI only as stated in the summary

sure, it's not UI-only. It also somewhat breaks the feature to allow blacklists (bug 1437145) which was implemented as a quick fix just in ovirt-guest-agent, So that feature only works depending on which agent is used

(In reply to Michal Skrivanek from comment #20)
I also believe a proper fix is not that hard either. Chopping off "%" in DAO sounds like a trivial short term fix until we fix it properly in database and report/show complete address.
Since we always filtered loopback in ovirt-guest-agent filtering out ::1 and 127.0.0.1 in engine sounds good enough to me as well, if that's the price I have to pay for showing guest IP information without additional agent

From my perspective:
{'name': 'lo', 'hw': '00:00:00:00:00:00', 'inet': ['127.0.0.1'], 'inet6': ['::1']}
- There is no need to collect or report the loopback device. No one cares about it, at all. It's a qemu-guest-agent and/or ovirt-guest-agent bug.

(In reply to Michal Skrivanek from comment #18)

> > I believe it also landed somewhat late.
> 
> I disagree on all above

no, let me take that back. The NIC address reporting itself landed very late indeed. That was done under the assumption that ovirt-guest-agent report supersedes it anyway. Which it does.

(In reply to Yaniv Kaul from comment #23)
> From my perspective:
> {'name': 'lo', 'hw': '00:00:00:00:00:00', 'inet': ['127.0.0.1'], 'inet6':
> ['::1']}
> - There is no need to collect or report the loopback device. No one cares
> about it, at all. It's a qemu-guest-agent and/or ovirt-guest-agent bug.

I think so too. Same as we should have the same interface blacklisting facility as implemented in ovirt-guest-agent for e.g. docker0 interface. I believe that should happen inside the guest to be able to control that based on specific OS

(In reply to Michal Skrivanek from comment #25)
> (In reply to Yaniv Kaul from comment #23)
> > From my perspective:
> > {'name': 'lo', 'hw': '00:00:00:00:00:00', 'inet': ['127.0.0.1'], 'inet6':
> > ['::1']}
> > - There is no need to collect or report the loopback device. No one cares
> > about it, at all. It's a qemu-guest-agent and/or ovirt-guest-agent bug.
> 
> I think so too. Same as we should have the same interface blacklisting
> facility as implemented in ovirt-guest-agent for e.g. docker0 interface. I
> believe that should happen inside the guest to be able to control that based
> on specific OS

Probably need to add something like:
if (ifa->ifa_flags & (IFF_LOOPBACK))
        continue;

To https://github.com/qemu/qemu/blob/master/qga/commands-posix.c#L1722

(In reply to Yaniv Kaul from comment #26)
> (In reply to Michal Skrivanek from comment #25)
> > (In reply to Yaniv Kaul from comment #23)
> > > From my perspective:
> > > {'name': 'lo', 'hw': '00:00:00:00:00:00', 'inet': ['127.0.0.1'], 'inet6':
> > > ['::1']}
> > > - There is no need to collect or report the loopback device. No one cares
> > > about it, at all. It's a qemu-guest-agent and/or ovirt-guest-agent bug.
> > 
> > I think so too. Same as we should have the same interface blacklisting
> > facility as implemented in ovirt-guest-agent for e.g. docker0 interface. I
> > believe that should happen inside the guest to be able to control that based
> > on specific OS
> 
> Probably need to add something like:
> if (ifa->ifa_flags & (IFF_LOOPBACK))
>         continue;
> 
> To https://github.com/qemu/qemu/blob/master/qga/commands-posix.c#L1722

Uhm, hold on guys. I think we're assuming too much here. The fact that such information is of no use to oVirt engine does not mean it is uninteresting to everyone else. We're not talking about *oVirt* guest agent anymore.

If we don't care about the info then let's filter it out in engine.

(In reply to Tomáš Golembiovský from comment #27)
> (In reply to Yaniv Kaul from comment #26)
> > (In reply to Michal Skrivanek from comment #25)
> > > (In reply to Yaniv Kaul from comment #23)
> > > > From my perspective:
> > > > {'name': 'lo', 'hw': '00:00:00:00:00:00', 'inet': ['127.0.0.1'], 'inet6':
> > > > ['::1']}
> > > > - There is no need to collect or report the loopback device. No one cares
> > > > about it, at all. It's a qemu-guest-agent and/or ovirt-guest-agent bug.
> > > 
> > > I think so too. Same as we should have the same interface blacklisting
> > > facility as implemented in ovirt-guest-agent for e.g. docker0 interface. I
> > > believe that should happen inside the guest to be able to control that based
> > > on specific OS
> > 
> > Probably need to add something like:
> > if (ifa->ifa_flags & (IFF_LOOPBACK))
> >         continue;
> > 
> > To https://github.com/qemu/qemu/blob/master/qga/commands-posix.c#L1722
> 
> Uhm, hold on guys. I think we're assuming too much here. The fact that such
> information is of no use to oVirt engine does not mean it is uninteresting
> to everyone else. We're not talking about *oVirt* guest agent anymore.

Ask in qemu upstream if anyone cares about the loopback address.

> 
> If we don't care about the info then let's filter it out in engine.

Or in VDSM? Why send non-needed data to Engine? Anything we can filter in VDSM is blessed.

(In reply to Yaniv Kaul from comment #28)

> Or in VDSM? Why send non-needed data to Engine? Anything we can filter in
> VDSM is blessed.

but it's also more difficult to control. With a simple vdc_option we can have custom blacklist filtering easily

Verified on - 4.2.4-0.1.el7

This bugzilla is included in oVirt 4.2.4 release, published on June 26th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.4 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.