Bug 2025303 - [RFE] Guest agent should report the interface Permanent address
Summary: [RFE] Guest agent should report the interface Permanent address
Keywords:
Status: NEW
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: qemu-kvm
Version: 9.0
Hardware: Unspecified
OS: Unspecified
medium
low
Target Milestone: rc
: ---
Assignee: Kostiantyn Kostiuk
QA Contact: dehanmeng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-11-21 15:07 UTC by oshoval
Modified: 2022-11-29 21:46 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Type: Feature Request
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-103433 0 None None None 2021-11-21 15:09:10 UTC

Description oshoval 2021-11-21 15:07:47 UTC
Description of problem:

At the kubevirt project, we use guest agent reports in order to know the status of the guest interfaces.

For each interface that we configure for the guest (via domain XML), we have some additional parameters
outside of the guest (for example a custom name that comes from the virtual machine declarative spec).
By setting the MAC address in the domain XML, we are able to correlate the interface from the guest agent report
with those parameters from the vmi spec (having the MAC as the key).

The problem is that if a user in the guest changes the mac address of the interface,
we cannot use the MAC address as the key anymore.

How reproducible:

Always

Steps to Reproduce:
change the interface mac address
1. sudo ip link set dev eth0 address  02:00:b5:b5:b5:c9


Actual results:

the new interface mac is reported

Expected results:

both the new interface mac and the Permanent address should be reported in the interface entry


Additional info:

Comment 2 Qianqian Zhu 2021-11-22 02:01:58 UTC
Hi Shoval,

Does it only support on RHEL8? Would you help clone it to RHEL9 if we also support it there?

Thanks,
Qianqian

Comment 3 oshoval 2021-11-22 07:24:53 UTC
Hi
It is not just for RHEL8, it is a general RFE for the guest agent from my understanding
just wasn't sure what to fill in the product field,

I am not part of RHEL product, but part of CNV (kubevirt).

see please here the motivation in case it is interesting,
https://bugzilla.redhat.com/show_bug.cgi?id=2013455


Thanks

Comment 4 John Ferlan 2021-11-23 18:44:59 UTC
Moving this to RHEL9 for action. Once a resolution is closer we can make the decision to clone to RHEL8. No sense in having 2 bugs for now.

Assigned to developer w/ Medium priority and added CNV as dependent product.

Comment 5 Marc-Andre Lureau 2021-11-23 20:27:15 UTC
Is the feature required for guests other than Linux?

It seems we will need to query it from a netlink socket. I can't find it from sysfs (nor inet ioctl).

Would optional interface "permanent-address" : "00:00:00:00:00:00" be good enough? (if it's not set either it's the same as "hardware-address", or unsupported)

Comment 6 oshoval 2021-11-24 10:08:04 UTC
It would be better if it is supported as well for other guests that support it and wide used, thanks

In case the "permanent-address" is unsupported it is fine to have "00:00:00:00:00:00".

In case it is the same as the "hardware-address", what is the motivation to not send it, less traffic ?
as long as the report won't be ambiguous in some scenarios it is fine, i just wonder if we
can find a scenario that is ambiguous.

Thanks

Comment 7 Michal Privoznik 2021-11-24 14:46:04 UTC
Alternatively, libvirt could emit an event when MAC address changes (QEMU is doing that already and libvirt does process it for internal needs). BTW: if a guest does change MAC address then corresponding <interface/> snippet in the domain XML is updated. CNV could listen to this event and update its internal state accordingly.

Comment 8 oshoval 2021-11-24 15:00:56 UTC
Hi Michal

The event is interesting, any reference please so i will learn more about it / how to listen to it ?

> BTW: if a guest does change MAC address then corresponding <interface/> snippet in the domain XML is updated. CNV could listen to this event and update its internal state accordingly.

I tried to change the MAC in the guest using
sudo ip link set dev eth0 address 02:00:b5:b5:b5:c9
ifconfig shows the new MAC,
ethtool -P shows the original MAC
but the virsh dumpxl still shows the original MAC as well (and not the new one)

We are using QEMU Guest Agent 4.2.1 on fedora.

Thanks

Comment 9 Michal Privoznik 2021-11-24 15:55:08 UTC
(In reply to oshoval from comment #8)
> Hi Michal
> 
> The event is interesting, any reference please so i will learn more about it
> / how to listen to it ?
> 
> > BTW: if a guest does change MAC address then corresponding <interface/> snippet in the domain XML is updated. CNV could listen to this event and update its internal state accordingly.
> 
> I tried to change the MAC in the guest using
> sudo ip link set dev eth0 address 02:00:b5:b5:b5:c9
> ifconfig shows the new MAC,
> ethtool -P shows the original MAC
> but the virsh dumpxl still shows the original MAC as well (and not the new
> one)
> 

You need to enable libvirt to trust guest first. This is done by setting trustGuestRxFilters attribute to yes, like this:

  <interface type='direct' trustGuestRxFilters='yes'>
    ...
  </interface>

https://libvirt.org/formatdomain.html#network-interfaces

but be aware that this works only for some models (those which can communicate the change outside, typically virtio).

And currently, there is no event for you to listen to, because it's not implemented yet.

Comment 10 oshoval 2021-11-24 16:34:04 UTC
Thanks

we are using virtio-non-transitional, and from my understanding we should not use "virtio"
according https://libvirt.org/formatdomain.html#virtio-transitional-devices
i just tried it there, and it didn't change it (as you said, as its non "virtio")


    <interface type='ethernet' trustGuestRxFilters='yes'>
      <mac address='52:54:00:97:e1:51'/>
      <target dev='tap0' managed='no'/>
      <model type='virtio-non-transitional'/>
      <mtu size='1480'/>
      <alias name='ua-default'/>
      <rom enabled='no'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>

Comment 11 Michal Privoznik 2021-11-25 12:30:16 UTC
(In reply to oshoval from comment #10)
> Thanks
> 
> we are using virtio-non-transitional, and from my understanding we should
> not use "virtio"
> according https://libvirt.org/formatdomain.html#virtio-transitional-devices
> i just tried it there, and it didn't change it (as you said, as its non
> "virtio")
> 
> 
>     <interface type='ethernet' trustGuestRxFilters='yes'>
>       <mac address='52:54:00:97:e1:51'/>
>       <target dev='tap0' managed='no'/>
>       <model type='virtio-non-transitional'/>

This is virtio! But also it is of type='ethernet' which means, an external entity is in charge of the interface.

>       <mtu size='1480'/>
>       <alias name='ua-default'/>
>       <rom enabled='no'/>
>       <address type='pci' domain='0x0000' bus='0x01' slot='0x00'
> function='0x0'/>
>     </interface>

Correct, libvirt doesn't reflect MAC address change even though QEMU did emit NIC_RX_FILTER_CHANGED event (the event means a MAC address inside the guest has changed). @laine do you remember what our reasoning was to reflect the change in domain XML only for certain <interface/> types? Since QEMU delives the event I believe we can update the XML regardless of the type of <interface/> and execute changes needed in the host only for a subset of types (just like we are doing now)?

Comment 12 Edward Haas 2022-01-19 10:07:00 UTC
@mprivozn , I am still unclear if changing the non-permanent address
is suppose to be reflected back to libvirt domxml or not.
I have not seen this happening, and per oshoval results, it is not seen.

If this would have worked with virtio, then I think we do not even
need the event getting forwarded by libvirt. kubevirt is already
refreshing the domain information periodically if it is idle for too
long.

But I must admit this is too odd and does not compute well logically for me.
In the domain xml, we set the desired MAC address for an interface, which
is suppose to be this permanent address, emulating a real HW NIC.
If the admin in the guest is changing the MAC, I would expect only
the soft address to be changed, and therefore, not to reflect back and
change the domxml details. But maybe I missed something here.

If the intention was that the user of libvirt needs to monitor an event
to catch such changes, and this not being persisted anywhere as a state,
then I find this unreliable. The client, kubevirt, may loose this information,
for various reasons (e.g. the app crashed). We depend on libvirt/qemu to
persist and monitor the state of the guest.

I would prefer the `"permanent-address"` option, it seems direct and simple
from a user perspective.

Comment 13 Michal Privoznik 2022-01-19 14:45:51 UTC
(In reply to Edward Haas from comment #12)
> @mprivozn , I am still unclear if changing the non-permanent address
> is suppose to be reflected back to libvirt domxml or not.
> I have not seen this happening, and per oshoval results, it is not seen.
> 
> If this would have worked with virtio, then I think we do not even
> need the event getting forwarded by libvirt. kubevirt is already
> refreshing the domain information periodically if it is idle for too
> long.
> 
> But I must admit this is too odd and does not compute well logically for me.
> In the domain xml, we set the desired MAC address for an interface, which
> is suppose to be this permanent address, emulating a real HW NIC.
> If the admin in the guest is changing the MAC, I would expect only
> the soft address to be changed, and therefore, not to reflect back and
> change the domxml details. But maybe I missed something here.
> 
> If the intention was that the user of libvirt needs to monitor an event
> to catch such changes, and this not being persisted anywhere as a state,
> then I find this unreliable. The client, kubevirt, may loose this
> information,
> for various reasons (e.g. the app crashed). We depend on libvirt/qemu to
> persist and monitor the state of the guest.
> 
> I would prefer the `"permanent-address"` option, it seems direct and simple
> from a user perspective.
 
The MAC address of a guest interface can be used for so much. If host blindly trusts guest then a malicious guest can set a MAC address of another guest and thus redirect that traffic onto itself. Then, libvirt has a submodule which allows users to automatically set up firewall rules. Again, MAC address can be used in those rules so a malicious guest can cut off another guest if it sets just right MAC address. Trusting guest blindly is very dangerous and should never ever be the default.

Having said that, libvirt introduced a knob which allows you to mark what interfaces within given guest you trust and thus reflect MAC address change in the domain XML.

I'm not against extending guest agent, but it will require new libvirt API as the current one does not offer a way to expose new fields.

Comment 14 Laine Stump 2022-02-06 01:00:58 UTC
Sorry, I should have jumped in with my memories of NIC_RX_FILTER_CHANGED back a couple months ago, but truthfully *had* no memories (that all happened a very long time ago!), so I resolved to look into it before typing anything, and then promptly forgot.

I've finally looked back at the code to remind myself, and what I've now recalled is that libvirt only added in processing of the NIC_RC_FILTER_CHANGED event for one specific purpose - in order to update the MAC address of the host ethernet interface when a macvtap passthrough mode interface had its MAC address changed in the guest (this is necessary because proper operation of macvtap passthrough mode requires that the hardware interface on the host have exactly the same MAC address and RX_FILTER as the emulated interface in the guest). There was never any intention of responding to this event for interface types other than macvtap passthrough mode, nor of mirroring that new setting back into the domain XML, and I'm pretty sure that doesn't ever happen anywhere (the code is just so convoluted that someone might assume that had happened somewhere :-))

BTW, with one exception [*], libvirt processing of NIC_RX_FILTER_CHANGED doesn't take any action for other types of interfaces (e.g. tap device-based interfaces, or even macvtap bridged mode) because there is no direct correlation between host and guest MAC address needed for proper operation of those types of interface. As a matter of fact, in the case of a tap device, the two ends of the tap must *not* match!

([*] the exception to the "no action for interface types other than macvtap/passthrough" is that I found Michal at some point added support for updating the bandwidth shaping rules for tap devices on a virtual network when the MAC address in the guest changes)

As Michal pointed out, since MAC address is used as an index for finding an interface in so many places (before user-named device aliases were available in libvirt, it was pretty much the *only* way to find an interface) we can't change the MAC address of an interface in the XML at runtime. Instead it should be reported in some other way, perhaps by a more extensible new generation of the virDomainInterfaceAddresses() API (which unfortunately, as Michal says, was designed to only report a name, a single hardware address (MAC), and a list of IP addresses with no extensibility in mind).

Along with that, libvirt could also add public reporting of the NIC_RX_FILTER_CHANGED event to allow libvirt consumers to deal with keeping track of current MAC address themselves.

Comment 15 John Ferlan 2022-11-29 21:46:03 UTC
Reassigning since you've been picking up qga tasks from Marc-Andre.

May be worth the cycles to just figure out what is necessary given the above and the reality that this bug hasn't had cycles for 9+ months.


Note You need to log in before you can comment on or make changes to this bug.