Bug 1676822

Created attachment 1534342 [details]
virt-v2v-rhel8-rhv-network-profile.log

(In reply to mxie from comment #0)
> Additional info:
> 1.Can't reproduce the problem with rhel7 build virt-v2v:
> virt-v2v-1.38.2-12.el7_6.2.x86_64
> libguestfs-1.38.2-12.el7_6.2.x86_64

Can you please attach a log when converting the same guest from the same VMware host to the same RHV, using the RHEL 7 version?

Sorry,I found the reproducible of the bug is 50%, pls ignore the log "virt-v2v-rhel8-rhv-network-profile", pls refer to logs "virt-v2v-rhel7-guest-network-profile-1.38.4-10.module+el8+2709.log" and "virt-v2v-rhel7-guest-network-profile-1.38.2-12.el7_6.2"

Created attachment 1534430 [details]
virt-v2v-rhel7-guest-network-profile-1.38.2-12.el7_6.2.log

Created attachment 1534431 [details]
virt-v2v-rhel7-guest-network-profile-1.38.4-10.module+el8+2709.log

Isn't this a RHV problem?  The dialog in the screenshot and the error message
all appear to come from RHV, not virt-v2v.  So I think this should be reassigned
to RHV unless I'm missing something here.

(In reply to Richard W.M. Jones from comment #7)
> Isn't this a RHV problem?  The dialog in the screenshot and the error message
> all appear to come from RHV, not virt-v2v.  So I think this should be
> reassigned
> to RHV unless I'm missing something here.

OK,grab some vdsm and engine log, then assign the bug to vdsm component

Related packages:
RHV:4.3.0.1-0.1.el7
vdsm-4.30.7-1.el7ev.x86_64
libvirt-client version of rhv node and rhv: 4.5.0-10.el7_6.3.x86_64

Created attachment 1541685 [details]
vdsm.log

Created attachment 1541686 [details]
engine.log

Given vdsm exception below, I suspect that the problem is a missing <mtu> element.

Can you try to reproduce with a fresh VM? Create it with a non-wired vNIC, run it, and try to change its profile. This may be a regression due to our introduction of the <mtu> element.

2019-03-07 14:36:49,649+0800 WARN  (jsonrpc/7) [virt.vm] (vmId='03eaaeda-74db-4383-ab58-0b5b684ab4b6') Request failed: <?xml version='1.0' encoding='utf-8'?>
<interface type="bridge">
    <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci" />
    <mac address="00:50:56:ac:1f:91" />
    <model type="virtio" />
    <source bridge="ovirtmgmt" />
    <link state="up" />
    <alias name="ua-7fefe718-eeaf-4403-a36b-b7a19826b3ac" />
    <mtu size="1500" />
</interface>
 (vm:3187)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3181, in setLinkAndNetwork
    libvirt.VIR_DOMAIN_AFFECT_LIVE)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2772, in updateDeviceFlags
    if ret == -1: raise libvirtError ('virDomainUpdateDeviceFlags() failed', dom=self)
libvirtError: Operation not supported: cannot modify MTU
2019-03-07 14:36:49,650+0800 WARN  (jsonrpc/7) [virt.vm] (vmId='03eaaeda-74db-4383-ab58-0b5b684ab4b6') Rolling back link and net for: ua-7fefe718-eeaf-4403-a36b-b7a19826b3ac (vm:3196)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3191, in setLinkAndNetwork
    raise SetLinkAndNetworkError(str(e))
SetLinkAndNetworkError: Operation not supported: cannot modify MTU

(In reply to Dan Kenigsberg from comment #11)
> Given vdsm exception below, I suspect that the problem is a missing <mtu>
> element.
> 
> Can you try to reproduce with a fresh VM? Create it with a non-wired vNIC,
> run it, and try to change its profile. This may be a regression due to our
> introduction of the <mtu> element.

Yes, I can reproduce the bug with creating a new guest on rhv4.3

Create new guest on rhv-> create disk for guest-> create a network for guest but set profile as empty-> power on guest -> try to set network profile from empty to ovirtmgmt, then pop up dialog box "Error while executing action Edit VM Interface properties: General Exception"

Another flow which points to the same logical problem is changing a non-empty
vNIC profile to a vNIC profile of a network with a different MTU than the
previous one.

The propagation of the MTU to the guest by libvirt introduces the limitation,
that only networks of the same MTU can be used without unplugging/plugging the vNIC.

To get rid of this limitation, we have to disable libvirt's MTU support, either for all
non-external networks or we could introduce a new attribute in the vNIC profile
(or vNIC, or network) to control the usage of libvirt's MTU.

A possible workaround is to unplug the vNIC, change the vNIC profile, and plug again.

The issue is reproduced without any relation to v2v. 
Start rhel8 guest with empty vNIC profile, try to update the profile to ovirtmgmt profile - fail with same error. 
This is not a v2v specific issue.

Note that the next scenario does work as expected.
- Start VM with ovirtmgmt profile
- Switch to empty vNIC - works
- Switch back to ovirtmgmt - works

Only if VM was started with an empty profile, the update is failing. If started with regular profile, the update work as expected.

(In reply to Michael Burman from comment #16)
> Note that the next scenario does work as expected.
> - Start VM with ovirtmgmt profile
> - Switch to empty vNIC - works
> - Switch back to ovirtmgmt - works
> 
> Only if VM was started with an empty profile, the update is failing. If
> started with regular profile, the update work as expected.

The flow is also failing if the VM is started with network that has MTU 1500 and 
you want to switch to network that has MTU different from this value.

The reason libvirt is giving the error "Operation not supported: cannot modify MTU" is because it's not possible to modify the host_mtu option of the virtio-net device after it has been added. The way host_mtu just sets a parameter in the device's pci config space, and that value is read by the guest's virtio-net driver during initialization and used to set the MTU on the guest side of the emulated device.

libvirt could easily modify the MTU of the tap device on the host side, but that would likely lead to confusion and probably dropped packets.

For reference, here is the commit message from the libvirt change that adds the check to precent modifying the MTU (prior to this patch, the changes weren't acted on, but were silently ignored):

    commit 5f44d7e357f61f7be636a0e2e6d35453cbc3b589
    Author: Michal Privoznik <mprivozn>
    Date:   Thu Jun 8 13:45:31 2017 +0200

    qemuDomainChangeNet: Forbid changing MTU
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1447618
    
    Currently, any attempt to change MTU on an interface that is
    plugged to a running domain is silently ignored. We should either
    do what's asked or error out. Well, we can update the host side
    of the interface, but we cannot change 'host_mtu' attribute for
    the virtio-net device. Therefore we have to error out.

If there was a way to "push" an MTU change to the guest driver, libvirt would happily support it, but as far as I understand from email discussions, this isn't considered a viable option (which essentially means that plugging into a network with a different MTU is a pre-destined "fail" unless PMTU is supported by all parties, and apparently *that* isn't the case either :-(

Michael, can you please confirm that this bug existed already in 4.2 (should be since 4.2.6)

(In reply to Dominik Holler from comment #22)
> Michael, can you please confirm that this bug existed already in 4.2 (should
> be since 4.2.6)

Issue exist on 4.2.8.3-0.1.el7ev
vdsm-4.20.47-1.el7ev.x86_64

But note, that i can only reproduce the empty vNIC scenario. I can't reproduce the update MTU scenario as Ales suggested in comment 18, not on 4.2 and not on 4.3

(In reply to Laine Stump from comment #21)
> 
> If there was a way to "push" an MTU change to the guest driver, libvirt
> would happily support it, but as far as I understand from email discussions,
> this isn't considered a viable option (which essentially means that plugging
> into a network with a different MTU is a pre-destined "fail" unless PMTU is
> supported by all parties, and apparently *that* isn't the case either :-(

Could we please convert this BZ or create a new one as an RFE?
Moving a vNIC on the fly between networks and keeping it sync with the MTU
of the connected network is a reasonable request/need.

We have decided to set default MTU for the empty vNic profile for 4.3. 
This way the empty network profile can be changed to network with default MTU.

Currently we have inconsistency between engine and vdsm. Which will be addressed in master branch.

(In reply to Ales Musil from comment #25)
> We have decided to set default MTU for the empty vNic profile for 4.3. 
> This way the empty network profile can be changed to network with default
> MTU.
> 
> Currently we have inconsistency between engine and vdsm. Which will be
> addressed in master branch.

HI Ales,
Just to clarify, will this bug will get fixed in 4.3.3? 
when you say master, you mean 4.4 only?

(In reply to Michael Burman from comment #26)
> (In reply to Ales Musil from comment #25)
> > We have decided to set default MTU for the empty vNic profile for 4.3. 
> > This way the empty network profile can be changed to network with default
> > MTU.
> > 
> > Currently we have inconsistency between engine and vdsm. Which will be
> > addressed in master branch.
> 
> HI Ales,
> Just to clarify, will this bug will get fixed in 4.3.3? 
> when you say master, you mean 4.4 only?

The changing of empty vnic to network with default MTU. Which should be this bug 
will be fixed in 4.3.3. 

The underlying problem will be fixed in master/4.4.

(In reply to Ales Musil from comment #27)
> (In reply to Michael Burman from comment #26)
> > (In reply to Ales Musil from comment #25)
> > > We have decided to set default MTU for the empty vNic profile for 4.3. 
> > > This way the empty network profile can be changed to network with default
> > > MTU.
> > > 
> > > Currently we have inconsistency between engine and vdsm. Which will be
> > > addressed in master branch.
> > 
> > HI Ales,
> > Just to clarify, will this bug will get fixed in 4.3.3? 
> > when you say master, you mean 4.4 only?
> 
> The changing of empty vnic to network with default MTU. Which should be this
> bug 
> will be fixed in 4.3.3. 
> 
> The underlying problem will be fixed in master/4.4.

ack

Verified on - 4.3.3.1-0.1.el7 and vdsm-4.30.12-1.el7ev.x86_64

Empty vNIC has now default MTU(1500) and it;s possible to change it's vNIC profile. 

<interface type='bridge'>
      <mac address='00:00:00:00:00:00'/>
      <source bridge=';vdsmdummy;'/>
      <target dev='vnet2'/>
      <model type='virtio'/>
      <driver name='vhost' queues='4'/>
      <link state='down'/>
      <mtu size='1500'/>

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:1085