Bug 1676822 - Cannot change Empty network profile of a running VM(default MTU)
Summary: Cannot change Empty network profile of a running VM(default MTU)
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine
Version: 4.3.0
Hardware: x86_64
OS: Unspecified
medium
high
Target Milestone: ovirt-4.3.3
: 4.3.0
Assignee: Ales Musil
QA Contact: Michael Burman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-02-13 10:15 UTC by mxie@redhat.com
Modified: 2019-05-08 12:39 UTC (History)
21 users (show)

Fixed In Version: ovirt-engine-4.3.3.1
Doc Type: Bug Fix
Doc Text:
Previously, while testing a RHEL 8 build of the virt-v2v daemon that turns a Red Hat Virtualization host into a conversion host for CloudForms migration, you could not update the network profile of a running virtual machine guest. The current release fixes this issue.
Clone Of:
Environment:
Last Closed: 2019-05-08 12:39:10 UTC
oVirt Team: Network
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
network-profile-v2v-rhel8.png (53.88 KB, image/png)
2019-02-13 10:15 UTC, mxie@redhat.com
no flags Details
virt-v2v-rhel8-rhv-network-profile.log (2.71 MB, text/plain)
2019-02-13 10:16 UTC, mxie@redhat.com
no flags Details
virt-v2v-rhel7-guest-network-profile-1.38.2-12.el7_6.2.log (3.01 MB, text/plain)
2019-02-13 15:00 UTC, mxie@redhat.com
no flags Details
virt-v2v-rhel7-guest-network-profile-1.38.4-10.module+el8+2709.log (2.98 MB, text/plain)
2019-02-13 15:02 UTC, mxie@redhat.com
no flags Details
vdsm.log (86.18 KB, text/plain)
2019-03-07 06:53 UTC, mxie@redhat.com
no flags Details
engine.log (8.76 KB, text/plain)
2019-03-07 06:53 UTC, mxie@redhat.com
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2019:1085 0 None None None 2019-05-08 12:39:21 UTC
oVirt gerrit 98485 0 master MERGED core: Set MTU of empty vnic profile 2020-12-16 04:03:52 UTC
oVirt gerrit 98765 0 ovirt-engine-4.3 MERGED core: Explicitly set the MTU for an empty vnic profile 2020-12-16 04:03:52 UTC

Description mxie@redhat.com 2019-02-13 10:15:07 UTC
Created attachment 1534341 [details]
network-profile-v2v-rhel8.png

Description of problem:
Can't change running guest's network profile after rhel8 build virt-v2v conversion on rhv

Version-Release number of selected component (if applicable):
virt-v2v-1.38.4-10.module+el8+2709+40ed2f2c.x86_64
libguestfs-1.38.4-10.module+el8+2709+40ed2f2c.x86_64
libvirt-4.5.0-21.module+el8+2777+e17f6250.x86_64
nbdkit-1.4.2-4.module+el8+2586+bf759444.x86_64
qemu-kvm-2.12.0-60.module+el8+2725+0ab65287.x86_64
rhv:4.3.0.1-0.1.el7

How reproducible:
100%

Steps to reproduce:
1.Convert a guest from VMware to rhv by virt-v2v on rhel8 server
# virt-v2v -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 -it vddk -io vddk-libdir=/home/vmware-vix-disklib-distrib -io  vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA -o rhv-upload -oo rhv-cafile=/home/ca.pem -oo rhv-direct -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /tmp/rhvpasswd --password-file /tmp/passwd -os nfs_data -b ovirtmgmt esx6.7-rhel7.6-x86_64 -oo rhv-cluster=nfs
[   0.5] Opening the source -i libvirt -ic vpx://root.73.141/data/10.73.75.219/?no_verify=1 esx6.7-rhel7.6-x86_64 -it vddk  -io vddk-libdir=/home/vmware-vix-disklib-distrib -io vddk-thumbprint=1F:97:34:5F:B6:C2:BA:66:46:CB:1A:71:76:7D:6B:50:1E:03:00:EA
[   2.2] Creating an overlay to protect the source from being modified
[   5.4] Initializing the target -o rhv-upload -oc https://ibm-x3250m5-03.rhts.eng.pek2.redhat.com/ovirt-engine/api -op /tmp/rhvpasswd -os nfs_data
[   6.7] Opening the overlay
[  14.3] Inspecting the overlay
[  40.0] Checking for sufficient free disk space in the guest
[  40.0] Estimating space required on target for each disk
[  40.0] Converting Red Hat Enterprise Linux Server 7.6 (Maipo) to run on KVM
virt-v2v: This guest has virtio drivers installed.
[ 201.3] Mapping filesystem data to avoid copying unused and blank areas
[ 202.0] Closing the overlay
[ 202.3] Checking if the guest needs BIOS or UEFI to boot
[ 202.3] Assigning disks to buses
[ 202.3] Copying disk 1/1 to qemu URI json:{ "file.driver": "nbd", "file.path": "/var/tmp/rhvupload.ySN6sm/nbdkit1.sock", "file.export": "/" } (raw)
    (100.00/100%)
[ 861.4] Creating output metadata
[ 877.1] Finishing off

2.Power on guest on rhv after finishing conversion,try to change network profile for guest in network interface but pop up error as screenshot"network-profile-v2v-rhel8"

3.Check guest's libvirtxml on rhv node
# virsh dumpxml esx6.7-rhel7.6-x86_64
Please enter your authentication name: test
Please enter your password:
.....
<interface type='bridge'>
      <mac address='00:50:56:ac:b6:b9'/>
      <source bridge=';vdsmdummy;'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <link state='down'/>
      <alias name='ua-7f74ffdd-8ab0-49d4-992b-c69be7de1611'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>


Actual results:
As above description

Expected results:
Can change running guest's network profile after rhel8 build virt-v2v conversion on rhv


Additional info:
1.Can't reproduce the problem with rhel7 build virt-v2v:
virt-v2v-1.38.2-12.el7_6.2.x86_64
libguestfs-1.38.2-12.el7_6.2.x86_64

Found the guest which is converted by rhel7 build v2v to rhv4.3 has below libvirtxml on rhv node
....
 <interface type='bridge'>
      <mac address='00:50:56:ac:20:28'/>
      <source bridge=';vdsmdummy;'/>
      <target dev='vnet3'/>
      <model type='virtio'/>
      <driver name='vhost' queues='2'/>
      <filterref filter='vdsm-no-mac-spoofing'/>
      <link state='down'/>
      <mtu size='1500'/>
      <alias name='ua-e01f20a8-3b16-48fe-b2c5-678f0ff3e5a1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
....

I think the problem is caused by guest of step3 has no "<filterref filter='vdsm-no-mac-spoofing'/>"


2.Also can't reproduce the bug if create a new guest on rhv and the guest's libvirtxml has <filterref filter='vdsm-no-mac-spoofing'/> by default, so the bug is not related to rhv

Comment 1 mxie@redhat.com 2019-02-13 10:16:29 UTC
Created attachment 1534342 [details]
virt-v2v-rhel8-rhv-network-profile.log

Comment 2 Pino Toscano 2019-02-13 11:41:34 UTC
(In reply to mxie from comment #0)
> Additional info:
> 1.Can't reproduce the problem with rhel7 build virt-v2v:
> virt-v2v-1.38.2-12.el7_6.2.x86_64
> libguestfs-1.38.2-12.el7_6.2.x86_64

Can you please attach a log when converting the same guest from the same VMware host to the same RHV, using the RHEL 7 version?

Comment 3 mxie@redhat.com 2019-02-13 14:59:10 UTC
Sorry,I found the reproducible of the bug is 50%, pls ignore the log "virt-v2v-rhel8-rhv-network-profile", pls refer to logs "virt-v2v-rhel7-guest-network-profile-1.38.4-10.module+el8+2709.log" and "virt-v2v-rhel7-guest-network-profile-1.38.2-12.el7_6.2"

Comment 4 mxie@redhat.com 2019-02-13 15:00:52 UTC
Created attachment 1534430 [details]
virt-v2v-rhel7-guest-network-profile-1.38.2-12.el7_6.2.log

Comment 5 mxie@redhat.com 2019-02-13 15:02:38 UTC
Created attachment 1534431 [details]
virt-v2v-rhel7-guest-network-profile-1.38.4-10.module+el8+2709.log

Comment 7 Richard W.M. Jones 2019-03-04 11:46:20 UTC
Isn't this a RHV problem?  The dialog in the screenshot and the error message
all appear to come from RHV, not virt-v2v.  So I think this should be reassigned
to RHV unless I'm missing something here.

Comment 8 mxie@redhat.com 2019-03-07 06:52:19 UTC
(In reply to Richard W.M. Jones from comment #7)
> Isn't this a RHV problem?  The dialog in the screenshot and the error message
> all appear to come from RHV, not virt-v2v.  So I think this should be
> reassigned
> to RHV unless I'm missing something here.

OK,grab some vdsm and engine log, then assign the bug to vdsm component

Related packages:
RHV:4.3.0.1-0.1.el7
vdsm-4.30.7-1.el7ev.x86_64
libvirt-client version of rhv node and rhv: 4.5.0-10.el7_6.3.x86_64

Comment 9 mxie@redhat.com 2019-03-07 06:53:15 UTC
Created attachment 1541685 [details]
vdsm.log

Comment 10 mxie@redhat.com 2019-03-07 06:53:58 UTC
Created attachment 1541686 [details]
engine.log

Comment 11 Dan Kenigsberg 2019-03-07 20:42:19 UTC
Given vdsm exception below, I suspect that the problem is a missing <mtu> element.

Can you try to reproduce with a fresh VM? Create it with a non-wired vNIC, run it, and try to change its profile. This may be a regression due to our introduction of the <mtu> element.

2019-03-07 14:36:49,649+0800 WARN  (jsonrpc/7) [virt.vm] (vmId='03eaaeda-74db-4383-ab58-0b5b684ab4b6') Request failed: <?xml version='1.0' encoding='utf-8'?>
<interface type="bridge">
    <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci" />
    <mac address="00:50:56:ac:1f:91" />
    <model type="virtio" />
    <source bridge="ovirtmgmt" />
    <link state="up" />
    <alias name="ua-7fefe718-eeaf-4403-a36b-b7a19826b3ac" />
    <mtu size="1500" />
</interface>
 (vm:3187)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3181, in setLinkAndNetwork
    libvirt.VIR_DOMAIN_AFFECT_LIVE)
  File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line 100, in f
    ret = attr(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/libvirtconnection.py", line 131, in wrapper
    ret = f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/common/function.py", line 94, in wrapper
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2772, in updateDeviceFlags
    if ret == -1: raise libvirtError ('virDomainUpdateDeviceFlags() failed', dom=self)
libvirtError: Operation not supported: cannot modify MTU
2019-03-07 14:36:49,650+0800 WARN  (jsonrpc/7) [virt.vm] (vmId='03eaaeda-74db-4383-ab58-0b5b684ab4b6') Rolling back link and net for: ua-7fefe718-eeaf-4403-a36b-b7a19826b3ac (vm:3196)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 3191, in setLinkAndNetwork
    raise SetLinkAndNetworkError(str(e))
SetLinkAndNetworkError: Operation not supported: cannot modify MTU

Comment 12 mxie@redhat.com 2019-03-08 05:35:45 UTC
(In reply to Dan Kenigsberg from comment #11)
> Given vdsm exception below, I suspect that the problem is a missing <mtu>
> element.
> 
> Can you try to reproduce with a fresh VM? Create it with a non-wired vNIC,
> run it, and try to change its profile. This may be a regression due to our
> introduction of the <mtu> element.

Yes, I can reproduce the bug with creating a new guest on rhv4.3

Create new guest on rhv-> create disk for guest-> create a network for guest but set profile as empty-> power on guest -> try to set network profile from empty to ovirtmgmt, then pop up dialog box "Error while executing action Edit VM Interface properties: General Exception"

Comment 14 Dominik Holler 2019-03-08 10:47:41 UTC
Another flow which points to the same logical problem is changing a non-empty
vNIC profile to a vNIC profile of a network with a different MTU than the
previous one.

The propagation of the MTU to the guest by libvirt introduces the limitation,
that only networks of the same MTU can be used without unplugging/plugging the vNIC.

To get rid of this limitation, we have to disable libvirt's MTU support, either for all
non-external networks or we could introduce a new attribute in the vNIC profile
(or vNIC, or network) to control the usage of libvirt's MTU.

A possible workaround is to unplug the vNIC, change the vNIC profile, and plug again.

Comment 15 Michael Burman 2019-03-10 09:46:03 UTC
The issue is reproduced without any relation to v2v. 
Start rhel8 guest with empty vNIC profile, try to update the profile to ovirtmgmt profile - fail with same error. 
This is not a v2v specific issue.

Comment 16 Michael Burman 2019-03-10 11:37:10 UTC
Note that the next scenario does work as expected.
- Start VM with ovirtmgmt profile
- Switch to empty vNIC - works
- Switch back to ovirtmgmt - works

Only if VM was started with an empty profile, the update is failing. If started with regular profile, the update work as expected.

Comment 18 Ales Musil 2019-03-12 12:50:13 UTC
(In reply to Michael Burman from comment #16)
> Note that the next scenario does work as expected.
> - Start VM with ovirtmgmt profile
> - Switch to empty vNIC - works
> - Switch back to ovirtmgmt - works
> 
> Only if VM was started with an empty profile, the update is failing. If
> started with regular profile, the update work as expected.

The flow is also failing if the VM is started with network that has MTU 1500 and 
you want to switch to network that has MTU different from this value.

Comment 21 Laine Stump 2019-03-20 13:43:57 UTC
The reason libvirt is giving the error "Operation not supported: cannot modify MTU" is because it's not possible to modify the host_mtu option of the virtio-net device after it has been added. The way host_mtu just sets a parameter in the device's pci config space, and that value is read by the guest's virtio-net driver during initialization and used to set the MTU on the guest side of the emulated device.

libvirt could easily modify the MTU of the tap device on the host side, but that would likely lead to confusion and probably dropped packets.

For reference, here is the commit message from the libvirt change that adds the check to precent modifying the MTU (prior to this patch, the changes weren't acted on, but were silently ignored):

    commit 5f44d7e357f61f7be636a0e2e6d35453cbc3b589
    Author: Michal Privoznik <mprivozn>
    Date:   Thu Jun 8 13:45:31 2017 +0200

    qemuDomainChangeNet: Forbid changing MTU
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1447618
    
    Currently, any attempt to change MTU on an interface that is
    plugged to a running domain is silently ignored. We should either
    do what's asked or error out. Well, we can update the host side
    of the interface, but we cannot change 'host_mtu' attribute for
    the virtio-net device. Therefore we have to error out.

If there was a way to "push" an MTU change to the guest driver, libvirt would happily support it, but as far as I understand from email discussions, this isn't considered a viable option (which essentially means that plugging into a network with a different MTU is a pre-destined "fail" unless PMTU is supported by all parties, and apparently *that* isn't the case either :-(

Comment 22 Dominik Holler 2019-03-20 20:53:44 UTC
Michael, can you please confirm that this bug existed already in 4.2 (should be since 4.2.6)

Comment 23 Michael Burman 2019-03-21 07:14:04 UTC
(In reply to Dominik Holler from comment #22)
> Michael, can you please confirm that this bug existed already in 4.2 (should
> be since 4.2.6)

Issue exist on 4.2.8.3-0.1.el7ev
vdsm-4.20.47-1.el7ev.x86_64

But note, that i can only reproduce the empty vNIC scenario. I can't reproduce the update MTU scenario as Ales suggested in comment 18, not on 4.2 and not on 4.3

Comment 24 Edward Haas 2019-03-21 09:20:34 UTC
(In reply to Laine Stump from comment #21)
> 
> If there was a way to "push" an MTU change to the guest driver, libvirt
> would happily support it, but as far as I understand from email discussions,
> this isn't considered a viable option (which essentially means that plugging
> into a network with a different MTU is a pre-destined "fail" unless PMTU is
> supported by all parties, and apparently *that* isn't the case either :-(

Could we please convert this BZ or create a new one as an RFE?
Moving a vNIC on the fly between networks and keeping it sync with the MTU
of the connected network is a reasonable request/need.

Comment 25 Ales Musil 2019-03-25 08:12:30 UTC
We have decided to set default MTU for the empty vNic profile for 4.3. 
This way the empty network profile can be changed to network with default MTU.

Currently we have inconsistency between engine and vdsm. Which will be addressed in master branch.

Comment 26 Michael Burman 2019-03-25 08:25:50 UTC
(In reply to Ales Musil from comment #25)
> We have decided to set default MTU for the empty vNic profile for 4.3. 
> This way the empty network profile can be changed to network with default
> MTU.
> 
> Currently we have inconsistency between engine and vdsm. Which will be
> addressed in master branch.

HI Ales,
Just to clarify, will this bug will get fixed in 4.3.3? 
when you say master, you mean 4.4 only?

Comment 27 Ales Musil 2019-03-25 08:32:12 UTC
(In reply to Michael Burman from comment #26)
> (In reply to Ales Musil from comment #25)
> > We have decided to set default MTU for the empty vNic profile for 4.3. 
> > This way the empty network profile can be changed to network with default
> > MTU.
> > 
> > Currently we have inconsistency between engine and vdsm. Which will be
> > addressed in master branch.
> 
> HI Ales,
> Just to clarify, will this bug will get fixed in 4.3.3? 
> when you say master, you mean 4.4 only?

The changing of empty vnic to network with default MTU. Which should be this bug 
will be fixed in 4.3.3. 

The underlying problem will be fixed in master/4.4.

Comment 28 Michael Burman 2019-03-25 08:34:05 UTC
(In reply to Ales Musil from comment #27)
> (In reply to Michael Burman from comment #26)
> > (In reply to Ales Musil from comment #25)
> > > We have decided to set default MTU for the empty vNic profile for 4.3. 
> > > This way the empty network profile can be changed to network with default
> > > MTU.
> > > 
> > > Currently we have inconsistency between engine and vdsm. Which will be
> > > addressed in master branch.
> > 
> > HI Ales,
> > Just to clarify, will this bug will get fixed in 4.3.3? 
> > when you say master, you mean 4.4 only?
> 
> The changing of empty vnic to network with default MTU. Which should be this
> bug 
> will be fixed in 4.3.3. 
> 
> The underlying problem will be fixed in master/4.4.

ack

Comment 30 Michael Burman 2019-03-31 06:35:16 UTC
Verified on - 4.3.3.1-0.1.el7 and vdsm-4.30.12-1.el7ev.x86_64

Empty vNIC has now default MTU(1500) and it;s possible to change it's vNIC profile. 

<interface type='bridge'>
      <mac address='00:00:00:00:00:00'/>
      <source bridge=';vdsmdummy;'/>
      <target dev='vnet2'/>
      <model type='virtio'/>
      <driver name='vhost' queues='4'/>
      <link state='down'/>
      <mtu size='1500'/>

Comment 32 errata-xmlrpc 2019-05-08 12:39:10 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:1085


Note You need to log in before you can comment on or make changes to this bug.