Bug 823453 - The upload/download a big file by guest vm will stop after migration.
The upload/download a big file by guest vm will stop after migration.
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
6.3
x86_64 Linux
medium Severity medium
: rc
: ---
Assigned To: Libvirt Maintainers
Virtualization Bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-05-21 06:01 EDT by lei wang
Modified: 2013-08-04 22:02 EDT (History)
12 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-05-29 09:08:27 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description lei wang 2012-05-21 06:01:52 EDT
Description of problem:
The upload/download a big file by guest vm will stop after migration.

Version-Release number of selected component (if applicable):
# rpm -qa |grep libvirt
libvirt-debuginfo-0.9.10-14.el6.x86_64
libvirt-0.9.10-20.el6.x86_64
libvirt-devel-0.9.10-20.el6.x86_64
libvirt-client-0.9.10-20.el6.x86_64
libvirt-python-0.9.10-20.el6.x86_64
# rpm -q virt-manager
virt-manager-0.9.0-14.el6.x86_64
# rpm -q python-virtinst
python-virtinst-0.600.0-8.el6.noarch
# uname -r
2.6.32-272.el6.x86_64
#

How reproducible:
100%

procadion 
Configure migration environment with below command on both src and dst host.
#iptables -F
#setenforce 1
#setsebool virt_use_nfs  on
#mount 10.66.90.121:/vol/S3/libvirtauto /var/lib/libvirt/images/ -o vers=3


Steps to Reproduce:
1. Start a domain;
   # virsh start <domain>

2. Start a live migration;
   # virsh migrate --live <domain> qemu+ssh://host2/system

3. During the proccess of migration, upload or download a big file by guest vm .

  
Actual results:
The process of ftp upload/download be stop after migrate to target host.

Expected results:
The migration has no effect on upload/download

Additional info:
The same issue will reproduce on both windows and linux guest.

Guest vm dumpxml:
<domain type='kvm' id='7'>
  <name>mig-1</name>
  <uuid>e75fcacf-9ddc-edff-9c4f-4ab972997fb6</uuid>
  <memory unit='KiB'>524288</memory>
  <currentMemory unit='KiB'>524288</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='rhel6.3.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/libvirt/images/mig-1.img'>
        <seclabel relabel='no'/>
      </source>
      <target dev='hda' bus='ide'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:9d:89:c2'/>
      <source network='default'/>
      <target dev='vnet0'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c34,c119</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c34,c119</imagelabel>
  </seclabel>
</domain>
Comment 2 Michal Privoznik 2012-05-28 10:51:04 EDT
Lei, is the 'default' network a NAT one? Can you please provide its XML? Thanks in advance.
Comment 3 lei wang 2012-05-28 21:36:29 EDT
(In reply to comment #2)
> Lei, is the 'default' network a NAT one? Can you please provide its XML?
> Thanks in advance.

Hi michal, the xml provides as below, I use the default network to test it before.

# virsh net-dumpxml default
<network>
  <name>default</name>
  <uuid>0adda191-5e1a-43ef-9039-240076c9a6ca</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0' />
  <mac address='52:54:00:92:E3:12'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254' />
    </dhcp>
  </ip>
</network>
Comment 4 Michal Privoznik 2012-05-29 05:57:10 EDT
Lei thanks.

So you are using nat-ed network. Currently, with libvirt it's not possible to migrate a nat-ed domain and not break up connections.

However, you can try setting up conntrackd which should monitor all NATed connections and flush them onto destination host hence connections wouldn't break up. However, I've tried to do it manually with conntrack but without any luck.

Question is, whether libvirt should set conntrack(d) up (turning this into huge RFE) or no (closing this as NOTABUG).

Thoughts?
Comment 7 Michal Privoznik 2012-05-29 09:08:27 EDT
On second thought, even with conntrack(d) it wouldn't be possible. This is the use case:

1. Guest G initiates a connection to the host S(ource): e.g. download a file via HTTP
2. The host A which is running guest G will perform NAT, meaning it will switch source IP address, therefore apache running on S will always create packets with A in destination field. Even after migration.
3. Apache replies, host A notice incoming packet match a record in NAT table, substitute destination address and resend to the guest G.

And here comes the magic:
4. Guest is migrated to the host B
5. Apache still sends data to the host A which initiated the connection.

Even if we would somehow transfer the record in NAT table from A to B, it won't work because packets are still being delivered to A.

It might be still possible to migrate NATed guest G without breaking connection, though. If only there were another NAT between {A|B} and S (assuming A and B are on the same subnet). Then we wouldn't have to touch NAT tables on A or B at all, just update a record on this superior NAT in between. But that is something libvirt can't do; messing up with a 3rd party network device under root privileges ...


However, it is still possible to migrate without breaking guest connections. But you have to use a different network mode, e.g. bridged network, or a direct interface mode (macvtap0) - have tried and works. Even a VEPA should work. But not NAT.

Therefore I am closing this one as NOTABUG. If anybody feels otherwise, please reopen.

Note You need to log in before you can comment on or make changes to this bug.