Bug 1202990

Summary:	corrupt downloads when using qemu-kvm
Product:	[Fedora] Fedora	Reporter:	Harm Weites <harm>
Component:	qemu	Assignee:	Fedora Virtualization Maintainers <virt-maint>
Status:	CLOSED INSUFFICIENT_DATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	21	CC:	amit.shah, berrange, cfergeau, clalancette, crobinso, dwmw2, ehabkost, extras-orphan, harm, itamar, markmc, pbonzini, quintela, rjones, scottt.tw, virt-maint
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2015-11-04 23:40:32 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Harm Weites 2015-03-17 20:40:56 UTC

Description of problem:
Openstack Juno 2014.2.1 (RDO) compute-node running Fedora 21, all kvm instances suffer from the same issue: (HTTP?) network traffic is corrupted. This behavior is seen with the following guests: Ubuntu 14.04, Fedora 21, Cirros 0.3.1

Launching a guest with virt_type=qemu does -not- show corruption and results in proper network communication.

Version-Release number of selected component (if applicable):
fedora-release-server-21-2.noarch
3.18.5-201.fc21.x86_64
openvswitch-2.3.1-2.git20150113.fc21.x86_64
qemu-kvm-2.1.2-7.fc21.x86_64

Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
Intel(R) Celeron(R) CPU J1900 @ 1.99GHz

How reproducible:
Sadly, always.

Steps to Reproduce:
1. login to Fedora 21 guest (over ssh, associated floating ip)
2. sudo yum update
3. observe output from yum

Actual results:
http://mirror.vutbr.cz/fedora/updates/21/x86_64/repodata/6be16a8c3437843dfb1672a757b8a50bf20a53c130b5a85c0608b0dbcd5b2c12-primary.sqlite.bz2: [Errno -1] Metadata file does not match check
[... and this continues for all other mirrors]

Expected results:
The regular list of new packages.

Additional info:
I've downloaded this file to the hypervisor or some other machine on the same subnet, no problem. The downloaded file is verified as ok according to sha256sum. Downloading the file from a local webserver results in the same corruption error: different sha256 sums.
Obviously, the yum and wget operations are fine when run from the hypervisor node.

Comment 1 Harm Weites 2015-03-18 18:08:42 UTC

Some additional info: this is seen on CentOS 7 as well. Furthermore, the instance is getting an MTU 1400 configured since the topology is using openvswitch vxlan tunnels.

Comment 2 Harm Weites 2015-03-18 19:47:39 UTC

Additional guests:

Ubuntu 10.04, kernel 2.6.32-71:
runs through apt-get update without issues

Ubuntu 12.04, kernel 3.2.0-77:
fails on apt-get update

CentOS 7, kernel 3.10.0:
fails on yum update

Comment 3 Harm Weites 2015-03-19 21:04:41 UTC

Some results after switching from VXLAN to GRE:

Cirros suddenly properly downloads files, the other images still show corruption... Interesting detail: ubuntu uses 3.2.0-77 and cirros uses 3.2.0-41, and only cirros is producing 98% proper results.

Comment 4 Harm Weites 2015-03-25 19:44:02 UTC

Ok, switched the hypervisor to NOT offer the guest a virtio device but rtl8139. This seems to do the trick.

Specifically:
00:03.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8100/8101L/8139 PCI Fast Ethernet Adapter (rev 20)

The tradeof is a severely decreased throughput though, from 940Mbit to roughly 60Mbit.

Comment 5 Richard W.M. Jones 2015-04-01 13:37:59 UTC

Can you do a simpler test.  There are so many factors involved
with apt-get/yum/... that I don't think you're getting much useful
information by testing using those commands.

Install 'wget' in the guests.

Go back to virtio-net.

Make sure that http_proxy environment variable is *not* set:

  unset http_proxy

Do:

  wget http://libguestfs.org/download/builder/centos-6.xz
  ls -l centos-6.xz
  sha512sum centos-6.xz

The correct size and sha512sum of that file is:

199265736

fc403ea3555a5608a25ad30ce2514b67288311a7197ddf9fb664475820f26db2bd95a86be9cd6e3f772187b384a02e0965430456dd518d343a80457057bc5441

Anything else would indicate corruption, likely on the openvswitch
network or at the guest NIC.

Comment 6 Harm Weites 2015-04-19 13:43:58 UTC

I've did this (only using wget) already, perhaps not made that clear enough in the first comment though.

Download a specific file from multiple locations over multiple iterations and keep track of the checksums. Result: corruption, as described, when using the virtio nic in my instances.

Comment 7 Richard W.M. Jones 2015-04-19 13:58:32 UTC

Can you do the test actually described in comment 5 and report
back on precisely what it says.

Comment 8 Fedora End Of Life 2015-11-04 12:36:53 UTC

This message is a reminder that Fedora 21 is nearing its end of life.
Approximately 4 (four) weeks from now Fedora will stop maintaining
and issuing updates for Fedora 21. It is Fedora's policy to close all
bug reports from releases that are no longer maintained. At that time
this bug will be closed as EOL if it remains open with a Fedora  'version'
of '21'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version.

Thank you for reporting this issue and we are sorry that we were not 
able to fix it before Fedora 21 is end of life. If you would still like 
to see this bug fixed and are able to reproduce it against a later version 
of Fedora, you are encouraged  change the 'version' to a later Fedora 
version prior this bug is closed as described in the policy above.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events. Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.