Bug 1460623 - There are differences between downloaded data from the given volume and the original volume content
There are differences between downloaded data from the given volume and the o...
Status: MODIFIED
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt (Show other bugs)
7.4
ppc64le Linux
unspecified Severity high
: rc
: 7.5
Assigned To: Andrea Bolognani
junli
: Patch
Depends On:
Blocks: 1399177
  Show dependency treegraph
 
Reported: 2017-06-12 04:56 EDT by wenli
Modified: 2017-12-05 05:56 EST (History)
17 users (show)

See Also:
Fixed In Version: libvirt-3.9.0-4.el7
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
auto script (639 bytes, text/plain)
2017-08-10 05:55 EDT, junli
no flags Details
dir_pool incorrect log (164.79 KB, text/plain)
2017-08-10 05:57 EDT, junli
no flags Details
logical_pool incorrect log (175.14 KB, text/plain)
2017-08-10 05:58 EDT, junli
no flags Details


External Trackers
Tracker ID Priority Status Summary Last Updated
IBM Linux Technology Center 155953 None None None 2017-06-23 09:36 EDT

  None (edit)
Description wenli 2017-06-12 04:56:39 EDT
Description of problem:
There are difference between downloaded data from the given volume and the original volume content. And this question is occurred on ppc machine only.

Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux Server release 7.4 Beta (Maipo)
Linux ibm-p8-garrison-02.rhts.eng.bos.redhat.com 3.10.0-675.el7.ppc64le #1 SMP Mon May 29 23:22
libvirt-3.2.0-9.el7.ppc64le

How reproducible:
80%

Steps to Reproduce:
1. virsh pool-create storage.xml [as follows]
<pool type='dir'>
  <name>dir_pool</name>
  <source>
  </source>
  <target>
    <path>/var/lib/libvirt/images/dir_pool</path>
  </target>
</pool>

2. virsh vol-create dir_pool volume.xml [as follows]
<volume>
  <name>vol_dir_pool.img</name>
  <capacity unit="M">50</capacity>
  <allocation>0</allocation>
  <target>
    <path>/var/lib/libvirt/images/dir_pool/vol_dir_pool.img</path>
    <format type="raw"/>
  </target>
</volume>

3. virsh vol-download /var/lib/libvirt/images/dir_pool/vol_dir_pool.img /var/lib/libvirt/images/dir_pool/vol_test

4. cd /var/lib/libvirt/images/dir_pool && diff vol_dir_pool.img vol_test

Actual results:
Binary files vol_dir_pool.img and vol_test are difference, so I hexdump both file, and get follow value.

# hexdump vol_dir_pool.img

0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
3200000

# hexdump vol_test

0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
317ed70

Expected results:
There are the same content and hex value, no different between them.
Comment 2 Michal Privoznik 2017-06-13 04:44:06 EDT
There was a lot of movement in this area lately. Although not until 3.2.0. What's the output of "ls -ls vol_dir_pool.img vol_test".
Comment 3 wenli 2017-06-13 04:56:25 EDT
# ls -ls vol_dir_pool.img vol_test
    0 -rw-------. 1 root root 52428800 Jun 13 04:54 vol_dir_pool.img
50684 -rw-r--r--. 1 root root 51899760 Jun 13 04:54 vol_test
Comment 4 David Gibson 2017-06-13 21:55:25 EDT
Looks like the downloaded volume is getting truncated.
Comment 5 wenli 2017-06-14 00:09:07 EDT
This situation is existed pool that type is logical. And above two xml files contents are as follows.

# vgcreate logical_pool /dev/sdk1

storage.xml

<pool type="logical">
  <name>logical_pool</name>
  <source>
    <name>logical_pool</name>
    <format type="lvm2"/>
    <device path="/dev/sdk1"/>
  </source>
  <target>
    <path>/dev/logical_pool</path>
  </target>
</pool>

volume.xml

<volume>
  <name>vol_logical_pool</name>
  <capacity unit="M">50</capacity>
  <allocation>0</allocation>
  <target>
    <path>
      /dev/logical_pool/vol_logical_pool
    </path>
  </target>
</volume>

The contents are different between vol_logical_pool and vol_test.
Comment 6 IBM Bug Proxy 2017-07-07 08:40:24 EDT
------- Comment From shivapbh@in.ibm.com 2017-07-07 08:37 EDT-------
On directory based pool after vol-download I don't see any difference between the original and downloaded volumes.

This is my host
# cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.4 (Pegas)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="7.4"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.4 Beta (Pegas)"
...
Libvirt - libvirt-3.2.0-15.el7a.ppc64le

On Comment 5, you mention the pool type is logical. Are you saying the issue was observed on pool backed on LVM ? What is the correlation to Comment 1 & 5? Can you reiterate the steps to reproduce as the steps mentioned in Comment 1 is not leading to the problem?

Thanks,
Shivaprasad
Comment 7 IBM Bug Proxy 2017-08-07 12:30:52 EDT
------- Comment From lagarcia@br.ibm.com 2017-08-07 12:24 EDT-------
Hello Red Hat,

Would you be able to answer the question from previous comment?
Comment 8 junli 2017-08-10 05:55 EDT
Created attachment 1311647 [details]
auto script
Comment 9 junli 2017-08-10 05:57 EDT
Created attachment 1311649 [details]
dir_pool incorrect log
Comment 10 junli 2017-08-10 05:58 EDT
Created attachment 1311650 [details]
logical_pool incorrect log
Comment 11 junli 2017-08-10 05:59:59 EDT
(In reply to IBM Bug Proxy from comment #6)
> ------- Comment From shivapbh@in.ibm.com 2017-07-07 08:37 EDT-------
> On directory based pool after vol-download I don't see any difference
> between the original and downloaded volumes.
> 
> This is my host
> # cat /etc/os-release
> NAME="Red Hat Enterprise Linux Server"
> VERSION="7.4 (Pegas)"
> ID="rhel"
> ID_LIKE="fedora"
> VERSION_ID="7.4"
> PRETTY_NAME="Red Hat Enterprise Linux Server 7.4 Beta (Pegas)"
> ...
> Libvirt - libvirt-3.2.0-15.el7a.ppc64le
> 
> On Comment 5, you mention the pool type is logical. Are you saying the issue
> was observed on pool backed on LVM ? What is the correlation to Comment 1 &
> 5? Can you reiterate the steps to reproduce as the steps mentioned in
> Comment 1 is not leading to the problem?
> 
> Thanks,
> Shivaprasad

This is my host
# cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.4 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.4"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.4 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.4:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.4"

# rpm -q libvirt qemu-kvm-rhev
libvirt-3.2.0-14.virtcov.el7_4.2.ppc64le
qemu-kvm-rhev-2.9.0-16.el7_4.3.ppc64le

And I feel so sorry about the reproducible is not correct.
I reproduce it by script, and dir_pool reproducible is about 10%, logical_pool reproducible is about 30%.
Finally, I add 3 attachments.
1. auto1460623.py is the script
2. dir_pool.log is the dir_pool incorrect log
3. logical_pool.log is the logical_pool incorrect log
Comment 12 IBM Bug Proxy 2017-11-16 11:15:12 EST
------- Comment From niteshkonkar@in.ibm.com 2017-11-16 11:08 EDT-------
Hello,

I figuered it out that this is the commit that is causing the bug.

commit b7d44f450c06803df7df3ad380f7a5c97425c1e6
Author: John Ferlan <jferlan@redhat.com>
Date:   Fri Mar 24 09:26:17 2017 -0400

storage: Fix capacity value for LUKS encrypted volumes

https://bugzilla.redhat.com/show_bug.cgi?id=1371892

The 'capacity' value (e.g. guest logical size) for a LUKS volume is
smaller than the 'physical' value of the file in the file system, so
we need to account for that.

When peeking at the encryption information about the volume add a fetch
of the payload_offset which is described as the offset to the start of
the volume data (in 512 byte sectors) in QEMU's QCryptoBlockLUKSHeader.

Then adjust the ->capacity appropriately when we determine that the
volume target encryption has a payload_offset value.

The following series seems to fix the issue.

https://www.redhat.com/archives/libvir-list/2016-April/msg01869.html

Thanks,
Nitesh Konkar.
Comment 13 Andrea Bolognani 2017-11-16 11:44:47 EST
The sparse stream changes you mention have been merged a while ago.

Does the issue still reproduce with a recent libvirt?
Comment 14 IBM Bug Proxy 2017-11-17 02:20:30 EST
------- Comment From niteshkonkar@in.ibm.com 2017-11-17 02:10 EDT-------
Hi,

I do not see the issue in the latest upstream libvirt.

1. virsh --version
3.10.0

2. cat /etc/os-release
PRETTY_NAME="Red Hat Enterprise Linux Server 7.4 Beta (Maipo)"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.4:beta:server"

REDHAT_SUPPORT_PRODUCT_VERSION="7.4 Beta"

4.hexdump  /var/lib/libvirt/images/dir_pool/vol_dir_pool.img
22600000

5. hexdump /var/lib/libvirt/images/dir_pool/vol_test
22600000
Comment 15 Hanns-Joachim Uhl 2017-11-17 03:48:28 EST
oops ... the previous comment has to read:
"
Hi,

I do not see the issue in the latest upstream libvirt. 

1. virsh --version
3.10.0

2. cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.4 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.4"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.4 Beta (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.4:beta:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.4 Beta"

3. virsh vol-download /var/lib/libvirt/images/dir_pool/vol_dir_pool.img /var/lib/libvirt/images/dir_pool/vol_test

4.hexdump  /var/lib/libvirt/images/dir_pool/vol_dir_pool.img 
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
22600000

5. hexdump /var/lib/libvirt/images/dir_pool/vol_test
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
22600000
"
...
Comment 16 Karen Noel 2017-11-29 20:14:38 EST
Andrea, Wenli, Can this one be closed?
Comment 17 Andrea Bolognani 2017-12-04 04:52:11 EST
Yup, moving it to POST now.

Note You need to log in before you can comment on or make changes to this bug.