Bug 1371892

qemu-kvm-rhev-2.8.0-6.el7.x86_64
libvirt-3.1.0-2.el7.x86_64

hmm, currently with latest libvirt and qemu-kvm-rhev. Behaviours changed. 
Libvirt and qemu seem use different ways to calculate virtual disk size.
** created vol with qemu, and check size with virsh ** or
** created vol with virsh, and check size with qemu **
libvirt will give incorrect results, as follow:

1. create vol with qemu and check with virsh.
# qemu-img create -f luks --object secret,id=sec0,data=`printf %s "redhat" | base64`,format=base64 -o key-secret=sec0 /var/lib/libvirt/images/luks/qemu_create.luks 10M   **10M here**


## qemu-img info /var/lib/libvirt/images/luks/qemu_create.luks
image: /var/lib/libvirt/images/luks/qemu_create.luks
file format: luks
virtual size: 10M (10485760 bytes) **qemu-img info has correct size**



## virsh vol-info /var/lib/libvirt/images/luks/qemu_create.luks --bytes
Name:           qemu_create.luks
Type:           file
Capacity:       12554240 bytes   **here, the virsh vol-info gives a incorrect size**
Allocation:     262144 bytes


2. create vol with virsh vol-create
## cat vol.xml 
<volume>
        <name>luks_1.img</name>
        <capacity unit='M'>10</capacity>
        <target>
          <path>/var/lib/libvirt/images/luks/luks_1.img</path>
          <format type='raw'/>
          <encryption format='luks'>
             <secret type='passphrase' uuid='f981dd17-143f-45bc-88e6-ed1fe20ce9da'/>
          </encryption>        
         </target>
</volume>


## virsh vol-create luks vol.xml
virsh pool-Vol luks_1.img created from vol.xml


## virsh vol-info /var/lib/libvirt/images/luks/luks_1.img
Name:           luks_1.img
Type:           file
Capacity:       11.97 MiB   **not 10m as xml defined**
Allocation:     256.00 KiB

## qemu-img info /var/lib/libvirt/images/luks/luks_1.img 
image: /var/lib/libvirt/images/luks/luks_1.img
file format: luks
virtual size: 10M (10485760 bytes) **qemu gives right size**
disk size: 256K
encrypted: yes

Typically libvirt would use a stat() return of 'st_size' as the capacity of a raw file (such as what LUKS file is described as).  For qcow* encrypted files, further "peeking" would alter the capacity value based up the encrypted volumes view of the logical size - IOW: a metadata capacity is provided. For a LUKS volume there isn't a metadata capacity provided, instead the LUKS metadata provides the value of the "payload offset" - IOW: the size of the header for LUKS metadata. That value is provided as a number of 512 byte blocks that needs to be subtracted from the calculated capacity.


Posted a patch upstream:

http://www.redhat.com/archives/libvir-list/2017-March/msg01178.html

Patch has been pushed upstream

commit b7d44f450c06803df7df3ad380f7a5c97425c1e6
Author: John Ferlan <jferlan>
Date:   Fri Mar 24 09:26:17 2017 -0400

    storage: Fix capacity value for LUKS encrypted volumes
...
    The 'capacity' value (e.g. guest logical size) for a LUKS volume is
    smaller than the 'physical' value of the file in the file system, so
    we need to account for that.
    
    When peeking at the encryption information about the volume add a fetch
    of the payload_offset which is described as the offset to the start of
    the volume data (in 512 byte sectors) in QEMU's QCryptoBlockLUKSHeader.
    
    Then adjust the ->capacity appropriately when we determine that the
    volume target encryption has a payload_offset value.


$ git describe b7d44f450c06803df7df3ad380f7a5c97425c1e6
v3.2.0-21-gb7d44f4
$

testing summary:the vol-info is still not the same as 'qemu-img info'

version:
libvirt-3.2.0-10.el7.x86_64
qemu-kvm-rhev-2.9.0-10.el7.x86_64

steps:
1.create vol with 'virsh vol-create '

[root@lmen1 ~]# virsh secret-list 
 UUID                                  Usage
--------------------------------------------------------------------------------
 d48bf526-62c7-4a8b-88d6-ba8ad6286073  iscsi libvirtiscsi
 f981dd17-143f-45bc-88e6-ed1fe20ce9da  volume /var/lib/libvirt/images/luk.img


[root@lmen1 ~]# cat luk.vol 
<volume>
        <name>luk.img</name>
        <capacity unit='M'>1</capacity>
        <target>
          <path>/var/lib/libvirt/images/luk.img</path>
          <format type='raw'/>
          <encryption format='luks'>
             <secret type='passphrase' uuid='f981dd17-143f-45bc-88e6-ed1fe20ce9da'/>
             <cipher name='aes' size='256' mode='cbc' hash='sha256'/>
             <ivgen name='plain64' hash='sha256'/>
          </encryption>        
</target>
</volume>

[root@lmen1 ~]# virsh vol-create default luk.vol 
Vol luk.img created from luk.vol


2.check the size of the volume

[root@lmen1 ~]# virsh vol-info /var/lib/libvirt/images/luk.img
Name:           luk.img
Type:           file
Capacity:       2.00 MiB     --->2M
Allocation:     132.00 KiB


[root@lmen1 ~]# qemu-img info /var/lib/libvirt/images/luk.img
image: /var/lib/libvirt/images/luk.img
file format: luks
virtual size: 1.0M (1048576 bytes)    -->1M
disk size: 132K
encrypted: yes

3.check the size in the guest
start a guest with the disk xml:
 <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/luk.img'/>
      <target dev='sdb' bus='sata'/>
      <encryption format='luks'>
        <secret type='passphrase' uuid='f981dd17-143f-45bc-88e6-ed1fe20ce9da'/>
      </encryption>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>

[root@lmen1 ~]# virsh destroy q35;virsh start q35
Domain q35 destroyed
Domain q35 started

in the guest,
#fdisk -l
  Disk /dev/sdb: 1 MB, 1048576 bytes, 2048 sectors   --->1M
  Units = sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes

So,I want to know,why the result of 'virsh vol-info' is different from 'qemu-img info' and the info in the guest.

PS:
If I create a 20M volume using 'virsh vol-create',the result of 'virsh vol-info' is 21M,'qemu-img info' is 20M,the info in guest is 20M

I'm curious too, because I did test this before throwing it over the wall.

So I've added some debug code to the two places I changed:

virStorageSourceUpdateCapacity:
...
    if (src->encryption && src->encryption->payload_offset != -1) {
        VIR_WARN("src cap=%llu offset=%d",
                  src->capacity, src->encryption->payload_offset);
        src->capacity -= src->encryption->payload_offset * 512;
        VIR_WARN("src cap=%llu", src->capacity);
    }
...

storageBackendProbeTarget:
...
    if (encryption && meta->encryption) {
        VIR_WARN("encryption");
        if (meta->encryption->payload_offset != -1) {
            VIR_WARN("target cap=%llu offset=%d",
                     target->capacity, meta->encryption->payload_offset);
            target->capacity -= meta->encryption->payload_offset * 512;
            VIR_WARN("target cap=%llu", target->capacity);
        }
...


Then run...

# virsh vol-create default luk.xml
...
2017-06-15 15:30:51.240+0000: 16910: warning : virStorageSourceUpdateCapacity:3460 : src cap=2101248 offset=0
2017-06-15 15:30:51.240+0000: 16910: warning : virStorageSourceUpdateCapacity:3462 : src cap=2101248
...
Vol luk.img created from luk.xml

#  virsh vol-create default luk.xml
...
2017-06-15 15:30:55.619+0000: 16914: warning : virStorageSourceUpdateCapacity:3460 : src cap=2101248 offset=0
2017-06-15 15:30:55.619+0000: 16914: warning : virStorageSourceUpdateCapacity:3462 : src cap=2101248
...

Name:           luk.img
Type:           file
Capacity:       2101248 bytes
Allocation:     135168 bytes

#

< very strange, but interestingly enough... and I have more than one disk here, but the end result is the curious part >

# virsh pool-refresh default
...
2017-06-15 15:32:50.481+0000: 16912: warning : virStorageFileGetEncryptionPayloadOffset:953 : payload_offset=2056
2017-06-15 15:32:50.481+0000: 16912: warning : storageBackendProbeTarget:3489 : encryption
2017-06-15 15:32:50.481+0000: 16912: warning : storageBackendProbeTarget:3492 : target cap=22024192 offset=2056
2017-06-15 15:32:50.481+0000: 16912: warning : storageBackendProbeTarget:3494 : target cap=20971520
2017-06-15 15:32:50.481+0000: 16912: warning : virStorageFileGetEncryptionPayloadOffset:953 : payload_offset=2056
2017-06-15 15:32:50.481+0000: 16912: warning : storageBackendProbeTarget:3489 : encryption
2017-06-15 15:32:50.481+0000: 16912: warning : storageBackendProbeTarget:3492 : target cap=2101248 offset=2056
2017-06-15 15:32:50.481+0000: 16912: warning : storageBackendProbeTarget:3494 : target cap=1048576
2017-06-15 15:32:50.481+0000: 16912: warning : virStorageFileGetEncryptionPayloadOffset:953 : payload_offset=4040
2017-06-15 15:32:50.481+0000: 16912: warning : storageBackendProbeTarget:3489 : encryption
2017-06-15 15:32:50.481+0000: 16912: warning : storageBackendProbeTarget:3492 : target cap=23040000 offset=4040
2017-06-15 15:32:50.481+0000: 16912: warning : storageBackendProbeTarget:3494 : target cap=20971520
2017-06-15 15:32:50.481+0000: 16912: warning : virStorageFileGetEncryptionPayloadOffset:953 : payload_offset=4040
2017-06-15 15:32:50.481+0000: 16912: warning : storageBackendProbeTarget:3489 : encryption
2017-06-15 15:32:50.481+0000: 16912: warning : storageBackendProbeTarget:3492 : target cap=1075810304 offset=4040
2017-06-15 15:32:50.481+0000: 16912: warning : storageBackendProbeTarget:3494 : target cap=1073741824
2017-06-15 15:32:50.481+0000: 16912: warning : virStorageFileGetEncryptionPayloadOffset:953 : payload_offset=-1
2017-06-15 15:32:50.481+0000: 16912: warning : storageBackendProbeTarget:3489 : encryption
...
Pool default refreshed

# virsh vol-info /home/vm-images/luk.img --bytes 
...
2017-06-15 15:35:25.573+0000: 16913: warning : virStorageSourceUpdateCapacity:3460 : src cap=2101248 offset=2056
2017-06-15 15:35:25.573+0000: 16913: warning : virStorageSourceUpdateCapacity:3462 : src cap=1048576
...
Name:           luk.img
Type:           file
Capacity:       1048576 bytes
Allocation:     135168 bytes

#

Well that's the right answer!

So what change (needle in the haystack) caused the difference? Or did I perhaps run my tests using a debug session second and because of that causing the refresh to magically happen.

So short term, there is a work-around - just refresh the pool...

After a bit digging - it seems it is possible to fetch the correct value for the create, build, refresh volume code path. Typically for a RAW defined volume we'd just assign physical to capacity; however, for a RAW and Encrypted volume we can do better.

I've proposed a patch upstream:

https://www.redhat.com/archives/libvir-list/2017-June/msg00676.html

I didn't (yet) set this back to assigned - it's where we are in the release cycle that causes me to pause before ensuring I can get a quick approval and then backport appropriately (since there is a workaround of sorts).

do you have the schedule when to put the patch into libvirt pkg？
Will it be too late in 7.4 phase？ If it is too late,it is suggested to move this bug into 7.5

At this point we need special flags for 7.4, but I see other patches still getting included so who knows. All we can do is ask/try.  Still there is a work around at least w/r/t checking directly after creation (to do the refresh). Restarts, reboots, finding the volume in a pool "later" would also show the right size. It's just the case of checking the size after create before any sort of refresh occurs.

(In reply to John Ferlan from comment #9)
> At this point we need special flags for 7.4, but I see other patches still
> getting included so who knows. All we can do is ask/try.  Still there is a
> work around at least w/r/t checking directly after creation (to do the
> refresh). Restarts, reboots, finding the volume in a pool "later" would also
> show the right size. It's just the case of checking the size after create
> before any sort of refresh occurs.

Move back to Assign per above comment 6 and 7, and move to RHEL7.5, since it seems the patch in comment 7 is still in reviewing phase till now, can not be backport to the downstream in current release cycle.

Pushed upstream:

commit 5431055d2b833463b25269cdb486dd08ef790097
Author: John Ferlan <jferlan>
Date:   Thu Jun 15 14:32:43 2017 -0400

    util: Force reading of meta data to get encryption capacity value
    
...

    As it turns out the volume create, build, and refresh path was not peeking
    at the meta data, so immediately after a create operation the value displayed
    for capacity was still incorrect. However, if a pool refresh was done the
    correct value was fetched as a result of a meta data peek.
    
    The reason is it seems historically if the file type is RAW then peeking
    at the file just took the physical value for the capacity. However, since
    we know if it's an encrypted file, then peeking at the meta data will be
    required in order to get a true capacity value.
    
    So check for encryption in the source and if present, use the meta data
    in order to fill in the capacity value and set the payload_offset.

...

$ git describe 5431055d2b833463b25269cdb486dd08ef790097
v3.4.0-168-g5431055
$

verify the bug

version:
libvirt-3.9.0-5.el7.x86_64
qemu-kvm-rhev-2.10.0-11.el7.x86_64


scenario1: create 1M vol with virsh vol-create

[root@localhost ~]# cat luks.xml 
<secret ephemeral='no' private='yes'>
  <uuid>f981dd17-143f-45bc-88e6-ed1fe20ce9da</uuid>
  <description>LUKS Sample Secret</description>
  <usage type='volume'>
    <volume>/var/libvirt/images/luks.img</volume>
  </usage>
</secret>

[root@localhost ~]# virsh secret-define luks.xml 
Secret f981dd17-143f-45bc-88e6-ed1fe20ce9da created


[root@localhost ~]# virsh secret-list
 UUID                                  Usage
--------------------------------------------------------------------------------
 a1d92516-1f84-4be8-9cf8-d84efbfdd4b1  iscsi libvirtiscsi
 f981dd17-143f-45bc-88e6-ed1fe20ce9da  volume /var/libvirt/images/luks.img

[root@localhost ~]#  MYSECRET=`echo "redhat" | base64`
[root@localhost ~]# virsh secret-set-value f981dd17-143f-45bc-88e6-ed1fe20ce9da $MYSECRET
Secret value set


[root@localhost ~]# cat luks-vol.xml 
<volume>
        <name>luks.img</name>
        <capacity unit='M'>1</capacity>
        <target>
          <path>/var/lib/libvirt/images/luks.img</path>
          <format type='raw'/>
          <encryption format='luks'>
             <secret type='passphrase' uuid='f981dd17-143f-45bc-88e6-ed1fe20ce9da'/>
             <cipher name='aes' size='256' mode='cbc' hash='sha256'/>
             <ivgen name='plain64' hash='sha256'/>
          </encryption>        
</target>
</volume>


[root@localhost ~]# virsh vol-create default luks-vol.xml
Vol luks.img created from luks-vol.xml


[root@localhost ~]# virsh vol-info /var/lib/libvirt/images/luks.img --bytes
Name:           luks.img
Type:           file
Capacity:       1048576 bytes
Allocation:     135168 bytes


[root@localhost ~]# qemu-img info /var/lib/libvirt/images/luks.img
image: /var/lib/libvirt/images/luks.img
file format: luks
virtual size: 1.0M (1048576 bytes)
disk size: 132K



scenario2:create 10M vol with virsh vol-create

[root@localhost ~]# cat luks-vol1.xml
<volume>
        <name>luks1.img</name>
        <capacity unit='M'>10</capacity>
        <target>
          <path>/var/lib/libvirt/images/luks1.img</path>
          <format type='raw'/>
          <encryption format='luks'>
             <secret type='passphrase' uuid='f981dd17-143f-45bc-88e6-ed1fe20ce9da'/>
             <cipher name='aes' size='256' mode='cbc' hash='sha256'/>
             <ivgen name='plain64' hash='sha256'/>
          </encryption>        
</target>
</volume>


[root@localhost ~]# virsh vol-create default luks-vol1.xml
Vol luks1.img created from luks-vol1.xml


[root@localhost ~]# virsh vol-info /var/lib/libvirt/images/luks1.img --bytes
Name:           luks1.img
Type:           file
Capacity:       10485760 bytes
Allocation:     135168 bytes

[root@localhost ~]# qemu-img info /var/lib/libvirt/images/luks1.img
image: /var/lib/libvirt/images/luks1.img
file format: luks
virtual size: 10M (10485760 bytes)
disk size: 132K



scenario3: create 10M vol with qemu and check with virsh



[root@localhost ~]# qemu-img create -f luks --object secret,id=sec0,data=`printf %s "redhat" | base64`,format=base64 -o key-secret=sec0 /var/lib/libvirt/images/luks2.img 10M
Formatting '/var/lib/libvirt/images/luks2.img', fmt=luks size=10485760 key-secret=sec0


[root@localhost ~]# qemu-img info /var/lib/libvirt/images/luks2.img
image: /var/lib/libvirt/images/luks2.img
file format: luks
virtual size: 10M (10485760 bytes)
disk size: 256K

[root@localhost ~]# virsh pool-refresh default
Pool default refreshed


[root@localhost ~]# virsh vol-info /var/lib/libvirt/images/luks2.img 
Name:           luks2.img
Type:           file
Capacity:       10.00 MiB
Allocation:     256.00 KiB

in the above 3 scenarios,I also check the disks' size in the guest,the size is right.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:0704