Bug 651803

Can you tell me what 'qemu-img info /big/long/path' shows ?

# qemu-img info -f qcow2 /rhev/data-center/2c765f7a-2cf1-4025-a107-2c76d44af6a9/5235507a-a6c7-4ec1-a34e-4f7f754a1230/images/4a79ca7d-371d-4a9b-8f8d-a0010b005f26/8e714d9f-b83c-47b5-9d70-8dbae7f8ea13
image: /rhev/data-center/2c765f7a-2cf1-4025-a107-2c76d44af6a9/5235507a-a6c7-4ec1-a34e-4f7f754a1230/images/4a79ca7d-371d-4a9b-8f8d-a0010b005f26/8e714d9f-b83c-47b5-9d70-8dbae7f8ea13
file format: qcow2
virtual size: 15G (16106127360 bytes)
disk size: 0
cluster_size: 65536
backing file: ../4a79ca7d-371d-4a9b-8f8d-a0010b005f26/31e65ec0-6ccf-476e-b3cd-5b07a55809ec (actual path: /rhev/data-center/2c765f7a-2cf1-4025-a107-2c76d44af6a9/5235507a-a6c7-4ec1-a34e-4f7f754a1230/images/4a79ca7d-371d-4a9b-8f8d-a0010b005f26/../4a79ca7d-371d-4a9b-8f8d-a0010b005f26/31e65ec0-6ccf-476e-b3cd-5b07a55809ec)


# qemu-img info /rhev/data-center/2c765f7a-2cf1-4025-a107-2c76d44af6a9/5235507a-a6c7-4ec1-a34e-4f7f754a1230/images/4a79ca7d-371d-4a9b-8f8d-a0010b005f26/../4a79ca7d-371d-4a9b-8f8d-a0010b005f26/31e65ec0-6ccf-476e-b3cd-5b07a55809ec
image: /rhev/data-center/2c765f7a-2cf1-4025-a107-2c76d44af6a9/5235507a-a6c7-4ec1-a34e-4f7f754a1230/images/4a79ca7d-371d-4a9b-8f8d-a0010b005f26/../4a79ca7d-371d-4a9b-8f8d-a0010b005f26/31e65ec0-6ccf-476e-b3cd-5b07a55809ec
file format: raw
virtual size: 15G (16106127360 bytes)
disk size: 0

Capacity:       16106127360
Allocation:     11483217408
Physical:       4294967296

So capacity matches the qcow2 size as per qemu-img info, so that's correct.

Physical is approx 4GB, which i presume is roughly the amount of data that has been written into the qcow2 file.

Allocation is 10 GB which should be the size of the LVM volume. libvirt determines this by opening the file, and running seek(fd, 0, SEEK_END). The return value of seek is the new position, which should be matching the file/device size.

What does this say ?

perl -we 'open FILE, $ARGV[0]; seek FILE,0,2; print ((tell FILE), "\n");' /path/to/big/device.

Indeed,

# perl -we 'open FILE, $ARGV[0]; seek FILE,0,2; print ((tell FILE), "\n");' /rhev/data-center/8f343126-1ef5-4939-89c7-d0d69468298d/a21af405-3101-432f-9b80-bf28a8acf168/images/93074596-7266-465e-8e72-9f78860e4eb8/d2c85070-4ee6-448e-a93d-7a5e1335103a
1073741824

however,

# lvs |grep d2c85070-4ee6-448e-a93d-7a5e1335103a
  d2c85070-4ee6-448e-a93d-7a5e1335103a a21af405-3101-432f-9b80-bf28a8acf168 -wi-ao   2.06g

The domain is paused; when resumed, it paused again due to

remoteRelayDomainEventIOErrorReason:274 : Relaying domain io error fc-win7x86-2 3 /rhev/data-center/8f3
43126-1ef5-4939-89c7-d0d69468298d/a21af405-3101-432f-9b80-bf28a8acf168/images/93074596-7266-465e-8e72-9f78860e4eb8/d2c85070-4
ee6-448e-a93d-7a5e1335103a ide0-0-0 1 enospc

The interesting thing is that qemu is reporting much higher "wr_highest_offset": 2096758272.

qemuMonitorJSONIOProcessLine:115 : Line [{"return": [{"device": "drive-ide0-0-0", "parent": {"stats": {"wr_highest_offset": 2096758272, "wr_bytes": 2466782720, "wr_operations": 75607, "rd_bytes": 786583552, "rd_operations": 24030}}, "stats": {"wr_highest_offset": 10808802816, "wr_bytes": 2466782720, "wr_operations": 60029, "rd_bytes": 4941649408, "rd_operations": 151075}}, {"device": "drive-ide0-1-0", "stats": {"wr_highest_offset": 0, "wr_bytes": 0, "wr_operations": 0, "rd_bytes": 0, "rd_operations": 0}}, {"device": "drive-fdc0-0-0", "parent": {"stats": {"wr_highest_offset": 0, "wr_bytes": 0, "wr_operations": 0, "rd_bytes": 0, "rd_operations": 0}}, "stats": {"wr_highest_offset": 0, "wr_bytes": 0, "wr_operations": 0, "rd_bytes": 0, "rd_operations": 0}}]}]

It seems that qemu writes to qcow does not update the SEEK_END of the host block device.

I am raising the severity of this bug, as it blocks me from circumventing bug https://bugzilla.redhat.com/show_bug.cgi?id=659301#c16 in my bug 660162.

> # perl -we 'open FILE, $ARGV[0]; seek FILE,0,2; print ((tell FILE), "\n");'
> /rhev/data-center/8f343126-1ef5-4939-89c7-d0d69468298d/a21af405-3101-432f-9b80-bf28a8acf168/images/93074596-7266-465e-8e72-9f78860e4eb8/d2c85070-4ee6-448e-a93d-7a5e1335103a
> 1073741824
>
> however,
>
> # lvs |grep d2c85070-4ee6-448e-a93d-7a5e1335103a
>  d2c85070-4ee6-448e-a93d-7a5e1335103a a21af405-3101-432f-9b80-bf28a8acf168
> -wi-ao   2.06g

This is really bizarre, LVM is reporting the file is 2 GB, in size, but SEEK_END stops at 1 GB. The QEMU wr_highest_offset is approx 2 GB too, which is matching what LVM reports.

I don't really understand how SEEK_END can ever disagree with LVM about what the volume size is though. libvirt definitely relies on SEEK_END being correct, so whatever is making SEEK_END not work, is in turn causing libvirt to report confusing results.

Even kernel thinks the device is still 1 GB in size

# blockdev  --getsize64 /dev/a21af405-3101-432f-9b80-bf28a8acf168/d2c85070-4ee6-448e-a93d-7a5e1335103a
1073741824


So, somehow the kernel's view of the volume size, and LVM's view of the volume size are different. This in turn causes libvirt to report wrong data.

After doing a further lvextend + 1M on the virt node, the kernel + lvm view of the size is back in sync.  This combination of factors is suggesting that there is a bug in LVM where a resize performed on the SPM does not correctly propagate to the node running the VM(s).

This bug is filed against libvirt but has vdsm-4.9-43.el6 in Fixed In Version field and ON_QA. Igor, was the change you made intentional?

This isn't a libvirt bug, it should have been moved to vdsm.

lvextend now works system wide (vdsm-4.9-47.el6.x86_64, qemu-kvm-0.12.1.2-2.138.el6.x86_64, libvirt-0.8.7-5.el6.x86_64). 

scenario: 

1) run new domain (live CD) from rhevm with thin-provisioned disk (cow) of 20G
2) create file system over new disk, and mount it under /mnt
3) using dd, fill disk with zeros 
4) lv is extended (with additional 1G).