Bug 1139707

It seems that since rhel7 we'll have to use compat=0.10 on image creation.

Since the "compat" option is not present in rhel6 either we decide to use it only with newer qemu-img binaries or we request to backport the option to rhel6 as well.

Ademar what do you think, can you clone this bz to qemu-kvm rhel6 to add the support of the option?

(In reply to Federico Simoncelli from comment #2)
> It seems that since rhel7 we'll have to use compat=0.10 on image creation.
> 
> Since the "compat" option is not present in rhel6 either we decide to use it
> only with newer qemu-img binaries or we request to backport the option to
> rhel6 as well.
> 
> Ademar what do you think, can you clone this bz to qemu-kvm rhel6 to add the
> support of the option?

Can you check what qemu-img version is present and then use compat=0.10 only when it's an old version (below qemu-1.1)? This is the libvirt way of dealing with old qemu versions and that's what we encourage.

Old qemu-img versions create images in the 0.10 format anyway, so a command line option would be a NOP and will be confusing for users who run it by hand (as it won't support compat=1.1).

Now verified a workaround:

Locate the problematic image file (sadly this isn't that easy to do - I had to search logs for that), then convert it to the 0.10 format, on the rhel7 host, with:

qemu-img convert -O qcow2 -o compat=0.10 $src $tempdest
mv $tempdest $src

(In reply to Yedidyah Bar David from comment #6)
> qemu-img convert -O qcow2 -o compat=0.10 $src $tempdest
> mv $tempdest $src

You can also use 'qemu-img amend -f qcow2 -o compat=0.10 $src', which is a much
quicker operation and doesn't involve copying data.

(In reply to Ademar Reis from comment #5)
> (In reply to Federico Simoncelli from comment #2)
> Can you check what qemu-img version is present and then use compat=0.10 only
> when it's an old version (below qemu-1.1)? This is the libvirt way of
> dealing with old qemu versions and that's what we encourage.

Actually, libvirt wouldn't check the version number, but check whether compat
is in the list printed by 'qemu-img create -f qcow2 -o "?" dummy', which works
more reliably in downstreams that backport random features.

(In reply to Kevin Wolf from comment #8)
> Actually, libvirt wouldn't check the version number, but check whether compat
> is in the list printed by 'qemu-img create -f qcow2 -o "?" dummy', which
> works
> more reliably in downstreams that backport random features.

Can promise that this api will be more stable then the image format?

(In reply to Kevin Wolf from comment #8)
> (In reply to Yedidyah Bar David from comment #6)
> > qemu-img convert -O qcow2 -o compat=0.10 $src $tempdest
> > mv $tempdest $src
> 
> You can also use 'qemu-img amend -f qcow2 -o compat=0.10 $src', which is a
> much
> quicker operation and doesn't involve copying data.
> 
> (In reply to Ademar Reis from comment #5)
> > (In reply to Federico Simoncelli from comment #2)
> > Can you check what qemu-img version is present and then use compat=0.10 only
> > when it's an old version (below qemu-1.1)? This is the libvirt way of
> > dealing with old qemu versions and that's what we encourage.
> 
> Actually, libvirt wouldn't check the version number, but check whether compat
> is in the list printed by 'qemu-img create -f qcow2 -o "?" dummy', which
> works
> more reliably in downstreams that backport random features.

Nir, can we bang out a quick fix based on this?

(In reply to Allon Mureinik from comment #10)
> (In reply to Kevin Wolf from comment #8)
Waiting for confirmation from Kevin about the stability of this api.

Kevin, qemu-img manual say:

       options
           is a comma separated list of format specific options in a
           name=value format. Use "-o ?" for an overview of the options
           supported by the used format or see the format descriptions
           below for details.

Running this produce this output:

$ qemu-img create -f qcow2 -o "?" dummy
Supported options:
size             Virtual disk size
compat           Compatibility level (0.10 or 1.1)
backing_file     File name of a base image
backing_fmt      Image format of the base image
encryption       Encrypt the image
cluster_size     qcow2 cluster size
preallocation    Preallocation mode (allowed values: off, metadata)
lazy_refcounts   Postpone refcount updates

But there is no specification of the output of this command ("-o ?"). This means that we depend on parsing undocumented format to understand the capabilities fo qemu-img.

For example, this does not give the list of compat values availale (they are listed in a user readable form, not in a machine readable form).

I don't think we can depend on it.

(In reply to Nir Soffer from comment #9)
> Can promise that this api will be more stable then the image format?

We're generally very careful to only extend APIs and not take things away. And
we're aware that in some places, output originally meant for humans is parsed
by management tools because it's the only way to get information. In such cases
we treat this as a stable API.

libvirt is doing a lot of things like this. If you're not confident enough to
have such checks in RHEV-M code, perhaps libvirt can offer the functionality
that you need and will abstract away any changes in qemu. I'm copying Eric
Blake for this

(In reply to Kevin Wolf from comment #8)

> (In reply to Ademar Reis from comment #5)
> > (In reply to Federico Simoncelli from comment #2)
> > Can you check what qemu-img version is present and then use compat=0.10 only
> > when it's an old version (below qemu-1.1)? This is the libvirt way of
> > dealing with old qemu versions and that's what we encourage.
> 
> Actually, libvirt wouldn't check the version number, but check whether compat
> is in the list printed by 'qemu-img create -f qcow2 -o "?" dummy', which
> works
> more reliably in downstreams that backport random features.

If you use libvirt to create the files, libvirt ALREADY supports the notion of requesting a compat level of 0.10 in a way that works with both old qemu (omitted to qemu-img, because it was unsupported but the default) and new qemu (supplied to qemu-img, to force back-compat behavior).

http://libvirt.org/formatstorage.html#StorageVolTarget documents the <compat> tag to virDomainStorageVolCreateXML

Here's how libvirt portably probes for whether -o compat works:

http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/storage/storage_backend.c;h=00cfe74c8824912811ff153b5d34082b06d578c7;hb=HEAD#l672

If this is good enough for libvirt it should be good enough for VDSM.

Thanks Kevin and Eric, we will use the same check as libvirt does.

Didi, can you check the attached patch on your environment?

Kevin, woul you like to review the attach patch?
http://gerrit.ovirt.org/32836

Looks good to me.

How to verify:

1. Setup a cluster with rhel7 and rhel6.5 hosts
2. Activate rhel7 host
3. Create vm with one disk using thin provisioning
4. Put host to maintenance
5. Start rhel6 host
6. Start vm
7. create live sanpshot

Additional verification:
Check that image is using qcow2 comapt=0.10 (on rhel7 machine):
qemu-img info /path/to/volume

Kevin do you know if "create" is the only command affected by this?

For example what about "convert" and "rebase"? E.g.:

$ qemu-img convert -f qcow2 -O qcow2 input_0.10_image.qcow2 output.qcow2

Is "convert" going to use compat=0.10 as the source or is it going to use 1.1 (when available)?

What about rebase?

Is there any guarantee that the compat version in input is always maintained in the output?

Thanks.

(In reply to Federico Simoncelli from comment #23)
> Kevin do you know if "create" is the only command affected by this?

The rule of thumb is that anything that creates a new image gets the same
defaults as qemu-img create. This includes convert (the target image is newly
created, except with -n), but not rebase (all images already exist and are only
modified). Consequently, convert does have an -o option and rebase doesn't.

It also potentially includes things like live snapshots, though I think you
don't make use of qemu's functionality to create a new image there, but rather
invoke qemu-img create manually.

(In reply to Kevin Wolf from comment #25)
> (In reply to Federico Simoncelli from comment #23)
> > Kevin do you know if "create" is the only command affected by this?
> 
> The rule of thumb is that anything that creates a new image gets the same
> defaults as qemu-img create. This includes convert (the target image is newly
> created, except with -n), but not rebase (all images already exist and are
> only
> modified). Consequently, convert does have an -o option and rebase doesn't.

Nir you may want to check this because it could be that this bz is not fully addressed by the current patch ON_QA.

Even if, by any chance, it is fixed in practice (glitch in "convert")... I trust Kevin when he says that "anything that creates a new image gets the same defaults as qemu-img create".

I checked qemu-img on Fedora 20. According to the online help, only create, convert and amend have a -o options parameter.

$ qemu-img convert -f qcow2 -o ?
Supported options:
size             Virtual disk size

$ qemu-img amend -f qcow2 -o ?
Supported options:
size             Virtual disk size
compat           Compatibility level (0.10 or 1.1)
backing_file     File name of a base image
backing_fmt      Image format of the base image
encryption       Encrypt the image
cluster_size     qcow2 cluster size
preallocation    Preallocation mode (allowed values: off, metadata)
lazy_refcounts   Postpone refcount updates

$ rpm -q qemu-img
qemu-img-2.1.1-1.fc20.x86_64

So I think that we should use the same logic when running these commands, and add a "-o compat=0.10" if compat is supported by the command. It seems that convert does not support this option currently, but if it does support this option, we will be covered.

Kevin, can you confirm that this is the right way to handle this?

Let me correct my self - convert do support compat if -O qcow2 is specified:

$ qemu-img convert -f qcow2 -O qcow2 -o ?
Supported options:
size             Virtual disk size
compat           Compatibility level (0.10 or 1.1)
backing_file     File name of a base image
backing_fmt      Image format of the base image
encryption       Encrypt the image
cluster_size     qcow2 cluster size
preallocation    Preallocation mode (allowed values: off, metadata)
lazy_refcounts   Postpone refcount updates

(In reply to Nir Soffer from comment #27)
> So I think that we should use the same logic when running these commands,
> and add a "-o compat=0.10" if compat is supported by the command. It seems
> that convert does not support this option currently, but if it does support
> this option, we will be covered.
> 
> Kevin, can you confirm that this is the right way to handle this?

Yes for create and convert (you noticed -O in convert yourself).

The case is a bit different with amend, because that doesn't create a new
image, but rather updates an existing one. If you don't specify compat there,
it will leave the image format version as it is. If you do specify it, you can
upgrade/downgrade existing images.

I managed to create a snapshot for a VM that was started on a RHEL6.5 host with 1 disk created while the SPM was a RHEL7 host. (according to comment #22)

Verified using vt4

*** Bug 1168958 has been marked as a duplicate of this bug. ***

(In reply to Kevin Wolf from comment #8)
> ...
> You can also use 'qemu-img amend -f qcow2 -o compat=0.10 $src', which is a
> much
> quicker operation and doesn't involve copying data.
> 

Thanks! So the way to fix the images (on iSCSI SD) for me was to:
1) shutdown all VMs, put hosts to maintenance
2) on .el7 host, activate all LVs on the respective domain:
    lvchange -ay SD_UUID
3) look at images info:
    for LV in /rhev/data-center/<DC_UUID>/<SD_UUID>/images/*/* ; do qemu-img info $LV ; done
4) convert images from previous point with qcow2 format and "compat: 1.1"
    for LV in LV1 ... LVi ; do qemu-img amend -f qcow2 -o compat=0.10 $LV ; done
5) verify format:
    for LV in LV1 ... LVi ; do qemu-img info $LV ; done
6) deactivate LVs again:
    lvchange -an /dev/f3897d4d-b938-45f1-b8ac-ff6a1dd8b509/[0-9a-f]*
7) activate the hosts again. Now the VMs can start on RHEL 6 hosts again

It would be nice to automate this procedure though.

(In reply to David Jaša from comment #33)
> (In reply to Kevin Wolf from comment #8)
> > ...
> > You can also use 'qemu-img amend -f qcow2 -o compat=0.10 $src', which is a
> > much
> > quicker operation and doesn't involve copying data.
> > 
> 
> Thanks! So the way to fix the images (on iSCSI SD) for me was to:
> 1) shutdown all VMs, put hosts to maintenance
> 2) on .el7 host, activate all LVs on the respective domain:
>     lvchange -ay SD_UUID
> 3) look at images info:
>     for LV in /rhev/data-center/<DC_UUID>/<SD_UUID>/images/*/* ; do qemu-img
> info $LV ; done
> 4) convert images from previous point with qcow2 format and "compat: 1.1"
>     for LV in LV1 ... LVi ; do qemu-img amend -f qcow2 -o compat=0.10 $LV ;
> done
> 5) verify format:
>     for LV in LV1 ... LVi ; do qemu-img info $LV ; done
> 6) deactivate LVs again:
>     lvchange -an /dev/f3897d4d-b938-45f1-b8ac-ff6a1dd8b509/[0-9a-f]*
> 7) activate the hosts again. Now the VMs can start on RHEL 6 hosts again
> 
> It would be nice to automate this procedure though.
Agreed.

Marina - Customers who used RHEL 7 with RHEV 3.4 prior to 3.4.4 may have encountered this issue. 
Perhaps GSS could provide a tool/kbase for this?

Allon, thank you.
Will do.

https://access.redhat.com/solutions/1284573

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0159.html