Description of problem: VDSM should use zerofree to reduce the image size before cloning disks and (probably) collapsing disks.
(In reply to comment #0) > Description of problem: > VDSM should use zerofree to reduce the image size before cloning disks and > (probably) collapsing disks. I would recommend using virt-sparsify instead or in addition ... http://libguestfs.org/virt-sparsify.1.html Zerofree really needs to be properly supported by our filesystem team before we can start using it with customers. That's the reason why it's not in RHEL at the moment. There is large scope for it to do very bad things to customer data. Also virt-sparsify, although it's much less efficient than zerofree, has the advantage that it works with any type of filesystem. In particular with NTFS, so it can sparsify Windows guests, which obviously zerofree (being ext2/3-based) cannot. I'm planning to put virt-sparsify in RHEL 6.3.
Rich - any updated thinking/status on this?
The current status is that virt-sparsify (the "copying" version of it) is in RHEL 6.4, RHEL 7 and Fedora everything. In future we want to have a version of virt-sparsify which works in-place. This is something that might happen in RHEL 7.1. It requires changes to qemu, kernel and more, see: http://libguestfs.org/guestfs-faq.1.html#why-doesnt-virt-sparsify-work-on-the-disk-image-in-place- HTH
(In reply to comment #4) > http://libguestfs.org/guestfs-faq.1.html#why-doesnt-virt-sparsify-work-on- > the-disk-image-in-place- Grrr, this link doesn't work unless you manually add a '-' to the end of it ...
thoughts on impact vs. effort?
Impact - After a while of usage probably >10% possibly >50% (i.e. cut usage down by half!). Effort - since this is a cold process and it starts by copying all the data from source to target then in our case the flow would be: 1. create a target disk 2. virt-sparsify (in case of error, delete target) 3. if all went well, the rest depends on the use case. there are several use cases: A. Attach to VM and detach original disk (copy all vm-disk properties) - to allow user to test the new disk. User would have to manually delete the old disk. Need to make sure they cannot be attached concurrently to avoid conflicts. B. Make the disk into a template (no point in copying again) - i.e. virt-sparsify would be a checkbox when creating template C. Keep as floating disk to be used with another VM (i.e. virt-sparsify would be a checkbox in clone disk operation or something). Probably a lot more. Effort depends on which use cases we would like to implement. The one for same VM is probably the most complex due to the logic around the existing copy vs. new one.
(In reply to Ayal Baron from comment #7) > Effort - since this is a cold process and it starts by copying all the data > from source to target It depends on the backing store being used, but starting with RHEL 7 it should be possible to make virt-sparsify work entirely in-place. (Note that no one's actually done that: virt-sparsify in RHEL 7.0 still works by copying ... just saying it's possible). Adding Pino Toscano to CC as he is working on libguestfs things.
(In reply to Richard W.M. Jones from comment #8) > (In reply to Ayal Baron from comment #7) > > Effort - since this is a cold process and it starts by copying all the data > > from source to target > > It depends on the backing store being used, but starting with RHEL 7 > it should be possible to make virt-sparsify work entirely in-place. > (Note that no one's actually done that: virt-sparsify in RHEL 7.0 > still works by copying ... just saying it's possible). Not sure we'd want to do this considering the scope of changes and the risk of loosing data... However, it would be nice to split this out of virt-sparsify to allow doing more clever things (e.g. instead of full copy, use storage based snapshot which would be instantaneous and would not take up much space). > > Adding Pino Toscano to CC as he is working on libguestfs things.
The current patches, both those merges in VDSM and those not yet merged in the engine assume an SPM. It would be a shame to invest work in doing something that will have to be re-written in a couple of weeks/months. Let's wait till we have a proper SPM-less solution and then move on.
We want to continue development, but due to RHEL 7.3 dependency to make it useful I'm postponing this to 4.1. Shahar, also note the need to properly lock the whole chain while the operation is in progress
we're going with virt-sparsify only, majority of remaining work is on engine side
Michal, do we have a recommendation on when to perform this sparsification? for example actual size is bigger than the virtual size.
I would say, when virtual disk on storage takes significantly more space than used space shown by df command.
And of course you can determine that using virt-df: http://libguestfs.org/virt-df.1.html
Tried to test this feature on current upstream and couldn't get any results. According to smelamud in order for the feature to work libguestfs should have rw permissions to access libvirtd - If this is the case then we should either configure this from our side or make sure in documentation of the feature to explain what and how should be configured in order for this feature to work.
(In reply to sefi litmanovich from comment #18) > Tried to test this feature on current upstream and couldn't get any results. What did you try? How did you check that the sparsification yield any result? You can use `du -s` to check the actual size taken by a file -- usually raw files are sparse. > According to smelamud in order for the feature to work libguestfs should > have rw permissions to access libvirtd libguestfs needs a rw connection to be able to launch its own VMs using libvirt; you can always pass your own connection URI to libguestfs, using the -c parameter: -c qemu:///system, for example. If that is not feasible, you can always change the backend to the "direct" one (exporting LIBGUESTFS_BACKEND=direct). libguestfs in RHEL 7+ and Fedora 19+ is built to use libvirt by default to launch its own VMs.
see section about "Sparsifying a Virtual Disk" in the admin guide
From the man page of virt-sparsify: IMPORTANT LIMITATIONS The virtual machine must be shut down before using this tool. Am I misreading the feature, or shouldn't this warning be added to: http://www.ovirt.org/documentation/admin-guide/administration-guide/ ? Thanks Sven
(In reply to Sven Kieske from comment #21) > From the man page of virt-sparsify: > > IMPORTANT LIMITATIONS > > The virtual machine must be shut down before using this tool. This is absolutely true. However doesn't ovirt itself enforce this condition? (I certainly hope so ...)
(In reply to Richard W.M. Jones from comment #22) > > The virtual machine must be shut down before using this tool. > > This is absolutely true. However doesn't ovirt itself enforce > this condition? (I certainly hope so ...) Enforces, for sure.
(In reply to Pino Toscano from comment #19) > (In reply to sefi litmanovich from comment #18) > > Tried to test this feature on current upstream and couldn't get any results. > > What did you try? How did you check that the sparsification yield any result? > You can use `du -s` to check the actual size taken by a file -- usually raw > files are sparse. > I used du -sh on the storage domain in the image's folder. > > According to smelamud in order for the feature to work libguestfs should > > have rw permissions to access libvirtd > > libguestfs needs a rw connection to be able to launch its own VMs using > libvirt; you can always pass your own connection URI to libguestfs, using > the -c parameter: -c qemu:///system, for example. If that is not feasible, > you can always change the backend to the "direct" one (exporting > LIBGUESTFS_BACKEND=direct). > libguestfs in RHEL 7+ and Fedora 19+ is built to use libvirt by default to > launch its own VMs. If that's the prerequisite for this feature to work then we should make sure that this is configured as expected on the host, or add this as a comment in feature's documentation
To be clear, DO NOT use LIBGUESTFS_BACKEND=direct unless you fully understand the implications of that. libguestfs is already used by oVirt for various purposes and it seems to have no problem getting a r/w libvirt connection for those, so likely the problem you are seeing is something else. You said: > Tried to test this feature on current upstream and couldn't get any results. but I don't even understand what that means. What did you try? What error message did you see?
I tried to use the new 'sparsify' button added in this patch to ovirt-engine. What I get is that action is successful but took 0.01 secs so I guess nothing was done or there was some uncought error or something: from vdsm log: 2016-12-12 17:05:48,521 INFO (jsonrpc/0) [dispatcher] Run and protect: sdm_sparsify_volume(job_id=u'a245582e-eed4-4359-9d98-1d22b15faf19', vol_info={u'img_id': u'928afe69-b268-4d12-8979-c38b7c1b4da4', u'sd_id': u'5fd07c1e-d75a-4fbb-9cf5- a6aefaa712f1', u'vol_id': u'3cffb204-d1f5-49de-a0c9-656a2b0d22a2'}) (logUtils:49) 2016-12-12 17:05:48,523 INFO (jsonrpc/0) [dispatcher] Run and protect: sdm_sparsify_volume, Return response: None (logUtils:52) 2016-12-12 17:05:48,525 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call SDM.sparsify_volume succeeded in 0.01 seconds (__init__:515) After this I was talking with smelamud and tried to run 'libguestfs-test-tool' on my host and I get: libguestfs: opening libvirt handle: URI = qemu:///system, auth = default+wrapper, flags = 0 libvirt needs authentication to connect to libvirt URI qemu:///system (see also: http://libvirt.org/auth.html http://libvirt.org/uri.html) Please enter your authentication name: I do not know the authentication name and password. I now succeeded running 'libguestfs-test-tool' after setting: auth_unix_rw="none" in libvirtd.conf and export LIBGUESTFS_BACKEND=direct. then restarting libvirtd. But when I tried the sparsify option in the engine again I cannot see any change at file size using du -sh, vdsm reports successful run after 0.01 secs as it did the first time.
(In reply to sefi litmanovich from comment #26) > I tried to use the new 'sparsify' button added in this patch to ovirt-engine. > What I get is that action is successful but took 0.01 secs so I guess > nothing was done or there was some uncought error or something: > > from vdsm log: > > 2016-12-12 17:05:48,521 INFO (jsonrpc/0) [dispatcher] Run and protect: > sdm_sparsify_volume(job_id=u'a245582e-eed4-4359-9d98-1d22b15faf19', > vol_info={u'img_id': u'928afe69-b268-4d12-8979-c38b7c1b4da4', u'sd_id': > u'5fd07c1e-d75a-4fbb-9cf5- > a6aefaa712f1', u'vol_id': u'3cffb204-d1f5-49de-a0c9-656a2b0d22a2'}) > (logUtils:49) > 2016-12-12 17:05:48,523 INFO (jsonrpc/0) [dispatcher] Run and protect: > sdm_sparsify_volume, Return response: None (logUtils:52) > 2016-12-12 17:05:48,525 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC call > SDM.sparsify_volume succeeded in 0.01 seconds (__init__:515) I'm not totally familiar with how this is implemented in VDSM. Maybe this is just sending the start message over to the SDM where virt-sparsify actually is run. > After this I was talking with smelamud and tried to run > 'libguestfs-test-tool' on my host and I get: > > libguestfs: opening libvirt handle: URI = qemu:///system, auth = > default+wrapper, flags = 0 > libvirt needs authentication to connect to libvirt URI qemu:///system > (see also: http://libvirt.org/auth.html http://libvirt.org/uri.html) > Please enter your authentication name: > > I do not know the authentication name and password. This bit is not relevant. What you need to do is to run 'libguestfs-test-tool' on the SDM node, and run it as exactly the same user as VDSM with the same environment etc. Which is generally quite difficult for end users to do. > I now succeeded running 'libguestfs-test-tool' after setting: > auth_unix_rw="none" > in libvirtd.conf and export LIBGUESTFS_BACKEND=direct. > then restarting libvirtd. These settings are contradictory anyway. LIBGUESTFS_BACKEND=direct says "don't use libvirt", so changing libvirt's configuration makes no difference. In any case VDSM does use libvirt, and we want to keep it that way, see earlier comment. > But when I tried the sparsify option in the engine again I cannot see any > change at file size using du -sh, vdsm reports successful run after 0.01 > secs as it did the first time. So that's obviously a problem. I think you need to track down the log file from the SDM or SPM node to find out what really happened.
Shmuel, in order to document this feature, please list all limitations that should appear in the documentation. From what I have read, this should include 1. a list of NFS types supported and not supported. 2. Whether libguestfs should have rw permissions to access libvirtd or whether this has been preconfigured. I would also like to understand whether this was implemented as a button in the UI, as the feature page suggested.
Shmuel, in addition to my questions in Comment 30, please let me know whether Sparsify is supported in Storage Domains defined as Posix.
(In reply to emma heftman from comment #30) Unfortunately, the feature page for virt-sparsify is very outdated. I've wrote a new text, but it is still not approved. You can read it in my pull request here: https://github.com/oVirt/ovirt-site/pull/772/commits/7e4d76881572210f70f1ee21ca1b1e6d406ec92f Also I've demonstrated the feature and how it is implemented in Deep Dive: https://www.youtube.com/watch?v=ayseKlGLwHI&feature=share NFS supported is 4.2 or newer. libguestfs must have rw permissions to access libvirtd, without it virt-sparsify will not work. And yes, this should be configured automatically on host deploy, but I didn't verify this. The feature is implemented as "Sparsify" button on "Disks" subtab of "Virtual Machines" tab. About POSIX Storage Domain I'll answer later, need to recheck this.
(In reply to Shmuel Melamud from comment #32) > (In reply to emma heftman from comment #30) > Unfortunately, the feature page for virt-sparsify is very outdated. I've > wrote a new text, but it is still not approved. You can read it in my pull > request here: > https://github.com/oVirt/ovirt-site/pull/772/commits/ > 7e4d76881572210f70f1ee21ca1b1e6d406ec92f > > Also I've demonstrated the feature and how it is implemented in Deep Dive: > https://www.youtube.com/watch?v=ayseKlGLwHI&feature=share > > NFS supported is 4.2 or newer. > > libguestfs must have rw permissions to access libvirtd, without it > virt-sparsify will not work. And yes, this should be configured > automatically on host deploy, but I didn't verify this. > > The feature is implemented as "Sparsify" button on "Disks" subtab of > "Virtual Machines" tab. > > About POSIX Storage Domain I'll answer later, need to recheck this. Shmuel, I removed the needinfo from this bug and added it to the documentation bug: https://bugzilla.redhat.com/show_bug.cgi?id=1420197 there you can see Yaniv's answer and the remaining open issues.
It was decided that this RFE shouldn't be blocked by bz#1425161 which is a corner case. The basic tier-1 flows of this RFE are working fine and users can consume the feature easily. Test run: https://polarion.engineering.redhat.com/polarion/#/project/RHEVM3/testrun?id=16032017