Up until now, qemu-kvm has had a dependency on the qemu-block-curl plugin, which allows qemu guests to use http/https network locations as a disk source. However, in rhel 9, this dependency is being removed from qemu-kvm, which results in issues such as bug #2014229. The long-term solution is to switch to using the more featureful and maintained nbdkit curl plugin (https://libguestfs.org/nbdkit-curl-plugin.1.html) within libvirt. From an email from Rich: It would need to instead run an nbdkit subprocess first, eg: nbdkit -U /tmp/randomly-named-sock --exit-with-parent -f -r \ --pidfile /tmp/randomly-named-file \ curl http://foo protocols=http,https,ftp,ftps & # wait for pidfile to get created # pass nbd:unix:/tmp/randomly-named-sock as qemu parameter # or: # <disk type='network' device='disk'> # <driver name='qemu'/> # <source protocol='nbd'> # <host transport='unix' socket='/tmp/randomly-named-sock'/> # </source> # <target dev='vda' bus='virtio'/> # </disk> # wait for qemu then: kill `cat /tmp/randomly-named-file` rm /tmp/randomly-named-sock (or nbdkit will clean itself up because of --exit-with-parent)
Is this going to be transparent in terms of libvirt XML, or do we plan to model this through the XML? In other words would we just want to transparently change the implementation of: <disk type='network'> <source protocol="https" name="url_path"> ... or would we model it like: <disk type='network'> <source protocol="external"> <program>nbdkit</program> <args>... (or however that would work)
(In reply to Richard W.M. Jones from comment #1) > Is this going to be transparent in terms of libvirt XML, or > do we plan to model this through the XML? In other words > would we just want to transparently change the implementation of: > > <disk type='network'> > <source protocol="https" name="url_path"> > ... If we want to transparently fix the issue of the curl driver being removed from qemu we need to switch to nbdkit when it's not present with identical configuration. Otherwise the higher level APP would need to change their behaviour to ... > or would we model it like: > > <disk type='network'> > <source protocol="external"> > <program>nbdkit</program> > <args>... > > (or however that would work) ... use this new XML. It would make sense to start nbdkit separately, but generally we try to avoid giving the user the ability to specify arbitrary arguments for a program as it doesn't give us a chance to abstract changes in the helper program. In this instance the only thing the user will get is security isolation.
Sure thing, I agree. By the way if the nbdkit support is generic enough then it could be used to enable VDDK support using nbdkit-vddk-plugin, eg: <disk type='network'> <source protocol="vddk" file="[datastore1] Fedora/Fedora.vmdk"> <host name="esxi"/> <auth username='root'/> <libdir>/opt/vmware-vix-disklib-distrib</libdir> </source> ... </disk> https://libguestfs.org/nbdkit-vddk-plugin.1.html
So, Rich's comment makes me wonder -- are http and https the only protocols we are currently planning to move up to the libvirt level? Or will there be more dependencies removed from qemu-kvm that will cause similar issues for other protocols (e.g. ssh, etc)?
ssh is likely too IMHO. We've spent less time on nbdkit-ssh-plugin, but it is still slightly more featureful than the qemu driver, and because of (low) performance characteristics makes sense to move out of qemu too.
oh, indeed the ssh plugin was actually removed as a dependency from qemu in rhel 9 as well, I thought it was only curl so far. Updating title.
(In reply to Jonathon Jongsma from comment #4) > So, Rich's comment makes me wonder -- are http and https the only protocols > we are currently planning to move up to the libvirt level? Or will there be > more dependencies removed from qemu-kvm that will cause similar issues for > other protocols (e.g. ssh, etc)? Yes, curl and ssh protocols are now optional (not required by qemu-kvm).
(In reply to Richard W.M. Jones from comment #3) > Sure thing, I agree. By the way if the nbdkit support is generic enough > then it could be used to enable VDDK support using nbdkit-vddk-plugin The more generic support sounds like something that could maybe supported by the future XML syntax that enables using a storage daemon, because nbdkit essentially just functions as a limited storage daemon in this case. Limited because it doesn't have QMP or anything like that, so I suppose certain features like storage migration or taking snapshots might not be possible when it's used. @pkrempa This sounds like something to discuss in our planned storage daemon meeting.
The direct answer to how to do snapshotting (etc) is that you'd do it in exactly the same way as snapshotting any NBD device, ie. qemu or q-s-d does it. It's entirely conceivable that q-s-d would be involved here. The point of this change is to deal with 3 or 4 different problems: (1a) Some drivers in qemu are not really maintained (eg. ssh), and the nbdkit equivalent is both maintained and far more featureful. (1b) nbdkit-curl-plugin is also much more featureful, and in particular can do cookie and header renewal which is essential for accessing certain non-public webservices like docker registries. (2) Some drivers cannot be in qemu for licensing reasons (eg. vddk). (3) Some drivers cannot be in qemu for technical reasons (eg. vddk because it requires a re-exec(2) to reinitialize the process, the imageio driver which is written in Python). (4) We'd like to apply fine-grained SELinux policies per block driver.
(In reply to Richard W.M. Jones from comment #9) > The direct answer to how to do snapshotting (etc) is that you'd do it > in exactly the same way as snapshotting any NBD device, ie. qemu or q-s-d > does it. It's entirely conceivable that q-s-d would be involved here. Right, so libvirt may need to support one storage daemon (nbdkit) serving another storage daemon (qsd), and also accessing the external process not for a whole disk image, but just for a backing file. That's a pretty important data point when designing how libvirt manages things. > (1a) Some drivers in qemu are not really maintained (eg. ssh), and the > nbdkit equivalent is both maintained and far more featureful. It's indeed a sad truth that we decided to duplicate some efforts with nbdkit, inevitably leading to one implementation being less actively maintained than the other. But what you're using it for is not really relevant for the point I was making, which is about a generic way to use storage daemons (potentially including nbdkit) from libvirt.
v3 on the list, https://listman.redhat.com/archives/libvir-list/2022-October/235052.html