Created attachment 1193160 [details] Error Description of problem: ======================= Testing the ceph 2.0 features on containers. Enabled the deep-flatten feature on the image and mapped it to a VM. The VM fails to boot with "Operation not supported message" Version-Release number of selected component (if applicable): ============================================================= 10.2.2-38.el7cp How reproducible: ================= 100% Steps to Reproduce: 1. Create rbd image with deep-flatten enabled 2. Map the image to a VM and start it. Actual results: Expected results: Additional info: Attaching the screenshot of the error message.
What do you mean by "map the image to a VM"? Are you just saying attach the image as a QEMU VM drive or are you mapping the image via krbd? Are all clients and OSDs at the same Ceph version? Can you provide logs?
My apologies for the very late reply. I meant attaching the image as a QEMU vm drive. All the client and OSD's have same ceph version. I will have to re-do to provide the logs. Will provide the logs at the earliest.
For clarity's sake, you have a containerized Ceph cluster deployed, then you are trying to boot VMs with specific Ceph features, correct?
Yes.. its the containerized ceph cluster.
Can you provide a copy of vagrant_variables.yml and group_vars/all, please.
I will provide the files shortly.
I have brought up the ceph cluster manually with 1 mon and 3 osd's. image is created with deep-flatten feature enabled and mapped it the virtual machine. Mon output: [root@dhcp41-220 /]# rbd info libvirt-pool/librbd rbd image 'librbd': size 102400 MB in 25600 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.1036238e1f29 format: 2 features: deep-flatten flags: [root@dhcp41-220 /]# Error when VM is started: Error starting domain: internal error: early end of file from monitor: possible problem: 2016-09-19T17:33:16.343831Z qemu-kvm: -drive file=rbd:libvirt-pool/librbd:id=libvirt:key=AQCYEOBX896oDhAAufuDMRq9eJtHelSSQGCfbA==:auth_supported=cephx\;none:mon_host=10.70.41.250\:6789,if=none,id=drive-virtio-disk1,format=raw: error reading header from librbd 2016-09-19T17:33:16.345895Z qemu-kvm: -drive file=rbd:libvirt-pool/librbd:id=libvirt:key=AQCYEOBX896oDhAAufuDMRq9eJtHelSSQGCfbA==:auth_supported=cephx\;none:mon_host=10.70.41.250\:6789,if=none,id=drive-virtio-disk1,format=raw: could not open disk image rbd:libvirt-pool/librbd:id=libvirt:key=AQCYEOBX896oDhAAufuDMRq9eJtHelSSQGCfbA==:auth_supported=cephx\;none:mon_host=10.70.41.250\:6789: Could not open 'rbd:libvirt-pool/librbd:id=libvirt:key=AQCYEOBX896oDhAAufuDMRq9eJtHelSSQGCfbA==:auth_supported=cephx\;none:mon_host=10.70.41.250\:6789': Function not implemented Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb callback(*args, **kwargs) File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 83, in newfn ret = fn(self, *args, **kwargs) File "/usr/share/virt-manager/virtManager/domain.py", line 1423, in startup self._backend.create() File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1029, in create if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self) libvirtError: internal error: early end of file from monitor: possible problem: 2016-09-19T17:33:16.343831Z qemu-kvm: -drive file=rbd:libvirt-pool/librbd:id=libvirt:key=AQCYEOBX896oDhAAufuDMRq9eJtHelSSQGCfbA==:auth_supported=cephx\;none:mon_host=10.70.41.250\:6789,if=none,id=drive-virtio-disk1,format=raw: error reading header from librbd 2016-09-19T17:33:16.345895Z qemu-kvm: -drive file=rbd:libvirt-pool/librbd:id=libvirt:key=AQCYEOBX896oDhAAufuDMRq9eJtHelSSQGCfbA==:auth_supported=cephx\;none:mon_host=10.70.41.250\:6789,if=none,id=drive-virtio-disk1,format=raw: could not open disk image rbd:libvirt-pool/librbd:id=libvirt:key=AQCYEOBX896oDhAAufuDMRq9eJtHelSSQGCfbA==:auth_supported=cephx\;none:mon_host=10.70.41.250\:6789: Could not open 'rbd:libvirt-pool/librbd:id=libvirt:key=AQCYEOBX896oDhAAufuDMRq9eJtHelSSQGCfbA==:auth_supported=cephx\;none:mon_host=10.70.41.250\:6789': Function not implemented VM boots if the feature is just layering. [root@dhcp41-220 /]# rbd info libvirt-pool/librbd rbd image 'librbd': size 102400 MB in 25600 objects order 22 (4096 kB objects) block_name_prefix: rbd_data.103b238e1f29 format: 2 features: layering flags: [root@dhcp41-220 /]#
Just to be sure features might be connected, can you apply the following in your ceph.conf? rbd default features = 61 This should give you the following features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
I have put this line in ceph.conf and mapped the rbd image to the VM. It goes to a paused state when trying to boot. Below is the error message shown : Error starting domain: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainCreate) Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper callback(asyncjob, *args, **kwargs) File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb callback(*args, **kwargs) File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 83, in newfn ret = fn(self, *args, **kwargs) File "/usr/share/virt-manager/virtManager/domain.py", line 1423, in startup self._backend.create() File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1029, in create if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self) libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainCreate)
I'm just not sure about this one, I'll try to bootstrap an OpenStack env and boot some VMs from a containerized Ceph.
I just built a setup for this, it consists of a DevStack running master and a containerized ceph using the ceph/demo container. I was able to successfully boot a VM with the following features: layering, exclusive-lock, object-map, fast-diff, deep-flatten In the end, I suspect you are experiencing a setup issue. Can you describe your setup? Thanks!
I am running the ceph cluster on atomic hosts. 1 mon and 3 osd's and all are running as containers. created a rbd image and mapped this to a VM in virt. While creating the rbd image given the feature as deep-flatten. When the VM boots it throws out an error. While plain enabling of features on the image does work.
How do you boot your vms?
From virt GUI. Tried CLI too but doesn't work.
No but is it libvirt+qemu-kvm?
yes..
@Bhaskarakiran: I am fairly sure that you have an old librbd1 RPM (0.94.x series) on your libvirt host. The default version that ships w/ base RHEL does not support the newer RHCS 2.0 features.
I will check that. Let me know the latest RPM location / repo so that i verify the same and close on this at the earlist.
Hmm yes.. i tried that as well with another librbd image. That too gave the same error / paused state.
closing per #26.
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days