Bug 1369323

What do you mean by "map the image to a VM"? Are you just saying attach the image as a QEMU VM drive or are you mapping the image via krbd? Are all clients and OSDs at the same Ceph version? Can you provide logs?

My apologies for the very late reply. I meant attaching the image as a QEMU vm drive. All the client  and OSD's have same ceph version. I will have to re-do to provide the logs. Will provide the logs at the earliest.

For clarity's sake, you have a containerized Ceph cluster deployed, then you are trying to boot VMs with specific Ceph features, correct?

Yes.. its the containerized ceph cluster.

Can you provide a copy of vagrant_variables.yml and group_vars/all, please.

I will provide the files shortly.

I have brought up the ceph cluster manually with 1 mon and 3 osd's. image is created with deep-flatten feature enabled and mapped it the virtual machine. 

Mon output:

[root@dhcp41-220 /]# rbd info libvirt-pool/librbd
rbd image 'librbd':
	size 102400 MB in 25600 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.1036238e1f29
	format: 2
	features: deep-flatten
	flags: 
[root@dhcp41-220 /]# 

Error when VM is started:

Error starting domain: internal error: early end of file from monitor: possible problem:
2016-09-19T17:33:16.343831Z qemu-kvm: -drive file=rbd:libvirt-pool/librbd:id=libvirt:key=AQCYEOBX896oDhAAufuDMRq9eJtHelSSQGCfbA==:auth_supported=cephx\;none:mon_host=10.70.41.250\:6789,if=none,id=drive-virtio-disk1,format=raw: error reading header from librbd
2016-09-19T17:33:16.345895Z qemu-kvm: -drive file=rbd:libvirt-pool/librbd:id=libvirt:key=AQCYEOBX896oDhAAufuDMRq9eJtHelSSQGCfbA==:auth_supported=cephx\;none:mon_host=10.70.41.250\:6789,if=none,id=drive-virtio-disk1,format=raw: could not open disk image rbd:libvirt-pool/librbd:id=libvirt:key=AQCYEOBX896oDhAAufuDMRq9eJtHelSSQGCfbA==:auth_supported=cephx\;none:mon_host=10.70.41.250\:6789: Could not open 'rbd:libvirt-pool/librbd:id=libvirt:key=AQCYEOBX896oDhAAufuDMRq9eJtHelSSQGCfbA==:auth_supported=cephx\;none:mon_host=10.70.41.250\:6789': Function not implemented

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 83, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1423, in startup
    self._backend.create()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1029, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: internal error: early end of file from monitor: possible problem:
2016-09-19T17:33:16.343831Z qemu-kvm: -drive file=rbd:libvirt-pool/librbd:id=libvirt:key=AQCYEOBX896oDhAAufuDMRq9eJtHelSSQGCfbA==:auth_supported=cephx\;none:mon_host=10.70.41.250\:6789,if=none,id=drive-virtio-disk1,format=raw: error reading header from librbd
2016-09-19T17:33:16.345895Z qemu-kvm: -drive file=rbd:libvirt-pool/librbd:id=libvirt:key=AQCYEOBX896oDhAAufuDMRq9eJtHelSSQGCfbA==:auth_supported=cephx\;none:mon_host=10.70.41.250\:6789,if=none,id=drive-virtio-disk1,format=raw: could not open disk image rbd:libvirt-pool/librbd:id=libvirt:key=AQCYEOBX896oDhAAufuDMRq9eJtHelSSQGCfbA==:auth_supported=cephx\;none:mon_host=10.70.41.250\:6789: Could not open 'rbd:libvirt-pool/librbd:id=libvirt:key=AQCYEOBX896oDhAAufuDMRq9eJtHelSSQGCfbA==:auth_supported=cephx\;none:mon_host=10.70.41.250\:6789': Function not implemented


VM boots if the feature is just layering.

[root@dhcp41-220 /]# rbd info libvirt-pool/librbd
rbd image 'librbd':
	size 102400 MB in 25600 objects
	order 22 (4096 kB objects)
	block_name_prefix: rbd_data.103b238e1f29
	format: 2
	features: layering
	flags: 
[root@dhcp41-220 /]#

Just to be sure features might be connected, can you apply the following in your ceph.conf?

rbd default features = 61

This should give you the following features: layering, exclusive-lock, object-map, fast-diff, deep-flatten

I have put this line in ceph.conf and mapped the rbd image to the VM. It goes to a paused state when trying to boot.  Below is the error message shown :

Error starting domain: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainCreate)

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 89, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 125, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/libvirtobject.py", line 83, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1423, in startup
    self._backend.create()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1029, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainCreate)

I'm just not sure about this one, I'll try to bootstrap an OpenStack env and boot some VMs from a containerized Ceph.

I just built a setup for this, it consists of a DevStack running master and a containerized ceph using the ceph/demo container.

I was able to successfully boot a VM with the following features: layering, exclusive-lock, object-map, fast-diff, deep-flatten

In the end, I suspect you are experiencing a setup issue.

Can you describe your setup?
Thanks!

I am running the ceph cluster on atomic hosts. 1 mon and 3 osd's and all are running as containers. created a rbd image and mapped this to a VM in virt. While creating the rbd image given the feature as deep-flatten. When the VM boots it throws out an error. While plain enabling of features on the image does work.

How do you boot your vms?

From virt GUI. Tried CLI too but doesn't work.

No but is it libvirt+qemu-kvm?

yes..

@Bhaskarakiran: I am fairly sure that you have an old librbd1 RPM (0.94.x series) on your libvirt host. The default version that ships w/ base RHEL does not support the newer RHCS 2.0 features.

I will check that.

Let me know the latest RPM location / repo so that i verify the same and close on this at the earlist.

Hmm yes.. i tried that as well with another librbd image. That too gave the same error / paused state.

closing per #26.

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days