Bug 1536861

Summary: Failed Launching VM which has disk on Cinder SD
Product: Red Hat Enterprise Linux 7 Reporter: Petr Balogh <pbalogh>
Component: qemu-kvm-rhevAssignee: Virtualization Maintenance <virt-maint>
Status: CLOSED CURRENTRELEASE QA Contact: CongLi <coli>
Severity: high Docs Contact:
Priority: unspecified    
Version: 7.5CC: areis, bugs, chayang, coli, frolland, gveitmic, jdillama, juzhang, knoel, pbalogh, virt-maint
Target Milestone: pre-dev-freezeKeywords: Regression, TestOnly
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-05-13 01:28:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Storage RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1421783    
Bug Blocks:    

Description Petr Balogh 2018-01-21 19:00:35 UTC 
Description of problem:
Cannot run VM which has disk on the Cinder sd with Ceph as backend.

Version-Release number of selected component (if applicable):
rhv-4.2.1-3
Hosts and engine is running with RHEL 7.5

How reproducible:
Steps to Reproduce:
1. Add the cinder EP with the Ceph as a backend.
2. Create VM and create disk on Cinder SD.
3. Run VM

Actual results:
Engine throw error: Failed Launching VM cinder-vm-name

Expected results:
Start VM


Additional info:
will send in private comment on which environemnt you can find this problem and will send logs.
Comment 2 Petr Balogh 2018-01-21 19:08:06 UTC 
In steps to reproduce I forgot mention that you also need to add Authentication Keys for cinder EP between step 1 and 2.
Comment 3 Fred Rolland 2018-01-24 15:52:48 UTC 
What is the version of QEMU in the host?

I have seen this issue before, when qemu is not supporting the new rbd image format (format-2).

Adding this entry in the ceph.conf , in the openstack host should solve the issue:

[client]
    rbd default features = 13
Comment 4 Yaniv Lavi 2018-02-01 15:33:38 UTC 
Is there a chance this will break OSP with Ceph in RHEL 7.5.
Comment 5 Ademar Reis 2018-02-03 19:47:17 UTC 
(In reply to Yaniv Lavi from comment #4)
> Is there a chance this will break OSP with Ceph in RHEL 7.5.

To evaluate this from the QEMU perspective, we need:

 - The version of QEMU and libvirt being used.
 - A way to reproduce it using virsh or libvirt.
 - The QEMU command line (ps -auxww, or please let me know in which logs I can find it)
 - Can you find the errors returned by libvirt in the log and paste them here? I had trouble finding them in a quick look.
Comment 6 Allon Mureinik 2018-02-18 12:20:50 UTC 
RHV has nothing to do here, unfortunately. Moving to QEMU so those stakeholders decide how to proceed.
Comment 10 Fred Rolland 2018-02-20 09:56:57 UTC 
Petr, can you please provide the needed info required in comment #5 ?
Thanks
Comment 13 Petr Balogh 2018-02-21 10:40:44 UTC 
After adding:

[client]
rbd default features = 13

to the ceph conf file we are able now to run VM with disk on cinder.

So this is workaround for this issue.
Comment 16 Petr Balogh 2018-02-21 12:23:31 UTC 
Openstack details from our cinder:
osp-11.0
Cinder version:
openstack-cinder-10.0.4-3.el7ost

Ceph packages on cinder machine:
libcephfs1-10.2.7-32.el7cp.x86_64
python-cephfs-10.2.7-32.el7cp.x86_64
ceph-common-10.2.7-32.el7cp.x86_64


Ceph packages on ceph machine:
rpm -qa |grep ceph
python-cephfs-10.2.7-32.el7cp.x86_64
ceph-selinux-10.2.7-32.el7cp.x86_64
ceph-osd-10.2.7-32.el7cp.x86_64
ceph-ansible-2.1.9-1.el7scon.noarch
libcephfs1-10.2.7-32.el7cp.x86_64
ceph-common-10.2.7-32.el7cp.x86_64
ceph-base-10.2.7-32.el7cp.x86_64
ceph-mds-10.2.7-32.el7cp.x86_64
ceph-radosgw-10.2.7-32.el7cp.x86_64
ceph-mon-10.2.7-32.el7cp.x86_64
Comment 19 Jason Dillaman 2018-06-06 17:17:00 UTC 
Just to clarify, this isn't a QEMU issue where is doesn't support the RBD v2 image format. Instead, the version of librbd1 that is currently available in RHEL 7.x is from the upstream hammer release which doesn't support the same set of add-on features to the RBD v2 image format that are currently available (and default-enabled) in later librbd1 releases (Jewel and  Luminous).

That is why you need to force a Cinder which is using a newer Ceph release (e.g. Jewel) to use a legacy feature set that is compatible w/ an older Ceph release (e.g. Hammer).

BZ 1421783 is tracking the request to upgrade the ceph-common packages that are available in RHEL 7 to a more recent (and non-EoLed) version of Ceph.
Comment 20 Petr Balogh 2018-06-07 10:55:12 UTC 
We got this cinder/ceph instance installed from Tzach Shefi for usage in our RHV QE testing. So I am not openstack/storage guy and don't really understand the details here, But now after this clarification it's much clear.  Any other details about Ceph/Cinder we are using we can ask Tzach directly to provide some more details if needed.
Comment 21 Jeff Cody 2018-06-20 18:28:55 UTC 
From Comment #19 and Comment #20, it sounds like:

1. This isn't a QEMU issue, as there is not any QEMU driver modification needed, and
2. The BZ mentioned in Comment #19 (BZ 1421783) will fix the issue reported in this BZ.

If both of that is true, then does it sounds reasonable to set this BZ to test-only?
Comment 22 Jason Dillaman 2018-06-20 18:34:42 UTC 
(In reply to Jeff Cody from comment #21)
> If both of that is true, then does it sounds reasonable to set this BZ to
> test-only?

Since this was a ticket from RHV QE testing, sounds reasonable to me.
Comment 23 Petr Balogh 2018-06-22 22:10:23 UTC 
Jeff, so if I understand correctly, if we will start using rhel 7.6 mentioned in BZ 1421783, it should be ok and we can remove WA from comment ceph conf file: rbd default features = 13 ?

Thanks
Comment 24 Jeff Cody 2018-07-16 00:52:07 UTC 
(In reply to Petr Balogh from comment #23)
> Jeff, so if I understand correctly, if we will start using rhel 7.6
> mentioned in BZ 1421783, it should be ok and we can remove WA from comment
> ceph conf file: rbd default features = 13 ?
> 
> Thanks

I'm not sure, to be honest.  The documentation I could find on the default features is this: https://tracker.ceph.com/issues/15000.  I don't know what features openstack is looking to enable/disable for Ceph.  But as Jason said in Comment #19, this is forcing a feature set compatible with older versions of Ceph, so I presume that once BZ #1421783 is in place that will no longer be needed.

I'm going to go ahead and set this bz to depend on BZ #1421783, move to modified, and to be TestOnly since there are no changes that are needed to be made by QEMU.
Comment 25 Longxiang Lyu 2018-07-18 06:13:21 UTC 
Hi, Jeff, is it necessary for kvm-qe to cover this scenario in testing?
Comment 26 Jeff Cody 2018-08-01 02:22:40 UTC 
(In reply to Longxiang Lyu from comment #25)
> Hi, Jeff, is it necessary for kvm-qe to cover this scenario in testing?

Hi, I think it would be good to do so - we should verify BZ #1421783 does indeed alleviate the concerns in this BZ.
Comment 27 CongLi 2018-08-01 10:36:03 UTC 
(In reply to Jeff Cody from comment #26)
> (In reply to Longxiang Lyu from comment #25)
> > Hi, Jeff, is it necessary for kvm-qe to cover this scenario in testing?
> 
> Hi, I think it would be good to do so - we should verify BZ #1421783 does
> indeed alleviate the concerns in this BZ.

Hi Jeff,

Could you please give QE some guidance on how to reproduce and verify this bug ?

Thanks.
Comment 28 Jeff Cody 2018-08-01 17:28:11 UTC 
(In reply to CongLi from comment #27)
indeed alleviate the concerns in this BZ.
> 
> Hi Jeff,
> 
> Could you please give QE some guidance on how to reproduce and verify this
> bug ?
> 
> Thanks.

I'm not entirely sure the best steps to reproduce; since this isn't really in the QEMU layer, it seems like the best reproduction steps for the original report is from the OpenStack Cinder layer.  Maybe Petr can provide a good method for QE to test?
Comment 29 CongLi 2018-08-07 08:13:32 UTC 
Hi Petr,

Is it ok for you to provide a method for QE to verify this issue ?

Thanks.
Comment 30 Petr Balogh 2018-08-09 17:07:05 UTC 
Hi @CongLi,

next week I can find time to test the scenario I hit when I opened this bug with our cinder. So I can remove WA mentioned in #comment 3 and try again.

Just let me know whit which OS version and RHV I need to verify?

RHEL 7.5 or 7.6?
And latest RHV 4.2.6-1?


Thanks, Petr
Comment 31 CongLi 2018-08-10 02:41:35 UTC 
(In reply to Petr Balogh from comment #30)
> Hi @CongLi,
> 
> next week I can find time to test the scenario I hit when I opened this bug
> with our cinder. So I can remove WA mentioned in #comment 3 and try again.
> 
> Just let me know whit which OS version and RHV I need to verify?
> 
> RHEL 7.5 or 7.6?
> And latest RHV 4.2.6-1?
> 
> 
> Thanks, Petr

Thanks Petr.

I think latest RHEL.7.6 and RHV 4.2.6-1 is fine.
Comment 32 CongLi 2018-08-10 02:45:44 UTC 
(In reply to CongLi from comment #31)
> (In reply to Petr Balogh from comment #30)
> > Hi @CongLi,
> > 
> > next week I can find time to test the scenario I hit when I opened this bug
> > with our cinder. So I can remove WA mentioned in #comment 3 and try again.
> > 
> > Just let me know whit which OS version and RHV I need to verify?
> > 
> > RHEL 7.5 or 7.6?
> > And latest RHV 4.2.6-1?
> > 
> > 
> > Thanks, Petr
> 
> Thanks Petr.
> 
> I think latest RHEL.7.6 and RHV 4.2.6-1 is fine.

Petr,

Please use ceph-common-10.2.5-4.el7 package for the verification based on Jeff's suggestion in comment 26.

recording from comment 26:
"""
Hi, I think it would be good to do so - we should verify BZ #1421783 does indeed alleviate the concerns in this BZ.
"""

Thanks.
Comment 33 CongLi 2018-08-23 05:29:32 UTC 
Hi Petr,

Could you please share the test result when you finish the verification ?

Thanks.
Comment 34 Petr Balogh 2018-08-23 12:21:47 UTC 
Hey, 

I tried verify with our already installed instance of cinder.

So I downloaded packages from here:
http://download.eng.bos.redhat.com/brewroot/packages/ceph-common/10.2.5/2.el7/x86_64/

And tried downgrade all packages from there but got dep issue cause I already have installed newer versions on our cinder which we are currently using.

Problematic one is for example python-cephfs-10.2.7 and librgw2-10.2.7-32 which I don't know from where I can get version 10.2.5-2


yum downgrade ./*10.2.5-2*.rpm
Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager
This system is not registered with an entitlement server. You can use subscription-manager to register.
Examining ./ceph-common-10.2.5-2.el7.x86_64.rpm: 1:ceph-common-10.2.5-2.el7.x86_64
Examining ./ceph-common-debuginfo-10.2.5-2.el7.x86_64.rpm: 1:ceph-common-debuginfo-10.2.5-2.el7.x86_64
No Match for available package: 1:ceph-common-debuginfo-10.2.5-2.el7.x86_64
Examining ./librados2-10.2.5-2.el7.x86_64.rpm: 1:librados2-10.2.5-2.el7.x86_64
Examining ./librados2-devel-10.2.5-2.el7.x86_64.rpm: 1:librados2-devel-10.2.5-2.el7.x86_64
No Match for available package: 1:librados2-devel-10.2.5-2.el7.x86_64
Examining ./librbd1-10.2.5-2.el7.x86_64.rpm: 1:librbd1-10.2.5-2.el7.x86_64
Examining ./librbd1-devel-10.2.5-2.el7.x86_64.rpm: 1:librbd1-devel-10.2.5-2.el7.x86_64
No Match for available package: 1:librbd1-devel-10.2.5-2.el7.x86_64
Examining ./python-rados-10.2.5-2.el7.x86_64.rpm: 1:python-rados-10.2.5-2.el7.x86_64
Examining ./python-rbd-10.2.5-2.el7.x86_64.rpm: 1:python-rbd-10.2.5-2.el7.x86_64
Resolving Dependencies
--> Running transaction check
---> Package ceph-common.x86_64 1:10.2.5-2.el7 will be a downgrade
--> Processing Dependency: hdparm for package: 1:ceph-common-10.2.5-2.el7.x86_64
--> Processing Dependency: redhat-lsb-core for package: 1:ceph-common-10.2.5-2.el7.x86_64
---> Package ceph-common.x86_64 2:10.2.7-32.el7cp will be erased
---> Package librados2.x86_64 1:10.2.5-2.el7 will be a downgrade
---> Package librados2.x86_64 2:10.2.7-32.el7cp will be erased                                                                                                                            
---> Package librbd1.x86_64 1:10.2.5-2.el7 will be a downgrade                                                                                                                            
---> Package librbd1.x86_64 2:10.2.7-32.el7cp will be erased                                                                                                                              
---> Package python-rados.x86_64 1:10.2.5-2.el7 will be a downgrade                                                                                                                       
---> Package python-rados.x86_64 2:10.2.7-32.el7cp will be erased                                                                                                                         
---> Package python-rbd.x86_64 1:10.2.5-2.el7 will be a downgrade                                                                                                                         
---> Package python-rbd.x86_64 2:10.2.7-32.el7cp will be erased                                                                                                                           
--> Running transaction check                                                                                                                                                             
---> Package hdparm.x86_64 0:9.43-5.el7 will be installed                                                                                                                                 
---> Package librados2.x86_64 2:10.2.7-32.el7cp will be erased                                                                                                                            
---> Package python-rados.x86_64 2:10.2.7-32.el7cp will be erased                                                                                                                         
---> Package redhat-lsb-core.x86_64 0:4.1-27.el7 will be installed                                                                                                                        
--> Processing Dependency: redhat-lsb-submod-security(x86-64) = 4.1-27.el7 for package: redhat-lsb-core-4.1-27.el7.x86_64                                                                 
--> Processing Dependency: /usr/bin/lp for package: redhat-lsb-core-4.1-27.el7.x86_64                                                                                                     
--> Processing Dependency: /usr/bin/lpr for package: redhat-lsb-core-4.1-27.el7.x86_64                                                                                                    
--> Processing Dependency: /usr/bin/m4 for package: redhat-lsb-core-4.1-27.el7.x86_64                                                                                                     
--> Processing Dependency: /usr/bin/patch for package: redhat-lsb-core-4.1-27.el7.x86_64                                                                                                  
--> Processing Dependency: spax for package: redhat-lsb-core-4.1-27.el7.x86_64                                                                                                            
--> Running transaction check                                                                                                                                                             
---> Package cups-client.x86_64 1:1.6.3-35.el7 will be installed                                                                                                                          
--> Processing Dependency: cups-libs(x86-64) = 1:1.6.3-35.el7 for package: 1:cups-client-1.6.3-35.el7.x86_64                                                                              
--> Processing Dependency: libcups.so.2()(64bit) for package: 1:cups-client-1.6.3-35.el7.x86_64                                                                                           
---> Package librados2.x86_64 2:10.2.7-32.el7cp will be erased                                                                                                                            
---> Package m4.x86_64 0:1.4.16-10.el7 will be installed                                                                                                                                  
---> Package patch.x86_64 0:2.7.1-10.el7_5 will be installed
---> Package python-rados.x86_64 2:10.2.7-32.el7cp will be erased
---> Package redhat-lsb-submod-security.x86_64 0:4.1-27.el7 will be installed
---> Package spax.x86_64 0:1.5.2-13.el7 will be installed
--> Running transaction check
---> Package cups-libs.x86_64 1:1.6.3-35.el7 will be installed
---> Package librados2.x86_64 2:10.2.7-32.el7cp will be erased
---> Package python-rados.x86_64 2:10.2.7-32.el7cp will be erased
--> Finished Dependency Resolution
Error: Package: 2:python-cephfs-10.2.7-32.el7cp.x86_64 (@rhelosp-11.0-ceph-2.0-mon)
           Requires: python-rados = 2:10.2.7-32.el7cp
           Removing: 2:python-rados-10.2.7-32.el7cp.x86_64 (@rhelosp-11.0-ceph-2.0-mon)
               python-rados = 2:10.2.7-32.el7cp
           Downgraded By: 1:python-rados-10.2.5-2.el7.x86_64 (/python-rados-10.2.5-2.el7.x86_64)
               python-rados = 1:10.2.5-2.el7
           Available: 1:python-rados-0.80.7-2.el7.x86_64 (rhelosp-rhel-7.4-server)
               python-rados = 1:0.80.7-2.el7
           Available: 1:python-rados-0.80.7-3.el7.x86_64 (rhelosp-rhel-7.4-server)
               python-rados = 1:0.80.7-3.el7
           Available: 1:python-rados-0.94.5-1.el7.x86_64 (rhelosp-rhel-7.4-server)
               python-rados = 1:0.94.5-1.el7
           Available: 1:python-rados-0.94.5-2.el7.x86_64 (rhel-7.4-zstream)
               python-rados = 1:0.94.5-2.el7
           Available: 1:python-rados-10.2.2-38.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 1:10.2.2-38.el7cp
           Available: 1:python-rados-10.2.2-41.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 1:10.2.2-41.el7cp
           Available: 1:python-rados-10.2.3-13.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 1:10.2.3-13.el7cp
           Available: 1:python-rados-10.2.3-17.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 1:10.2.3-17.el7cp
           Available: 1:python-rados-10.2.5-37.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 1:10.2.5-37.el7cp
           Available: 1:python-rados-10.2.7-27.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 1:10.2.7-27.el7cp
           Available: 1:python-rados-10.2.7-28.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 1:10.2.7-28.el7cp
           Available: 2:python-rados-10.2.7-48.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 2:10.2.7-48.el7cp
           Available: 2:python-rados-10.2.10-16.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 2:10.2.10-16.el7cp
           Available: 2:python-rados-10.2.10-17.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 2:10.2.10-17.el7cp
           Available: 2:python-rados-10.2.10-28.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 2:10.2.10-28.el7cp
Error: Package: 2:librgw2-10.2.7-32.el7cp.x86_64 (@rhelosp-11.0-ceph-2.0-mon)
           Requires: librados2 = 2:10.2.7-32.el7cp
           Removing: 2:librados2-10.2.7-32.el7cp.x86_64 (@rhelosp-11.0-ceph-2.0-mon)
               librados2 = 2:10.2.7-32.el7cp
           Downgraded By: 1:librados2-10.2.5-2.el7.x86_64 (/librados2-10.2.5-2.el7.x86_64)
               librados2 = 1:10.2.5-2.el7
           Available: 1:librados2-0.80.7-2.el7.x86_64 (rhelosp-rhel-7.4-server)
               librados2 = 1:0.80.7-2.el7
           Available: 1:librados2-0.80.7-3.el7.x86_64 (rhelosp-rhel-7.4-server)
               librados2 = 1:0.80.7-3.el7
           Available: 1:librados2-0.94.5-1.el7.x86_64 (rhelosp-rhel-7.4-server)
               librados2 = 1:0.94.5-1.el7
           Available: 1:librados2-0.94.5-2.el7.i686 (rhel-7.4-zstream)
               librados2 = 1:0.94.5-2.el7
           Available: 1:librados2-10.2.2-38.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 1:10.2.2-38.el7cp
           Available: 1:librados2-10.2.2-41.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 1:10.2.2-41.el7cp
           Available: 1:librados2-10.2.3-13.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 1:10.2.3-13.el7cp
           Available: 1:librados2-10.2.3-17.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 1:10.2.3-17.el7cp
           Available: 1:librados2-10.2.5-37.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 1:10.2.5-37.el7cp
           Available: 1:librados2-10.2.7-27.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 1:10.2.7-27.el7cp
           Available: 1:librados2-10.2.7-28.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 1:10.2.7-28.el7cp
           Available: 2:librados2-10.2.7-48.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 2:10.2.7-48.el7cp
           Available: 2:librados2-10.2.10-16.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 2:10.2.10-16.el7cp
           Available: 2:librados2-10.2.10-17.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 2:10.2.10-17.el7cp
           Available: 2:librados2-10.2.10-28.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 2:10.2.10-28.el7cp


I can touch only our cinder machine, cause ceph machines are provided by other team from Tzach Shefi and they are using them as well.  So I guess I cannot really help here with verification as I am not storage guy and I don't have resources and experiences to install everything from scratch.

If someone can provide me the installed env with versions you need I can verify integration with RHVM.

Thanks, petr
Comment 35 CongLi 2018-08-27 10:03:20 UTC 
(In reply to Petr Balogh from comment #34)
> Hey, 
> 
> I tried verify with our already installed instance of cinder.

Thanks for your help of the verification.

> 
> So I downloaded packages from here:
> http://download.eng.bos.redhat.com/brewroot/packages/ceph-common/10.2.5/2.
> el7/x86_64/
> 

Petr,

Sorry, I'm a little confused why downgrade packages to ceph-common-10.2.5-2.el7, is there anything I'm missing?

I think package ceph-common-10.2.5-4.el7 is fine.
http://download.eng.bos.redhat.com/brewroot/packages/ceph-common/10.2.5/4.el7/x86_64/
Comment 36 Petr Balogh 2018-08-27 15:20:42 UTC 
Hey @CongLi,

I think that changing of version doesn't solve my issue. I tried both cause version I wasn't sure from related bug and also was little bit confused in which version it's exactly fixed, if in both 10.2.5-2, 10.2.5-4,  or just second one.

But same issue cause I already have some newer packages installed I cannot downgrade neither to 10.2.5-4:  

3/3): rhelosp-rhel-7.4-server/x86_64/primary_db                                                                                                                      |  55 MB  00:08:27     
--> Processing Dependency: redhat-lsb-core for package: 1:ceph-common-10.2.5-4.el7.x86_64
---> Package ceph-common.x86_64 2:10.2.7-32.el7cp will be erased
---> Package librados2.x86_64 1:10.2.5-4.el7 will be a downgrade
---> Package librados2.x86_64 2:10.2.7-32.el7cp will be erased
---> Package librbd1.x86_64 1:10.2.5-4.el7 will be a downgrade
---> Package librbd1.x86_64 2:10.2.7-32.el7cp will be erased
---> Package python-rados.x86_64 1:10.2.5-4.el7 will be a downgrade
---> Package python-rados.x86_64 2:10.2.7-32.el7cp will be erased
---> Package python-rbd.x86_64 1:10.2.5-4.el7 will be a downgrade
---> Package python-rbd.x86_64 2:10.2.7-32.el7cp will be erased
--> Running transaction check
---> Package hdparm.x86_64 0:9.43-5.el7 will be installed
---> Package librados2.x86_64 2:10.2.7-32.el7cp will be erased
---> Package python-rados.x86_64 2:10.2.7-32.el7cp will be erased
---> Package redhat-lsb-core.x86_64 0:4.1-27.el7 will be installed
--> Processing Dependency: redhat-lsb-submod-security(x86-64) = 4.1-27.el7 for package: redhat-lsb-core-4.1-27.el7.x86_64
--> Processing Dependency: /usr/bin/lp for package: redhat-lsb-core-4.1-27.el7.x86_64
--> Processing Dependency: /usr/bin/lpr for package: redhat-lsb-core-4.1-27.el7.x86_64
--> Processing Dependency: /usr/bin/m4 for package: redhat-lsb-core-4.1-27.el7.x86_64
--> Processing Dependency: /usr/bin/patch for package: redhat-lsb-core-4.1-27.el7.x86_64
--> Processing Dependency: spax for package: redhat-lsb-core-4.1-27.el7.x86_64
--> Running transaction check
---> Package cups-client.x86_64 1:1.6.3-35.el7 will be installed
--> Processing Dependency: cups-libs(x86-64) = 1:1.6.3-35.el7 for package: 1:cups-client-1.6.3-35.el7.x86_64
--> Processing Dependency: libcups.so.2()(64bit) for package: 1:cups-client-1.6.3-35.el7.x86_64
---> Package librados2.x86_64 2:10.2.7-32.el7cp will be erased
---> Package m4.x86_64 0:1.4.16-10.el7 will be installed
---> Package patch.x86_64 0:2.7.1-10.el7_5 will be installed
---> Package python-rados.x86_64 2:10.2.7-32.el7cp will be erased
---> Package redhat-lsb-submod-security.x86_64 0:4.1-27.el7 will be installed
---> Package spax.x86_64 0:1.5.2-13.el7 will be installed
--> Running transaction check
---> Package cups-libs.x86_64 1:1.6.3-35.el7 will be installed
---> Package librados2.x86_64 2:10.2.7-32.el7cp will be erased
---> Package python-rados.x86_64 2:10.2.7-32.el7cp will be erased
--> Finished Dependency Resolution
Error: Package: 2:librgw2-10.2.7-32.el7cp.x86_64 (@rhelosp-11.0-ceph-2.0-mon)
           Requires: librados2 = 2:10.2.7-32.el7cp
           Removing: 2:librados2-10.2.7-32.el7cp.x86_64 (@rhelosp-11.0-ceph-2.0-mon)
               librados2 = 2:10.2.7-32.el7cp
           Downgraded By: 1:librados2-10.2.5-4.el7.x86_64 (/librados2-10.2.5-4.el7.x86_64)
               librados2 = 1:10.2.5-4.el7
           Available: 1:librados2-0.80.7-2.el7.x86_64 (rhelosp-rhel-7.4-server)
               librados2 = 1:0.80.7-2.el7
           Available: 1:librados2-0.80.7-3.el7.x86_64 (rhelosp-rhel-7.4-server)
               librados2 = 1:0.80.7-3.el7
           Available: 1:librados2-0.94.5-1.el7.x86_64 (rhelosp-rhel-7.4-server)
               librados2 = 1:0.94.5-1.el7
           Available: 1:librados2-0.94.5-2.el7.i686 (rhel-7.4-zstream)
               librados2 = 1:0.94.5-2.el7
           Available: 1:librados2-10.2.2-38.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 1:10.2.2-38.el7cp
           Available: 1:librados2-10.2.2-41.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 1:10.2.2-41.el7cp
           Available: 1:librados2-10.2.3-13.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 1:10.2.3-13.el7cp
           Available: 1:librados2-10.2.3-17.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 1:10.2.3-17.el7cp
           Available: 1:librados2-10.2.5-37.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 1:10.2.5-37.el7cp
           Available: 1:librados2-10.2.7-27.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 1:10.2.7-27.el7cp
           Available: 1:librados2-10.2.7-28.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 1:10.2.7-28.el7cp
           Available: 2:librados2-10.2.7-48.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 2:10.2.7-48.el7cp
           Available: 2:librados2-10.2.10-16.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 2:10.2.10-16.el7cp
           Available: 2:librados2-10.2.10-17.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 2:10.2.10-17.el7cp
           Available: 2:librados2-10.2.10-28.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               librados2 = 2:10.2.10-28.el7cp
Error: Package: 2:python-cephfs-10.2.7-32.el7cp.x86_64 (@rhelosp-11.0-ceph-2.0-mon)
           Requires: python-rados = 2:10.2.7-32.el7cp
           Removing: 2:python-rados-10.2.7-32.el7cp.x86_64 (@rhelosp-11.0-ceph-2.0-mon)
               python-rados = 2:10.2.7-32.el7cp
           Downgraded By: 1:python-rados-10.2.5-4.el7.x86_64 (/python-rados-10.2.5-4.el7.x86_64)
               python-rados = 1:10.2.5-4.el7
           Available: 1:python-rados-0.80.7-2.el7.x86_64 (rhelosp-rhel-7.4-server)
               python-rados = 1:0.80.7-2.el7
           Available: 1:python-rados-0.80.7-3.el7.x86_64 (rhelosp-rhel-7.4-server)
               python-rados = 1:0.80.7-3.el7
           Available: 1:python-rados-0.94.5-1.el7.x86_64 (rhelosp-rhel-7.4-server)
               python-rados = 1:0.94.5-1.el7
           Available: 1:python-rados-0.94.5-2.el7.x86_64 (rhel-7.4-zstream)
               python-rados = 1:0.94.5-2.el7
           Available: 1:python-rados-10.2.2-38.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 1:10.2.2-38.el7cp
           Available: 1:python-rados-10.2.2-41.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 1:10.2.2-41.el7cp
           Available: 1:python-rados-10.2.3-13.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 1:10.2.3-13.el7cp
           Available: 1:python-rados-10.2.3-17.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 1:10.2.3-17.el7cp
           Available: 1:python-rados-10.2.5-37.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 1:10.2.5-37.el7cp
           Available: 1:python-rados-10.2.7-27.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 1:10.2.7-27.el7cp
           Available: 1:python-rados-10.2.7-28.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 1:10.2.7-28.el7cp
           Available: 2:python-rados-10.2.7-48.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 2:10.2.7-48.el7cp
           Available: 2:python-rados-10.2.10-16.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 2:10.2.10-16.el7cp
           Available: 2:python-rados-10.2.10-17.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 2:10.2.10-17.el7cp
           Available: 2:python-rados-10.2.10-28.el7cp.x86_64 (rhelosp-11.0-ceph-2.0-mon)
               python-rados = 2:10.2.10-28.el7cp


I see that OS on cinder machine is still running on 7.4, so I can try to upgrade it to 7.5. 

I used rhos-release to upgrade rhel repos for 7.5 as well.

Now when I am trying to downgrade to 10.2.5-4:
Error: Package: 2:librgw2-10.2.10-28.el7cp.x86_64 (@rhelosp-11.0-ceph-2.0-mon)
           Requires: librados2 = 2:10.2.10-28.el7cp
           Removing: 2:librados2-10.2.10-28.el7cp.x86_64 (@rhelosp-11.0-ceph-2.0-mon)
               librados2 = 2:10.2.10-28.el7cp
           Downgraded By: 1:librados2-10.2.5-4.el7.x86_64 (/librados2-10.2.5-4.el7.x86_64)
               librados2 = 1:10.2.5-4.el7
           Available: 1:librados2-0.94.5-2.el7.i686 (rhel-7.5-base)
               librados2 = 1:0.94.5-2.el7
           Available: 1:librados2-10.2.2-38.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               librados2 = 1:10.2.2-38.el7cp
           Available: 1:librados2-10.2.2-41.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               librados2 = 1:10.2.2-41.el7cp
           Available: 1:librados2-10.2.3-13.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               librados2 = 1:10.2.3-13.el7cp
           Available: 1:librados2-10.2.3-17.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               librados2 = 1:10.2.3-17.el7cp
           Available: 1:librados2-10.2.5-37.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               librados2 = 1:10.2.5-37.el7cp
           Available: 1:librados2-10.2.7-27.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               librados2 = 1:10.2.7-27.el7cp
           Available: 1:librados2-10.2.7-28.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               librados2 = 1:10.2.7-28.el7cp
           Available: 2:librados2-10.2.7-32.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               librados2 = 2:10.2.7-32.el7cp
           Available: 2:librados2-10.2.7-48.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               librados2 = 2:10.2.7-48.el7cp
           Available: 2:librados2-10.2.10-16.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               librados2 = 2:10.2.10-16.el7cp
           Available: 2:librados2-10.2.10-17.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               librados2 = 2:10.2.10-17.el7cp
Error: Package: 2:python-cephfs-10.2.10-28.el7cp.x86_64 (@rhelosp-11.0-ceph-2.0-mon)
           Requires: python-rados = 2:10.2.10-28.el7cp
           Removing: 2:python-rados-10.2.10-28.el7cp.x86_64 (@rhelosp-11.0-ceph-2.0-mon)
               python-rados = 2:10.2.10-28.el7cp
           Downgraded By: 1:python-rados-10.2.5-4.el7.x86_64 (/python-rados-10.2.5-4.el7.x86_64)
               python-rados = 1:10.2.5-4.el7
           Available: 1:python-rados-0.94.5-2.el7.x86_64 (rhel-7.5-base)
               python-rados = 1:0.94.5-2.el7
           Available: 1:python-rados-10.2.2-38.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               python-rados = 1:10.2.2-38.el7cp
           Available: 1:python-rados-10.2.2-41.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               python-rados = 1:10.2.2-41.el7cp
           Available: 1:python-rados-10.2.3-13.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               python-rados = 1:10.2.3-13.el7cp
           Available: 1:python-rados-10.2.3-17.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               python-rados = 1:10.2.3-17.el7cp
           Available: 1:python-rados-10.2.5-37.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               python-rados = 1:10.2.5-37.el7cp
           Available: 1:python-rados-10.2.7-27.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               python-rados = 1:10.2.7-27.el7cp
           Available: 1:python-rados-10.2.7-28.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               python-rados = 1:10.2.7-28.el7cp
           Available: 2:python-rados-10.2.7-32.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               python-rados = 2:10.2.7-32.el7cp
           Available: 2:python-rados-10.2.7-48.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               python-rados = 2:10.2.7-48.el7cp
           Available: 2:python-rados-10.2.10-16.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               python-rados = 2:10.2.10-16.el7cp
           Available: 2:python-rados-10.2.10-17.el7cp.x86_64 (rhelosp-ceph-2.0-mon)
               python-rados = 2:10.2.10-17.el7cp



Do I really need to verify it with this old version when I have available newer?  This fix is not inside version of ceph-common which I have installed now after upgrade: ceph-common-10.2.10-28.el7cp.x86_64 ?
Comment 37 CongLi 2018-08-28 02:35:53 UTC 
(In reply to Petr Balogh from comment #36)
> Do I really need to verify it with this old version when I have available
> newer?  This fix is not inside version of ceph-common which I have installed
> now after upgrade: ceph-common-10.2.10-28.el7cp.x86_64 ?

Hi Jeff,

Could you please help confirm comment 36 if ceph-common-10.2.10-28.el7cp.x86_64 is fine for this bug verification?

Thanks.
Comment 41 Petr Balogh 2018-09-11 11:11:55 UTC 
Sorry for delay, but I was PTO last week.

I removed WA from config file:
rbd default features = 13

restarted services also rebooted whole machine and indeed I can run VM with disk on cinder without any issue now.

So confirmed it's working well now with ceph-common-10.2.10-28.el7cp.x86_64 .

Petr
Comment 42 CongLi 2018-09-11 11:15:27 UTC 
(In reply to Petr Balogh from comment #41)
> Sorry for delay, but I was PTO last week.
> 
> I removed WA from config file:
> rbd default features = 13
> 
> restarted services also rebooted whole machine and indeed I can run VM with
> disk on cinder without any issue now.
> 
> So confirmed it's working well now with ceph-common-10.2.10-28.el7cp.x86_64 .
> 
> Petr

Petr,

Many thanks for helping verify this issue.