Description of problem: RHOSP 13 is a long term support release that is used by Dell EMC customers. The release contains Cinder driver for VxFlex OS storage systems up to version 2.6.1. The latter is an older release of the storage system and our customers demand support of newly built VxFlex OS 3.0.1 with much better space efficiency. We submitted necessary changes for Cinder driver to upstream (already in Train) but cannot port them back to Queens because OpenStack community considers it as a new feature. Please advise us on how to proceed - do we need to create a new container with backported Cinder driver source code? If so - shall we re-certify already existing image [1]? [1] https://access.redhat.com/containers/#/registry.connect.redhat.com/dellemc/rhosp13-cinder-volume-dellemc-vxflexos
Also due to many changes including VxFlex OS re-branding it won't be a small and fast-forward porting (just copy of files from newer OpenStack will not work). But at the end VxFlexOS 3.0.1 driver will support 2.6.1 and older releases of the storage system.
Can you provide a list of the upstream patches that would be involved with this change?
Hello, there will be changes from this commit https://github.com/openstack/cinder/commit/e102f3a715aad3bc55724f83320948fe71872be1#diff-a382adbdf8040f6d0e88a312c72bda34, but without driver renaming. I can provide patched driver.py for review if needed.
Eric, How do we want to proceed? We just got of customer call and our mutual large telco customer is waiting for it. The functionality they need is vxflexOS 3.0 that in turn need 3.0.1 vxflexOS cinder driver. This has been merge upstream in master and in train. We can create a smaller patch that does not include renaming from scaleio to vxflexOS for OSP13. To our knowledge no changes are required in base cinder. And as Ivan stated they already have it working with OSP13.
Note from Ivan: VxFlex OS 3.0.x features are backported to OSP13 driver. Patched driver and updated dockerfile for cinder container are in separate branch in our GitHub repo. https://github.com/dell/rhosp-vxflexos-deploy/tree/vxflexos_3.0.x_support I have made testing with unit-tests (also modified but not stated in git repo) and tempest. Seems that all is working as expected. However, this branch is yet to be merged into master
We needed only changes in cinder driver to support VxFlex OS 3.0.x, containers for Nova and Glance stay unchanged.
RH team, how do you want us to proceed? We need to backport to vxflexOS cinder driver? And then update vxflexOS cinder container.
Ken, we have customer escalations coming up on this issue. Please help to move this forward.
Sorry for the delayed response, here's a summary of the needs in reply to Arkady. The RH cinder team needs to backport the patches to OSP-13 for release in the next Z stream (need to get an ETA) and will likely need to provide a test build or Hotfix set of containers to enable Dell-EMC to test with. Dell-EMC vxFlexOS containers will get rebuilt by the RH catalog system, when that Z stream releases. We are confirming the RH cinder team has what then need to proceed with the backports based on the patches and info here from Rajini. What is the timeline for the customer need? Hotfix is not an optimal path in terms of complexity, so ideally they can take the next Z stream and we need to determine we can get this in and update you on that target schedule.
We just took a look at the github (comment 3) and looks like we still need the dell-emc backport of the cinder driver patches here to queens/OSP-13 level before we can backport. For all base cinder driver changes, we need vendors to supply the upstream backport to the same level (even if no stable branch), so we have something that works and is tested for that code base (which only the vendor/owner can provide). A github or other method can work for this, but we need to be sure we get the queens/osp-13 level code as our starting point. As soon as we get that pointer here, the RH cinder team can start the backport. In the meantime, we will chase the target dates and release planning we need to go with this one. thanks Paul
Paul, appreciate the update. We will take it from here.
Paul, the proposed patch in comment 3 cannot be backported upstream since it adds new functionality VxFlex OS driver now supports VxFlex OS 3.0 features: storage pools with fine granularity layout, volume compression(SPEF). Thus, it will not be accepted upstream to Queens. But that code change is in master in Train. That is why Ivan extracted from that merged master patch for queens and tested with queens.
Arkady, I believe you have misinterpreted Paul's reply. We acknowledge that this can't be back ported upstream. What we are asking is that you produce a Queens version of your patch, and put this in a private location that we can access. We will then produce a new version of OSP-13 with your changes included.
John, Patched driver and updated dockerfile for cinder container are in separate branch in our GitHub repo. https://github.com/dell/rhosp-vxflexos-deploy/tree/vxflexos_3.0.x_support Will this work?
Hi Rajini, Yes, I believe it will. I apologize for missing that you had given this to use previously. If we need anything else I will let you know. I'll also tell you when a new OSP-13 will be available. jv
Hi again, Examining the files in the repo you provided it appears to not be a proper Queens branch. We need to see a repo containing the upstream cinder code, with a branch (created by Dell) containing the latest vxflex driver code. jv
Also, please base your vxflex branch off the upstream stable/queens branch.
Hi, I forked openstack/cinder on GitHub and created branch with patched driver on top of upstream stable/queens as it was asked. Check this: https://github.com/ivanpch/cinder/tree/vxflexos_3.0.x_support
Back from our PTO/shutdown, Clearing the needinfo and looks like things are moving forward.
(In reply to Ivan Pchelintsev from comment #18) > I forked openstack/cinder on GitHub and created branch with patched driver > on top of upstream stable/queens as it was asked. > Check this: https://github.com/ivanpch/cinder/tree/vxflexos_3.0.x_support Hi, Does this branch pass unit tests? I see many errors similar to: oslo_config.cfg.NoSuchOptError: no such option sio_storage_pool_id in group [backend_defaults] when trying to execute the unit tests. These must pass for us to backport the patch.
Hi, I have fixed broken tests. All scaleio unit-tests are now ok. ====== Totals ====== Ran: 11773 tests in 266.0000 sec. - Passed: 11766 - Skipped: 7 - Expected Fail: 0 - Unexpected Success: 0 - Failed: 0 Sum of execute time for each test: 1386.0000 sec.
Thank you. Can we move it forward?
Paul, Eric, any update on this item would be appreciated. Thanks
Adding some people to follow up, we will want to get a container "test build" out to Dell to verify this and it should be headed to the next 13 erratta release - z11.
Hi team, any update? Sorry for the persistence but my account teams are after me on this one.
(In reply to a.stripeikis from comment #26) > Hi team, any update? Sorry for the persistence but my account teams are > after me on this one. Sorry for the delay, but now have a package that is ready for Dell EMC to test. Normally, we provide customers with signed RPM's and a Dockerfile to be used for creating a new cinder-volume container image. But in this situation, Red Hat is relying on Dell EMC to verify the fix, as we don't have the hardware to verify it ourselves. We also know the Unity driver has to be packaged into a Dell EMC plugin. With this in mind, I propose Red Hat simply email a copy of the updated (but unsigned) cinder RPM's to someone at Dell EMC. The Dell EMC team should be able to incorporate the new cinder RPM's into the Unity plugin you'll use to test the new code. If this is agreeable, then please let me know the email address(es) to send the RPM's.
Sorry, when I mentioned Unity I meant to say VxFlex OS.
I appreciate for the update. Please email the image to Vlad Belogrudov and CC me.
OK, email was sent. Please keep us posted on the status your testing!
For the customers that will upgrade from OSP10 to OSP13 , the VxFlexOS on the environment will be v2.6. How will the update path look like? Is the VXFlexOS 3.0.1 container on RHOSP13 backwards compatible?
Hi Rajini, You raise an interesting question, but I don't have any insight on how the Dell EMC driver interoperates with different versions. When cinder is updated from OSP-10 to 13, it's migrated from running on the host to running in a container. This operation should be seemless, as long as customers have guidance to ensure they use the VxFlex OS plugin in OSP-13. The fact that the driver version will also change is a separate issue, but I can't comment on how that affects interoperability.
Let's consider separate scenarios. 1. We have a brand new deployment of OSP13 with VXflexOS 3.0.x. I assume that the containers that are coming out for z11 are the ones that will work for it. That includes containers for cinder, nova and glance. 2. We have a brand new deployment of OSP13 with VXflexOS 2.6.x. I hope that the containers that are coming out for z11 are the ones that will work for it also. That includes containers for cinder, nova and glance. That means that the same containers support two different versions of VXflexOS. If it does not work then we need separate set of containers for vxflex 2.6.x and 3.0.x. 3. We do upgrade from OSP10 to OSP13. That upgrade does not upgrade VXflexOS that stays on 2.6.x version. If #2 is true I expect that to just work. 4. We upgrade from OSP10 to OSP13. The upgrade does include upgrade of VXflexOS from 2.6.x to 3.0.x. We cannot upgrade VXflexOS first, since we do not have OSP10 support for vxflex 3.0.x So we upgrade vxflex to 3.0,x after we upgrade to OSP13. If #2 works I do not expect any issues. If #2 does not work we need a different documented way to do this type of upgrade. Expect Vlad and Ivan to provide guidance
Scenario$3 would be rare. When you have a brand new OSP13, you want go with latest VXFlexOS "If it does not work then we need separate set of containers for vxflex 2.6.x and 3.0.x." - I'm not sure how you can create two sets of containers with different versions of VxFlexOS driver and all the latest base cinder code with bug fixes from Z11... - I'm guessing the lastest VXFlexOS Cinder Driver (v3.0.x) is backwards compatible so we won't need two different containers Will wait for Vlad and Ivan to confirm
(In reply to Rajini Karthik from comment #35) > Scenario$3 would be rare. When you have a brand new OSP13, you want go with > latest VXFlexOS > > > "If it does not work then we need separate set of containers for vxflex > 2.6.x and 3.0.x." > - I'm not sure how you can create two sets of containers with different > versions of VxFlexOS driver and all the latest base cinder code with bug > fixes from Z11... > - I'm guessing the lastest VXFlexOS Cinder Driver (v3.0.x) is backwards > compatible so we won't need two different containers > > Will wait for Vlad and Ivan to confirm Hi Rajini, you're right, Cinder driver supports VxFlex OS (ScaleIO) versions greater than 2.0.
Rob, Do we need to go thru formal certification with new containers that support both vxflex OS 2.6.x and 3.0.x? If yes, what config we need to cert? Mike Burns, when we be able to get z11 containers we can use for validation/certification?
Cave Cain, suggest you use z11 container for upgrade testing as that is what we will need to use at customer site.
(In reply to Alan Bishop from comment #27) > (In reply to a.stripeikis from comment #26) > > Hi team, any update? Sorry for the persistence but my account teams are > > after me on this one. > > Sorry for the delay, but now have a package that is ready for Dell EMC to > test. > > Normally, we provide customers with signed RPM's and a Dockerfile to be used > for creating a new cinder-volume container image. But in this situation, Red > Hat is relying on Dell EMC to verify the fix, as we don't have the hardware > to verify it ourselves. We also know the Unity driver has to be packaged > into a Dell EMC plugin. > > With this in mind, I propose Red Hat simply email a copy of the updated (but > unsigned) cinder RPM's to someone at Dell EMC. The Dell EMC team should be > able to incorporate the new cinder RPM's into the Unity plugin you'll use to > test the new code. > > If this is agreeable, then please let me know the email address(es) to send > the RPM's. Thanks Alan! Dell EMC team has successfully tested new RPMs with VxFlex OS 3.0.1 support.
Thanks to the vxFlex developers and testers. The updated code does not touch core cinder code, so no risk of regressions. Moving to VERIFIED.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0764