Description of problem: - In order to write qcow2 images into volumes (openstack volume create --image <image-uuid> <to-be-assigned-volume-name> --size <size> ) they're temporarily turned into raw format (/var/lib/cinder/conversion). It currently impossible to use a NFS mount for said conversion operations location. - It would be great to enable this, as it would allow small environment to simply plug a NFS mount on a necessity base. Version-Release number of selected component (if applicable): RHOSP 16.1 How reproducible: 100% Steps to Reproduce: 1. Prepare the NFS mount ============== [root@xxxx-ctr01 ~]# mkdir /tmp/nfs [root@xxxx-ctr01 ~]# mount -t nfs -o vers=4.1 10.X.Y.Z:/aaaaaaaaaaaa /tmp/nfs [root@xxxx-ctr01 ~]# mkdir /tmp/nfs/conversion [root@xxxx-ctr01 ~]# ll -dZ /tmp/nfs/conversion/ drwxr-xr-x. 2 root root system_u:object_r:nfs_t:s0 4096 Oct 6 13:24 /tmp/nfs/conversion/ [root@xxxx-ctr01 ~]# chown root:42400 /tmp/nfs/conversion/ -R [root@xxxx-ctr01 ~]# chmod 2755 /tmp/nfs/conversion/ [root@xxxx-ctr01 ~]# ll -dZ /tmp/nfs/conversion/ drwxr-sr-x. 2 root 42400 system_u:object_r:nfs_t:s0 4096 Oct 6 13:24 /tmp/nfs/conversion/ [root@xxxx-ctr01 ~]# umount /tmp/nfs/ ============== 2. Mount it on the controller node ============== [heat-admin@xxxx-ctr02 ~]$ sudo pcs status * openstack-cinder-volume-podman-0 (ocf::heartbeat:podman): Started xxxx-ctr03 ============== Verify conversion directory permissions and SE labels ============== [root@xxxx-ctr02 ~]# ll -dZ /var/lib/cinder/conversion/ drwxr-sr-x. 2 root 42400 system_u:object_r:container_file_t:s0 6 Oct 6 12:53 /var/lib/cinder/conversion/ ============== Update /etc/fstab ============== [root@xxxx-ctr02 ~]# vi /etc/fstab ・・・ 10.X.Y.Z:/aaaaaaaaaaaa/conversion /var/lib/cinder/conversion nfs4 _netdev,bg,intr,nosharecache,context=system_u:object_r:container_file_t:s0 0 0 ============== Mount it ============== [root@xxxx-ctr02 ~]# mount /var/lib/cinder/conversion [root@xxxx-ctr02 ~]# ll -dZ /var/lib/cinder/conversion/ drwxr-sr-x. 2 root 42400 system_u:object_r:container_file_t:s0 4096 Oct 6 13:24 /var/lib/cinder/conversion/ ============== 3. Repeat all the steps for other other controllers, and reflect the changes in Cinder ============== [root@xxxx-ctr02 ~]# pcs resource ban openstack-cinder-volume xxxx-ctr01 [root@xxxx-ctr02 ~]# pcs resource ban openstack-cinder-volume xxxx-ctr03 [root@xxxx-ctr02 ~]# pcs resource clear openstack-cinder-volume ============== 4. Restart the resources and now all cinder-volumes are failing ============== [root@xxxx-ctr02 ~]# pcs status * openstack-cinder-volume-podman-0 (ocf::heartbeat:podman): Stopped Failed Resource Actions: * openstack-cinder-volume-podman-0_start_0 on xxxx-ctr01 'error' (1): call=217, status='complete', exitreason='podman failed to launch container', last-rc-change='2020-10-06 13:41:47 +09:00', queued=0ms, exec=387ms * openstack-cinder-volume-podman-0_start_0 on xxxx-ctr02 'error' (1): call=251, status='complete', exitreason='podman failed to launch container', last-rc-change='2020-10-06 13:35:10 +09:00', queued=0ms, exec=389ms * openstack-cinder-volume-podman-0_start_0 on xxxx-ctr03 'error' (1): call=182, status='complete', exitreason='podman failed to launch container', last-rc-change='2020-10-06 13:41:46 +09:00', queued=0ms, exec=435ms ============== Actual results: - cinder-volume container fails to start when it tries mount NFS at /var/lib/cinder/conversions. The reason is that the bind for /var/lib/cinder has the z flag[2]. The reason is the container tries to relabel all the selinux type on startup. However, NFS storage doesn't support updating selinux type then that operation fails as shown below. /var/log/pacemaker/pacemaker.log on xxx-ctr03. ~~~ Oct 06 13:41:46 podman(openstack-cinder-volume-podman-0)[227480]: INFO: running container openstack-cinder-volume-podman-0 for the first time Oct 06 13:41:46 podman(openstack-cinder-volume-podman-0)[227480]: ERROR: Error: relabel failed "/var/lib/cinder": operation not supported Oct 06 13:41:46 podman(openstack-cinder-volume-podman-0)[227480]: ERROR: podman failed to launch container ~~~ Expected results: - Generally speaking we expect local file system is used for conversions operations. If NFS is used instead, the network speed will likely be a bottleneck for the entire duration of said operations. However, there might be some cases, in where CU have special requirement, and are unable to deploy their env to allocate enough space for the reconversion. As we've addressed [2], it would be nice to enable operators to mount NFS to overcome lack of space in small deployments. Additional info: - This issue partially resembles BZ[1] - Related code [2] [1] https://bugzilla.redhat.com/show_bug.cgi?id=1816918 [2] https://github.com/openstack/tripleo-heat-templates/blob/9fadd88753ce877fdaa09e65edcada03979bceb7/deployment/cinder/cinder-common-container-puppet.yaml#L213
I've done some digging and we see from the workaround [1] it changes the bind propagation from "Propagation": "rprivate" , the workaround sets to "Propagation": "shared" [2] . We also notice it's propagation is not configured by default [3]. So the problem here is how to make the workaround persistent, we can apply it during post deploy scripts but that means during the deploy it would be set back to default which causes cinder-volume service to go down; then when post deploy scripts run it would change it to propagation shared which would fix it back. I guess the goal would be to make it configurable in puppet so we can pass that with the deploy so it always stays shared and never breaks cinder-volume during deploys. [1] 1. Remove the definition of /var/lib/cinder bind ~~~ pcs resource bundle update openstack-cinder-volume \ storage-map remove cinder-volume-var-lib-cinder ~~~ 2. Re-configure /var/lib/cinder bind ~~~ pcs resource bundle update openstack-cinder-volume \ storage-map add \ id=cinder-volume-var-lib-cinder \ source-dir=/var/lib/cinder \ target-dir=/var/lib/cinder \ options=shared ~~~ [2] https://docs.docker.com/storage/bind-mounts/#configure-bind-propagation ###before applying workaround podman inspect $cinder-volume { "Type": "bind", "Name": "", "Source": "/var/lib/cinder", "Destination": "/var/lib/cinder", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": true, "Propagation": "rprivate" }, ###after applying workaround podman inspect $cinder-volume { "Type": "bind", "Name": "", "Source": "/var/lib/cinder", "Destination": "/var/lib/cinder", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": true, "Propagation": "shared" }, [3] https://opendev.org/openstack/puppet-tripleo/src/branch/master/manifests/profile/pacemaker/cinder/volume_bundle.pp#L187
Thanks and agree that's what i meant by day 2, if we do stack updates the the change may disappear. I ll talk to the team see what we can do, as this is not a core cinder change we may be able to backport it.
I'm starting to look at this now, and just want to point out the final (engineering) solution may not follow the path used by workarounds. Changing the volume propagation settings for the entire /var/lib/cinder directory is probably not a good idea. One possibility (which I have not tested) is to use the CinderVolumeOptVolumes THT parameter to add an additional bind mount for the conversion directory.
Alan, While waiting for this to reach ON_QA, verification wise all I need to to populate the two below? CinderImageConversionNfsShare: 'a.b.c.d:/someshare' CinderImageConversionNfsOptions: <- anything special I should check here? Then deploy the overcloud followed by creating a volume from a qcow2 image, while I refresh/monitor the said NFS conversion folder for temp file(s)? Should I test this whilst Glance/Cinder also use NFS as back ends Or just say a simple Cinder LVM back end but with NFS conversion folder? Thanks
(In reply to Tzach Shefi from comment #27) > Alan, > > While waiting for this to reach ON_QA, > verification wise all I need to to populate the two below? > > CinderImageConversionNfsShare: 'a.b.c.d:/someshare' > CinderImageConversionNfsOptions: <- anything special I should check > here? That should do it. Nothing special about CinderImageConversionNfsOptions, just need to verify what you put there is used by the client. > Then deploy the overcloud followed by creating a volume from a qcow2 image, > while I refresh/monitor the said NFS conversion folder for temp file(s)? That's what I did. Just be sure to use a large enough image so it takes long enough to do the conversion that you "see" the file. > Should I test this whilst Glance/Cinder also use NFS as back ends > Or just say a simple Cinder LVM back end but with NFS conversion folder? I don't think it matters _what_ triggers the conversion, just that it takes place on the NFS storage.
Verified on: openstack-tripleo-heat-templates-11.6.1-2.20220116004909.64b2e88.el8ost.noarch Added this to my overcloud_deploy.sh (undercloud) [stack@undercloud-0 ~]$ cat virt/extra_templates.yaml parameter_defaults: BarbicanSimpleCryptoGlobalDefault: true CinderImageConversionNfsShare: 'A.B.C.D:/root/cinderconv' used an real internal IP address instead of letters. CinderImageConversionNfsOptions: 'nfsvers=3' ---> I wanted to test a mount option also works. As it runs out I due to my network/nfs share I kept failing with with the default nfsv4.x mount, but v3 worked fine. After successfully redeploying the overcloud, we now get the conversion mounts added on controllers. A.B.C.D:/root/cinderconv on /var/lib/cinder_image_conversion type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=A.B.C.D,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.46.4.48) Now lets upload a few images: (overcloud) [stack@undercloud-0 ~]$ glance image-create --disk-format qcow2 --container-format bare --file rhel-server-7.9-update-9-x86_64-kvm.qcow2 --name rhel7.9.qcow2 +------------------+----------------------------------------------------------------------------------+ | Property | Value | +------------------+----------------------------------------------------------------------------------+ | checksum | 82dea14f2fac8b27009c5f926f885f5a | | container_format | bare | | created_at | 2022-02-07T17:58:09Z | | direct_url | swift+config://ref1/glance/54fc0b3e-ad1d-4419-9f88-67adba52a9eb | | disk_format | qcow2 | | id | 54fc0b3e-ad1d-4419-9f88-67adba52a9eb | | min_disk | 0 | | min_ram | 0 | | name | rhel7.9.qcow2 | | os_hash_algo | sha512 | | os_hash_value | 5076869a5a0bed649fbfcb3373d1d85090bc315c1c9ccf714307dbe20b2b52f6af9ebc1657f0d3c0 | | | 2dd11775966d67165f3530a8fb176b0ddfccc850059fbb31 | | os_hidden | False | | owner | e07efcd0337e4d5eb28345c068695f84 | | protected | False | | size | 845833216 | | status | active | | stores | default_backend | | tags | [] | | updated_at | 2022-02-07T17:58:14Z | | virtual_size | Not available | | visibility | shared | +------------------+----------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ glance image-create --disk-format raw --container-format bare --file rhel-server-7.9-update-9-x86_64-kvm.raw --name rhel7.9.raw +------------------+----------------------------------------------------------------------------------+ | Property | Value | +------------------+----------------------------------------------------------------------------------+ | checksum | 1e45e025443492de8d6e4b38ec265abe | | container_format | bare | | created_at | 2022-02-07T18:38:15Z | | direct_url | swift+config://ref1/glance/3d648be7-fd06-46a2-a69b-3df5d3899d21 | | disk_format | raw | | id | 3d648be7-fd06-46a2-a69b-3df5d3899d21 | | min_disk | 0 | | min_ram | 0 | | name | rhel7.9.raw | | os_hash_algo | sha512 | | os_hash_value | 9266d25c2b74881526a5807fd594d4a145db99cde1e52fdbbefcade841daaefffa18cef0eb131760 | | | 06369a91e5c6e7d72f1df2ae68d24d889e057236b9d59e77 | | os_hidden | False | | owner | e07efcd0337e4d5eb28345c068695f84 | | protected | False | | size | 10737418240 | | status | active | | stores | default_backend | | tags | [] | | updated_at | 2022-02-07T18:39:11Z | | virtual_size | Not available | | visibility | shared | +------------------+----------------------------------------------------------------------------------+ Now lets create a volume from each of the images, while also monitoring the conversion folder/mount (overcloud) [stack@undercloud-0 ~]$ cinder create 10 --image rhel7.9.qcow2 --name VolFromQcow2Image --poll +--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | true | | consistencygroup_id | None | | created_at | 2022-02-07T18:43:55.000000 | | description | None | | encrypted | False | | id | d734f0fe-5ba3-4fb5-a74e-35aa7920bc7b | | links | [{'rel': 'self', 'href': 'http://10.0.0.129:8776/v3/e07efcd0337e4d5eb28345c068695f84/volumes/d734f0fe-5ba3-4fb5-a74e-35aa7920bc7b'}, {'rel': 'bookmark', 'href': 'http://10.0.0.129:8776/e07efcd0337e4d5eb28345c068695f84/volumes/d734f0fe-5ba3-4fb5-a74e-35aa7920bc7b'}] | | metadata | {} | | migration_status | None | | multiattach | False | | name | VolFromQcow2Image | | os-vol-host-attr:host | hostgroup@tripleo_iscsi#tripleo_iscsi | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | e07efcd0337e4d5eb28345c068695f84 | | replication_status | None | | size | 10 | | snapshot_id | None | | source_volid | None | | status | available | | updated_at | 2022-02-07T18:45:03.000000 | | user_id | 1c242ea20ca040d5b4d3af171142119d | | volume_image_metadata | {'signature_verified': 'False', 'image_id': '54fc0b3e-ad1d-4419-9f88-67adba52a9eb', 'image_name': 'rhel7.9.qcow2', 'checksum': '82dea14f2fac8b27009c5f926f885f5a', 'container_format': 'bare', 'disk_format': 'qcow2', 'min_disk': '0', 'min_ram': '0', 'size': '845833216'} | | volume_type | tripleo | +--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ Below is the nfs mounted conversion folder, on controller-2 where c-vol is running. [root@controller-2 cinder_image_conversion]# ll total 847808 -rw-------. 1 nobody nobody 0 Feb 7 18:44 image_download_54fc0b3e-ad1d-4419-9f88-67adba52a9eb_em8kadwy -rw-------. 1 nobody nobody 845833216 Feb 7 18:44 image_fetch_54fc0b3e-ad1d-4419-9f88-67adba52a9eb_62kd66m4hostgroup@tripleo_iscsi -rwxrwxrwx. 1 root root 0 Feb 6 07:26 titan32 -> temp file created by me on nfs share. Same folder after conversion/volume creation completed, as expected the files other than my temp test file were deleted. [root@controller-2 cinder_image_conversion]# ll total 0 -rwxrwxrwx. 1 root root 0 Feb 6 07:26 titan32 Good to verify, we noticed a conversion taking place on the nfs mapped share.
OK here is what I managed to make work, After several failed attempts to troubleshot nfsv4 share mount from my nfs server as well as from a netapp NFS. I decided to create a new NFS share on my undercloud [stack@undercloud-0 ~]$ cat /etc/exports /data *(rw) Soppted iptables on firewall so as to not have to deal with NFS firewall rules. Changed my conversion parameter to: CinderImageConversionNfsShare: '192.168.24.1:/data' Reran overcloud_deploy.sh which completed without an error yay!! Now having an NFSv4 mount: [root@controller-2 ~]# mount | grep 192.168 192.168.24.1:/data on /var/lib/cinder_image_conversion type nfs4 (rw,relatime,context=system_u:object_r:container_file_t:s0,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.24.23,local_lock=none,addr=192.168.24.1,_netdev) Lets upload two images to glance: (overcloud) [stack@undercloud-0 ~]$ glance image-create --name rhe7.9.qcow2 --disk-format qcow2 --container-format bare --file rhel-server-7.9-update-9-x86_64-kvm.qcow2 --progress [=============================>] 100% +------------------+----------------------------------------------------------------------------------+ | Property | Value | +------------------+----------------------------------------------------------------------------------+ | checksum | 82dea14f2fac8b27009c5f926f885f5a | | container_format | bare | | created_at | 2022-02-15T13:55:57Z | | direct_url | swift+config://ref1/glance/00dc6ba8-5543-489e-9039-28edb6003053 | | disk_format | qcow2 | | id | 00dc6ba8-5543-489e-9039-28edb6003053 | | min_disk | 0 | | min_ram | 0 | | name | rhe7.9.qcow2 | | os_hash_algo | sha512 | | os_hash_value | 5076869a5a0bed649fbfcb3373d1d85090bc315c1c9ccf714307dbe20b2b52f6af9ebc1657f0d3c0 | | | 2dd11775966d67165f3530a8fb176b0ddfccc850059fbb31 | | os_hidden | False | | owner | 4cadb1d0a677497697e05e865067e3e6 | | protected | False | | size | 845833216 | | status | active | | stores | default_backend | | tags | [] | | updated_at | 2022-02-15T13:56:03Z | | virtual_size | Not available | | visibility | shared | +------------------+----------------------------------------------------------------------------------+ (overcloud) [stack@undercloud-0 ~]$ glance image-create --name rhe7.9.raw --disk-format raw --container-format bare --file rhel-server-7.9-update-9-x86_64-kvm.raw --progress [=============================>] 100% +------------------+----------------------------------------------------------------------------------+ | Property | Value | +------------------+----------------------------------------------------------------------------------+ | checksum | 1e45e025443492de8d6e4b38ec265abe | | container_format | bare | | created_at | 2022-02-15T13:57:39Z | | direct_url | swift+config://ref1/glance/aeba71b0-a321-449f-9118-a6e2d2f3adbc | | disk_format | raw | | id | aeba71b0-a321-449f-9118-a6e2d2f3adbc | | min_disk | 0 | | min_ram | 0 | | name | rhe7.9.raw | | os_hash_algo | sha512 | | os_hash_value | 9266d25c2b74881526a5807fd594d4a145db99cde1e52fdbbefcade841daaefffa18cef0eb131760 | | | 06369a91e5c6e7d72f1df2ae68d24d889e057236b9d59e77 | | os_hidden | False | | owner | 4cadb1d0a677497697e05e865067e3e6 | | protected | False | | size | 10737418240 | | status | active | | stores | default_backend | | tags | [] | | updated_at | 2022-02-15T13:58:36Z | | virtual_size | Not available | | visibility | shared | +------------------+----------------------------------------------------------------------------------+ Now lets create volumes from these image which should cause the conversion to kick in and use the nfs mount. (overcloud) [stack@undercloud-0 ~]$ cinder create 10 --image rhe7.9.qcow2 --name rhel.qcow.vol-t2 --poll +--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | true | | consistencygroup_id | None | | created_at | 2022-02-15T14:23:14.000000 | | description | None | | encrypted | False | | id | c9c6d447-f1aa-48f2-b61c-c8bda421b603 | | links | [{'rel': 'self', 'href': 'http://10.0.0.131:8776/v3/4cadb1d0a677497697e05e865067e3e6/volumes/c9c6d447-f1aa-48f2-b61c-c8bda421b603'}, {'rel': 'bookmark', 'href': 'http://10.0.0.131:8776/4cadb1d0a677497697e05e865067e3e6/volumes/c9c6d447-f1aa-48f2-b61c-c8bda421b603'}] | | metadata | {} | | migration_status | None | | multiattach | False | | name | rhel.qcow.vol-t2 | | os-vol-host-attr:host | hostgroup@tripleo_netapp#cinder_volumes | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 4cadb1d0a677497697e05e865067e3e6 | | replication_status | None | | size | 10 | | snapshot_id | None | | source_volid | None | | status | available | | updated_at | 2022-02-15T14:25:17.000000 | | user_id | fe350c30be2b4ae7807c7f90960a99e4 | | volume_image_metadata | {'signature_verified': 'False', 'image_id': '00dc6ba8-5543-489e-9039-28edb6003053', 'image_name': 'rhe7.9.qcow2', 'checksum': '82dea14f2fac8b27009c5f926f885f5a', 'container_format': 'bare', 'disk_format': 'qcow2', 'min_disk': '0', 'min_ram': '0', 'size': '845833216'} | | volume_type | tripleo | +--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ I was missing I final chmod 777 on share to enable permission. During volume creation I the share folder contained the temp files, which were later removed. [root@undercloud-0 ~]# ll /data/ total 826012 -rw-------. 1 nobody nobody 0 Feb 15 09:23 image_download_00dc6ba8-5543-489e-9039-28edb6003053_2j1v0ofd -rw-------. 1 nobody nobody 845833216 Feb 15 09:23 image_fetch_00dc6ba8-5543-489e-9039-28edb6003053_3k1xgayqhostgroup@tripleo_netapp -rw-r--r--. 1 nobody nobody 0 Feb 15 09:22 kuk After creation completed: [root@undercloud-0 ~]# ll /data/ total 0 -rw-r--r--. 1 nobody nobody 0 Feb 15 09:22 kuk --> ignore my permissions test file. Thus happy to report that I also tested support NFS mount at the conversion directory over an NFSv4.2 share.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat OpenStack Platform 16.2 (openstack-tripleo-heat-templates) security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0995
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days