Bug 1886762
| Summary: | [RFE] support NFS mount at the conversion directory | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Michele Valsecchi <mvalsecc> |
| Component: | openstack-tripleo-heat-templates | Assignee: | Alan Bishop <abishop> |
| Status: | CLOSED ERRATA | QA Contact: | Tzach Shefi <tshefi> |
| Severity: | medium | Docs Contact: | Erin Peterson <erpeters> |
| Priority: | high | ||
| Version: | 16.1 (Train) | CC: | abishop, erpeters, gcharot, gfidente, jamsmith, jmelvin, knoha, ltoscano, mariel, mburns, migawa, mvalsecc, tshefi |
| Target Milestone: | z2 | Keywords: | FutureFeature, Triaged |
| Target Release: | 16.2 (Train on RHEL 8.4) | ||
| Hardware: | Unspecified | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-heat-templates-11.6.1-2.20220116004909.64b2e88.el8ost | Doc Type: | Enhancement |
| Doc Text: |
The Block Storage service (cinder) can now use an external NFS share to perform image format conversion of Image service (glance) images on the overcloud Controller nodes. Using this functionality prevents the space on the node from being completely filled during a conversion operation.
+
See https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html-single/advanced_overcloud_customization/index#proc_configuring-external-nfs-share-conversion_portal
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-03-23 22:28:29 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Michele Valsecchi
2020-10-09 10:54:29 UTC
I've done some digging and we see from the workaround [1] it changes the bind propagation from "Propagation": "rprivate" , the workaround sets to "Propagation": "shared" [2] . We also notice it's propagation is not configured by default [3]. So the problem here is how to make the workaround persistent, we can apply it during post deploy scripts but that means during the deploy it would be set back to default which causes cinder-volume service to go down; then when post deploy scripts run it would change it to propagation shared which would fix it back. I guess the goal would be to make it configurable in puppet so we can pass that with the deploy so it always stays shared and never breaks cinder-volume during deploys. [1] 1. Remove the definition of /var/lib/cinder bind ~~~ pcs resource bundle update openstack-cinder-volume \ storage-map remove cinder-volume-var-lib-cinder ~~~ 2. Re-configure /var/lib/cinder bind ~~~ pcs resource bundle update openstack-cinder-volume \ storage-map add \ id=cinder-volume-var-lib-cinder \ source-dir=/var/lib/cinder \ target-dir=/var/lib/cinder \ options=shared ~~~ [2] https://docs.docker.com/storage/bind-mounts/#configure-bind-propagation ###before applying workaround podman inspect $cinder-volume { "Type": "bind", "Name": "", "Source": "/var/lib/cinder", "Destination": "/var/lib/cinder", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": true, "Propagation": "rprivate" }, ###after applying workaround podman inspect $cinder-volume { "Type": "bind", "Name": "", "Source": "/var/lib/cinder", "Destination": "/var/lib/cinder", "Driver": "", "Mode": "", "Options": [ "rbind" ], "RW": true, "Propagation": "shared" }, [3] https://opendev.org/openstack/puppet-tripleo/src/branch/master/manifests/profile/pacemaker/cinder/volume_bundle.pp#L187 Thanks and agree that's what i meant by day 2, if we do stack updates the the change may disappear. I ll talk to the team see what we can do, as this is not a core cinder change we may be able to backport it. I'm starting to look at this now, and just want to point out the final (engineering) solution may not follow the path used by workarounds. Changing the volume propagation settings for the entire /var/lib/cinder directory is probably not a good idea. One possibility (which I have not tested) is to use the CinderVolumeOptVolumes THT parameter to add an additional bind mount for the conversion directory. Alan, While waiting for this to reach ON_QA, verification wise all I need to to populate the two below? CinderImageConversionNfsShare: 'a.b.c.d:/someshare' CinderImageConversionNfsOptions: <- anything special I should check here? Then deploy the overcloud followed by creating a volume from a qcow2 image, while I refresh/monitor the said NFS conversion folder for temp file(s)? Should I test this whilst Glance/Cinder also use NFS as back ends Or just say a simple Cinder LVM back end but with NFS conversion folder? Thanks (In reply to Tzach Shefi from comment #27) > Alan, > > While waiting for this to reach ON_QA, > verification wise all I need to to populate the two below? > > CinderImageConversionNfsShare: 'a.b.c.d:/someshare' > CinderImageConversionNfsOptions: <- anything special I should check > here? That should do it. Nothing special about CinderImageConversionNfsOptions, just need to verify what you put there is used by the client. > Then deploy the overcloud followed by creating a volume from a qcow2 image, > while I refresh/monitor the said NFS conversion folder for temp file(s)? That's what I did. Just be sure to use a large enough image so it takes long enough to do the conversion that you "see" the file. > Should I test this whilst Glance/Cinder also use NFS as back ends > Or just say a simple Cinder LVM back end but with NFS conversion folder? I don't think it matters _what_ triggers the conversion, just that it takes place on the NFS storage. Verified on:
openstack-tripleo-heat-templates-11.6.1-2.20220116004909.64b2e88.el8ost.noarch
Added this to my overcloud_deploy.sh
(undercloud) [stack@undercloud-0 ~]$ cat virt/extra_templates.yaml
parameter_defaults:
BarbicanSimpleCryptoGlobalDefault: true
CinderImageConversionNfsShare: 'A.B.C.D:/root/cinderconv' used an real internal IP address instead of letters.
CinderImageConversionNfsOptions: 'nfsvers=3' ---> I wanted to test a mount option also works.
As it runs out I due to my network/nfs share
I kept failing with with the default nfsv4.x mount, but v3 worked fine.
After successfully redeploying the overcloud, we now get the conversion mounts added on controllers.
A.B.C.D:/root/cinderconv on /var/lib/cinder_image_conversion type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=A.B.C.D,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.46.4.48)
Now lets upload a few images:
(overcloud) [stack@undercloud-0 ~]$ glance image-create --disk-format qcow2 --container-format bare --file rhel-server-7.9-update-9-x86_64-kvm.qcow2 --name rhel7.9.qcow2
+------------------+----------------------------------------------------------------------------------+
| Property | Value |
+------------------+----------------------------------------------------------------------------------+
| checksum | 82dea14f2fac8b27009c5f926f885f5a |
| container_format | bare |
| created_at | 2022-02-07T17:58:09Z |
| direct_url | swift+config://ref1/glance/54fc0b3e-ad1d-4419-9f88-67adba52a9eb |
| disk_format | qcow2 |
| id | 54fc0b3e-ad1d-4419-9f88-67adba52a9eb |
| min_disk | 0 |
| min_ram | 0 |
| name | rhel7.9.qcow2 |
| os_hash_algo | sha512 |
| os_hash_value | 5076869a5a0bed649fbfcb3373d1d85090bc315c1c9ccf714307dbe20b2b52f6af9ebc1657f0d3c0 |
| | 2dd11775966d67165f3530a8fb176b0ddfccc850059fbb31 |
| os_hidden | False |
| owner | e07efcd0337e4d5eb28345c068695f84 |
| protected | False |
| size | 845833216 |
| status | active |
| stores | default_backend |
| tags | [] |
| updated_at | 2022-02-07T17:58:14Z |
| virtual_size | Not available |
| visibility | shared |
+------------------+----------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ glance image-create --disk-format raw --container-format bare --file rhel-server-7.9-update-9-x86_64-kvm.raw --name rhel7.9.raw
+------------------+----------------------------------------------------------------------------------+
| Property | Value |
+------------------+----------------------------------------------------------------------------------+
| checksum | 1e45e025443492de8d6e4b38ec265abe |
| container_format | bare |
| created_at | 2022-02-07T18:38:15Z |
| direct_url | swift+config://ref1/glance/3d648be7-fd06-46a2-a69b-3df5d3899d21 |
| disk_format | raw |
| id | 3d648be7-fd06-46a2-a69b-3df5d3899d21 |
| min_disk | 0 |
| min_ram | 0 |
| name | rhel7.9.raw |
| os_hash_algo | sha512 |
| os_hash_value | 9266d25c2b74881526a5807fd594d4a145db99cde1e52fdbbefcade841daaefffa18cef0eb131760 |
| | 06369a91e5c6e7d72f1df2ae68d24d889e057236b9d59e77 |
| os_hidden | False |
| owner | e07efcd0337e4d5eb28345c068695f84 |
| protected | False |
| size | 10737418240 |
| status | active |
| stores | default_backend |
| tags | [] |
| updated_at | 2022-02-07T18:39:11Z |
| virtual_size | Not available |
| visibility | shared |
+------------------+----------------------------------------------------------------------------------+
Now lets create a volume from each of the images,
while also monitoring the conversion folder/mount
(overcloud) [stack@undercloud-0 ~]$ cinder create 10 --image rhel7.9.qcow2 --name VolFromQcow2Image --poll
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2022-02-07T18:43:55.000000 |
| description | None |
| encrypted | False |
| id | d734f0fe-5ba3-4fb5-a74e-35aa7920bc7b |
| links | [{'rel': 'self', 'href': 'http://10.0.0.129:8776/v3/e07efcd0337e4d5eb28345c068695f84/volumes/d734f0fe-5ba3-4fb5-a74e-35aa7920bc7b'}, {'rel': 'bookmark', 'href': 'http://10.0.0.129:8776/e07efcd0337e4d5eb28345c068695f84/volumes/d734f0fe-5ba3-4fb5-a74e-35aa7920bc7b'}] |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | VolFromQcow2Image |
| os-vol-host-attr:host | hostgroup@tripleo_iscsi#tripleo_iscsi |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | e07efcd0337e4d5eb28345c068695f84 |
| replication_status | None |
| size | 10 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2022-02-07T18:45:03.000000 |
| user_id | 1c242ea20ca040d5b4d3af171142119d |
| volume_image_metadata | {'signature_verified': 'False', 'image_id': '54fc0b3e-ad1d-4419-9f88-67adba52a9eb', 'image_name': 'rhel7.9.qcow2', 'checksum': '82dea14f2fac8b27009c5f926f885f5a', 'container_format': 'bare', 'disk_format': 'qcow2', 'min_disk': '0', 'min_ram': '0', 'size': '845833216'} |
| volume_type | tripleo |
+--------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Below is the nfs mounted conversion folder, on controller-2 where c-vol is running.
[root@controller-2 cinder_image_conversion]# ll
total 847808
-rw-------. 1 nobody nobody 0 Feb 7 18:44 image_download_54fc0b3e-ad1d-4419-9f88-67adba52a9eb_em8kadwy
-rw-------. 1 nobody nobody 845833216 Feb 7 18:44 image_fetch_54fc0b3e-ad1d-4419-9f88-67adba52a9eb_62kd66m4hostgroup@tripleo_iscsi
-rwxrwxrwx. 1 root root 0 Feb 6 07:26 titan32 -> temp file created by me on nfs share.
Same folder after conversion/volume creation completed, as expected the files other than my temp test file were deleted.
[root@controller-2 cinder_image_conversion]# ll
total 0
-rwxrwxrwx. 1 root root 0 Feb 6 07:26 titan32
Good to verify, we noticed a conversion taking place on the nfs mapped share.
OK here is what I managed to make work,
After several failed attempts to troubleshot nfsv4 share mount from my nfs server
as well as from a netapp NFS.
I decided to create a new NFS share on my undercloud
[stack@undercloud-0 ~]$ cat /etc/exports
/data *(rw)
Soppted iptables on firewall so as to not have to deal with NFS firewall rules.
Changed my conversion parameter to:
CinderImageConversionNfsShare: '192.168.24.1:/data'
Reran overcloud_deploy.sh which completed without an error yay!!
Now having an NFSv4 mount:
[root@controller-2 ~]# mount | grep 192.168
192.168.24.1:/data on /var/lib/cinder_image_conversion type nfs4 (rw,relatime,context=system_u:object_r:container_file_t:s0,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.24.23,local_lock=none,addr=192.168.24.1,_netdev)
Lets upload two images to glance:
(overcloud) [stack@undercloud-0 ~]$ glance image-create --name rhe7.9.qcow2 --disk-format qcow2 --container-format bare --file rhel-server-7.9-update-9-x86_64-kvm.qcow2 --progress
[=============================>] 100%
+------------------+----------------------------------------------------------------------------------+
| Property | Value |
+------------------+----------------------------------------------------------------------------------+
| checksum | 82dea14f2fac8b27009c5f926f885f5a |
| container_format | bare |
| created_at | 2022-02-15T13:55:57Z |
| direct_url | swift+config://ref1/glance/00dc6ba8-5543-489e-9039-28edb6003053 |
| disk_format | qcow2 |
| id | 00dc6ba8-5543-489e-9039-28edb6003053 |
| min_disk | 0 |
| min_ram | 0 |
| name | rhe7.9.qcow2 |
| os_hash_algo | sha512 |
| os_hash_value | 5076869a5a0bed649fbfcb3373d1d85090bc315c1c9ccf714307dbe20b2b52f6af9ebc1657f0d3c0 |
| | 2dd11775966d67165f3530a8fb176b0ddfccc850059fbb31 |
| os_hidden | False |
| owner | 4cadb1d0a677497697e05e865067e3e6 |
| protected | False |
| size | 845833216 |
| status | active |
| stores | default_backend |
| tags | [] |
| updated_at | 2022-02-15T13:56:03Z |
| virtual_size | Not available |
| visibility | shared |
+------------------+----------------------------------------------------------------------------------+
(overcloud) [stack@undercloud-0 ~]$ glance image-create --name rhe7.9.raw --disk-format raw --container-format bare --file rhel-server-7.9-update-9-x86_64-kvm.raw --progress
[=============================>] 100%
+------------------+----------------------------------------------------------------------------------+
| Property | Value |
+------------------+----------------------------------------------------------------------------------+
| checksum | 1e45e025443492de8d6e4b38ec265abe |
| container_format | bare |
| created_at | 2022-02-15T13:57:39Z |
| direct_url | swift+config://ref1/glance/aeba71b0-a321-449f-9118-a6e2d2f3adbc |
| disk_format | raw |
| id | aeba71b0-a321-449f-9118-a6e2d2f3adbc |
| min_disk | 0 |
| min_ram | 0 |
| name | rhe7.9.raw |
| os_hash_algo | sha512 |
| os_hash_value | 9266d25c2b74881526a5807fd594d4a145db99cde1e52fdbbefcade841daaefffa18cef0eb131760 |
| | 06369a91e5c6e7d72f1df2ae68d24d889e057236b9d59e77 |
| os_hidden | False |
| owner | 4cadb1d0a677497697e05e865067e3e6 |
| protected | False |
| size | 10737418240 |
| status | active |
| stores | default_backend |
| tags | [] |
| updated_at | 2022-02-15T13:58:36Z |
| virtual_size | Not available |
| visibility | shared |
+------------------+----------------------------------------------------------------------------------+
Now lets create volumes from these image which should cause the conversion to kick in and use the nfs mount.
(overcloud) [stack@undercloud-0 ~]$ cinder create 10 --image rhe7.9.qcow2 --name rhel.qcow.vol-t2 --poll
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2022-02-15T14:23:14.000000 |
| description | None |
| encrypted | False |
| id | c9c6d447-f1aa-48f2-b61c-c8bda421b603 |
| links | [{'rel': 'self', 'href': 'http://10.0.0.131:8776/v3/4cadb1d0a677497697e05e865067e3e6/volumes/c9c6d447-f1aa-48f2-b61c-c8bda421b603'}, {'rel': 'bookmark', 'href': 'http://10.0.0.131:8776/4cadb1d0a677497697e05e865067e3e6/volumes/c9c6d447-f1aa-48f2-b61c-c8bda421b603'}] |
| metadata | {} |
| migration_status | None |
| multiattach | False |
| name | rhel.qcow.vol-t2 |
| os-vol-host-attr:host | hostgroup@tripleo_netapp#cinder_volumes |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 4cadb1d0a677497697e05e865067e3e6 |
| replication_status | None |
| size | 10 |
| snapshot_id | None |
| source_volid | None |
| status | available |
| updated_at | 2022-02-15T14:25:17.000000 |
| user_id | fe350c30be2b4ae7807c7f90960a99e4 |
| volume_image_metadata | {'signature_verified': 'False', 'image_id': '00dc6ba8-5543-489e-9039-28edb6003053', 'image_name': 'rhe7.9.qcow2', 'checksum': '82dea14f2fac8b27009c5f926f885f5a', 'container_format': 'bare', 'disk_format': 'qcow2', 'min_disk': '0', 'min_ram': '0', 'size': '845833216'} |
| volume_type | tripleo |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
I was missing I final chmod 777 on share to enable permission.
During volume creation I the share folder contained the temp files, which were later removed.
[root@undercloud-0 ~]# ll /data/
total 826012
-rw-------. 1 nobody nobody 0 Feb 15 09:23 image_download_00dc6ba8-5543-489e-9039-28edb6003053_2j1v0ofd
-rw-------. 1 nobody nobody 845833216 Feb 15 09:23 image_fetch_00dc6ba8-5543-489e-9039-28edb6003053_3k1xgayqhostgroup@tripleo_netapp
-rw-r--r--. 1 nobody nobody 0 Feb 15 09:22 kuk
After creation completed:
[root@undercloud-0 ~]# ll /data/
total 0
-rw-r--r--. 1 nobody nobody 0 Feb 15 09:22 kuk --> ignore my permissions test file.
Thus happy to report that I also tested support NFS mount at the conversion directory over an NFSv4.2 share.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat OpenStack Platform 16.2 (openstack-tripleo-heat-templates) security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0995 The needinfo request[s] on this closed bug have been removed as they have been unresolved for 365 days |