Bug 1898484 - [OSP 16.1 Quickstart] Cinder iSCSI/LVM configuration lost after reboot leading to cinder volumes being unavailable
Summary: [OSP 16.1 Quickstart] Cinder iSCSI/LVM configuration lost after reboot leadin...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-tripleo-heat-templates
Version: 16.1 (Train)
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: z4
: 16.1 (Train on RHEL 8.2)
Assignee: Alan Bishop
QA Contact: David Rosenfeld
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-11-17 09:52 UTC by Robert Heinzmann
Modified: 2021-03-17 15:36 UTC (History)
6 users (show)

Fixed In Version: openstack-tripleo-heat-templates-11.3.2-1.20210104205651.el8ost
Doc Type: Bug Fix
Doc Text:
Before this update, the connection data created by an iSCSI/LVM Block Storage backend was not stored persistently, which resulted in volumes not being accessible after a reboot. With this update, the connection data is stored persistently, and the volumes are accessible after a system reboot.
Clone Of:
Environment:
Last Closed: 2021-03-17 15:35:42 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1905617 0 None None None 2020-11-25 20:07:52 UTC
OpenStack gerrit 769239 0 None MERGED Ensure cinder LVM volumes work after system restart 2021-02-15 06:41:38 UTC
Red Hat Product Errata RHBA-2021:0817 0 None None None 2021-03-17 15:36:05 UTC

Description Robert Heinzmann 2020-11-17 09:52:55 UTC
Description of problem:

When deploying the OSP 16.1 quickstart guide (https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/quick_start_guide/index) on a single node for test environments, after a node reboot, all cinder volumes are unavailable and the VM's using it do not start as all iSCSI configuration is lost.

Configuration (pre reboot)

~~~
[stack@osp16 ]$ sudo podman exec -it iscsid targetcli  ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 3]
  | | o- iqn.2010-10.org.openstack:volume-2ea6c184-0e82-4349-920d-e3e3f97fa427  [/dev/cinder-volumes/volume-2ea6c184-0e82-4349-920d-e3e3f97fa427 (20.0GiB) write-thru activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- iqn.2010-10.org.openstack:volume-7484aa13-7330-491d-a8a0-49543f8321e7  [/dev/cinder-volumes/volume-7484aa13-7330-491d-a8a0-49543f8321e7 (100.0GiB) write-thru activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- iqn.2010-10.org.openstack:volume-9d6f6666-fe67-4d6c-9a59-b74d303c6e54  [/dev/cinder-volumes/volume-9d6f6666-fe67-4d6c-9a59-b74d303c6e54 (20.0GiB) write-thru activated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 3]
  | o- iqn.2010-10.org.openstack:volume-2ea6c184-0e82-4349-920d-e3e3f97fa427 ............................................. [TPGs: 1]
  | | o- tpg1 .......................................................................................... [no-gen-acls, auth per-acl]
  | |   o- acls .......................................................................................................... [ACLs: 1]
  | |   | o- iqn.1994-05.com.redhat:cb878caf4fcd ...................................................... [1-way auth, Mapped LUNs: 1]
  | |   |   o- mapped_lun0 ................. [lun0 block/iqn.2010-10.org.openstack:volume-2ea6c184-0e82-4349-920d-e3e3f97fa427 (rw)]
  | |   o- luns .......................................................................................................... [LUNs: 1]
  | |   | o- lun0  [block/iqn.2010-10.org.openstack:volume-2ea6c184-0e82-4349-920d-e3e3f97fa427 (/dev/cinder-volumes/volume-2ea6c184-0e82-4349-920d-e3e3f97fa427) (default_tg_pt_gp)]
  | |   o- portals .................................................................................................... [Portals: 1]
  | |     o- xxx.xxx.xxx.xxx:3260 ............................................................................................. [OK]
  | o- iqn.2010-10.org.openstack:volume-7484aa13-7330-491d-a8a0-49543f8321e7 ............................................. [TPGs: 1]
  | | o- tpg1 .......................................................................................... [no-gen-acls, auth per-acl]
  | |   o- acls .......................................................................................................... [ACLs: 1]
  | |   | o- iqn.1994-05.com.redhat:cb878caf4fcd ...................................................... [1-way auth, Mapped LUNs: 1]
  | |   |   o- mapped_lun0 ................. [lun0 block/iqn.2010-10.org.openstack:volume-7484aa13-7330-491d-a8a0-49543f8321e7 (rw)]
  | |   o- luns .......................................................................................................... [LUNs: 1]
  | |   | o- lun0  [block/iqn.2010-10.org.openstack:volume-7484aa13-7330-491d-a8a0-49543f8321e7 (/dev/cinder-volumes/volume-7484aa13-7330-491d-a8a0-49543f8321e7) (default_tg_pt_gp)]
  | |   o- portals .................................................................................................... [Portals: 1]
  | |     o- xxx.xxx.xxx.xxx:3260 ............................................................................................. [OK]
  | o- iqn.2010-10.org.openstack:volume-9d6f6666-fe67-4d6c-9a59-b74d303c6e54 ............................................. [TPGs: 1]
  |   o- tpg1 .......................................................................................... [no-gen-acls, auth per-acl]
  |     o- acls .......................................................................................................... [ACLs: 1]
  |     | o- iqn.1994-05.com.redhat:cb878caf4fcd ...................................................... [1-way auth, Mapped LUNs: 1]
  |     |   o- mapped_lun0 ................. [lun0 block/iqn.2010-10.org.openstack:volume-9d6f6666-fe67-4d6c-9a59-b74d303c6e54 (rw)]
  |     o- luns .......................................................................................................... [LUNs: 1]
  |     | o- lun0  [block/iqn.2010-10.org.openstack:volume-9d6f6666-fe67-4d6c-9a59-b74d303c6e54 (/dev/cinder-volumes/volume-9d6f6666-fe67-4d6c-9a59-b74d303c6e54) (default_tg_pt_gp)]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- xxx.xxx.xxx.xxx:3260 ............................................................................................. [OK]
  o- loopback ......................................................................................................... [Targets: 0]
~~~

Reboot the node

~~~
[stack@osp16 ]$ sudo reboot 
Connection to osp16 closed by remote host.
Connection to osp16 closed.
~~~

After the reboot:

~~~
[stack@osp16 ]$ sudo podman exec -it iscsid targetcli  ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 0]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 0]
  o- loopback ......................................................................................................... [Targets: 0]

~~~



Version-Release number of selected component (if applicable):

OpenStack 16.1 QuickStart 

How reproducible:

Always

Steps to Reproduce:

1. Deploy QuickStart Environment (using the default LVM loop)
2. Create VM using volume
3. Reboot node
3. Try to start the VM

Actual results:

VM fails to start, reason is iSCSI configuration is empty.

Expected results:

VM can be started after a reboot.

Additional info:

Comment 1 Rabi Mishra 2020-11-17 13:14:17 UTC
Is the cinder-lvm-losetup.service running after reboot? If yes, then it sounds like a recurrence of https://bugzilla.redhat.com/show_bug.cgi?id=1638922. Assigning to correct DFG i.e DFG:Storage.

#sudo systemctl status cinder-lvm-losetup

Comment 3 Rabi Mishra 2020-11-18 03:55:40 UTC
Did you try restarting the service and check why it's failing? Once running it would restore loopback device after reboot.

Comment 9 Luigi Toscano 2020-11-18 16:57:52 UTC
Robert, can you please change the systemd unit cinder-lvm-losetup.service and remove any reference to lvm2-lvmetad.service  from the Requires and After keys. Does it help?

I suspect we are simply missing Id3b807955dadf8c99a9f90a96069f85c9d05e18d, and especially https://review.opendev.org/#/c/760899/.

Comment 11 Robert Heinzmann 2020-11-18 17:20:11 UTC
I already did this as shown in https://bugzilla.redhat.com/show_bug.cgi?id=1898484#c4

Comment 13 Luigi Toscano 2020-11-18 17:39:27 UTC
(In reply to Robert Heinzmann from comment #11)
> I already did this as shown in
> https://bugzilla.redhat.com/show_bug.cgi?id=1898484#c4

Sorry, you are right. Then maybe something else can be used as dependency. Adding Alan.

Comment 15 Alan Bishop 2020-11-18 18:18:46 UTC
I don't know what to make of the details in comment #5, except to say that nothing about the LVM backend will work if the losetup fails as reported in comment #4. 

Gating it on waiting for the lvm2-monitor service makes sense. The point is we need lvm2 ready, so that it can scan and update things when the loopback device goes active. 

But doesn't explain why losetup is failing. Maybe it's running waaaay too soon, and needs to include local-fs.target in the list of Requires.

Comment 16 Robert Heinzmann 2020-11-18 21:00:27 UTC
I applied the following patch

~~~
[stack@osp16 ~]$ diff -u /etc/systemd/system/cinder-lvm-losetup.service /tmp/cinder-lvm-losetup.service.ORG
--- /etc/systemd/system/cinder-lvm-losetup.service	2020-11-18 20:46:26.920007897 +0000
+++ /tmp/cinder-lvm-losetup.service.ORG	2020-11-18 07:49:16.161265600 +0000
@@ -2,9 +2,9 @@
 Description=Cinder LVM losetup
 DefaultDependencies=no
 Conflicts=umount.target
-Requires=lvm2-monitor.service systemd-udev-settle.service
+Requires=lvm2-lvmetad.service systemd-udev-settle.service
 Before=local-fs.target umount.target
-After=var.mount lvm2-monitor.service systemd-udev-settle.service
+After=lvm2-lvmetad.service systemd-udev-settle.service
 
 [Service]
 Type=oneshot
~~~

Now after a reboot, I see the loopdevice as well as all LV's automatically started

~~~
[stack@osp16 ~]$  sudo systemctl status cinder-lvm-losetup.service 
● cinder-lvm-losetup.service - Cinder LVM losetup
   Loaded: loaded (/etc/systemd/system/cinder-lvm-losetup.service; enabled; vendor preset: disabled)
   Active: active (exited) since Wed 2020-11-18 20:48:32 UTC; 1min 34s ago
 Main PID: 1281 (code=exited, status=0/SUCCESS)
    Tasks: 0 (limit: 838860)
   Memory: 0B
   CGroup: /system.slice/cinder-lvm-losetup.service

Nov 18 20:48:32 osp16.elconaslab.de systemd[1]: Starting Cinder LVM losetup...
Nov 18 20:48:32 osp16.elconaslab.de systemd[1]: Started Cinder LVM losetup.
~~~

LVM:
~~~
[stack@osp16 ~]$ sudo lvs
  LV                                          VG             Attr       LSize   Pool                Origin Data%  Meta%  Move Log Cpy%Sync Convert
  cinder-volumes-pool                         cinder-volumes twi-aotz-- 400.00g                            27.64  18.12                           
  volume-0a0837ed-e59d-4eb5-aaee-227df76e4fb9 cinder-volumes Vwi-a-tz--  20.00g cinder-volumes-pool        0.00                                   
  volume-0c5f4253-d203-4589-8cda-3c7a50d391e3 cinder-volumes Vwi-a-tz--  15.00g cinder-volumes-pool        14.56                                  
  volume-119f7533-42c1-42ea-9a27-07c6bb4b87bb cinder-volumes Vwi-a-tz--  15.00g cinder-volumes-pool        12.70                                  
  volume-1968b643-1275-4921-9846-0d18adf37092 cinder-volumes Vwi-a-tz--  20.00g cinder-volumes-pool        2.21                                   
  volume-22aa046f-6ee1-4b8f-aa1d-07bfc3617b34 cinder-volumes Vwi-a-tz--  15.00g cinder-volumes-pool        10.85                                  
  volume-27f02af7-5200-478c-be94-cc1943dc263e cinder-volumes Vwi-a-tz--  10.00g cinder-volumes-pool        8.61                                   
  volume-59907561-472e-42e2-bcc6-c29694b28449 cinder-volumes Vwi-a-tz--  20.00g cinder-volumes-pool        94.29                                  
  volume-5c0dd7fa-3e6d-423d-8b57-c85023034fe7 cinder-volumes Vwi-a-tz--  15.00g cinder-volumes-pool        41.52                                  
  volume-7484aa13-7330-491d-a8a0-49543f8321e7 cinder-volumes Vwi-a-tz-- 100.00g cinder-volumes-pool        55.71                                  
  volume-81872a1f-cf62-4929-9ac2-e969e688bf33 cinder-volumes Vwi-a-tz--  20.00g cinder-volumes-pool        94.78                                  
  volume-988ec97b-7fc4-4461-b270-3726d0dff7da cinder-volumes Vwi-a-tz--  30.00g cinder-volumes-pool        12.66                                  
  volume-af48c814-5a88-43a1-b485-a856b8bd2b78 cinder-volumes Vwi-a-tz--  15.00g cinder-volumes-pool        0.00                                   
  root                                        vg0            -wi-ao----  55.00g                                                                   
  swap                                        vg0            -wi-ao----   8.00g                                                                   
  tmp                                         vg0            -wi-ao----  20.00g                                                                   
  var                                         vg0            -wi-ao---- 600.00g                                                                   
~~~

However iSCSI is still empty and VM's can not be started

~~~
[openstack ADMIN] [stack@osp16 ~]$ sudo podman exec -it iscsid targetcli  ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 0]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 0]
  o- loopback ......................................................................................................... [Targets: 0]
~~~

Comment 17 Robert Heinzmann 2020-11-18 21:01:44 UTC
Sorry, this is the right direction of the patch 

~~~
[openstack ADMIN] [stack@osp16 ~]$ diff -u /tmp/cinder-lvm-losetup.service.ORG /etc/systemd/system/cinder-lvm-losetup.service
--- /tmp/cinder-lvm-losetup.service.ORG	2020-11-18 07:49:16.161265600 +0000
+++ /etc/systemd/system/cinder-lvm-losetup.service	2020-11-18 20:46:26.920007897 +0000
@@ -2,9 +2,9 @@
 Description=Cinder LVM losetup
 DefaultDependencies=no
 Conflicts=umount.target
-Requires=lvm2-lvmetad.service systemd-udev-settle.service
+Requires=lvm2-monitor.service systemd-udev-settle.service
 Before=local-fs.target umount.target
-After=lvm2-lvmetad.service systemd-udev-settle.service
+After=var.mount lvm2-monitor.service systemd-udev-settle.service
 
 [Service]
 Type=oneshot
~~~

Comment 18 Robert Heinzmann 2020-11-19 07:49:49 UTC
It seems the problem is related that "sudo pcs resource restart openstack-cinder-volume" actually cleanes up /etc/target where the config is stored. Looking at the logs I found: 

The config is stored:

~~~
[root@osp16 /]# grep -e save /var/log/cinder/cinder-volume.log
2020-11-19 06:28:21.823 73 DEBUG oslo_concurrency.processutils [req-f2174bf8-fc03-48b0-a5c6-2f313cac7c6e e144c6f0a4704b39a8e6bec9253dc990 71bb3a3122764bba97c48a95efc93d00 - default default] Running cmd (subprocess): cinder-rtstool save execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:372
2020-11-19 06:28:21.979 73 DEBUG oslo_concurrency.processutils [req-f2174bf8-fc03-48b0-a5c6-2f313cac7c6e e144c6f0a4704b39a8e6bec9253dc990 71bb3a3122764bba97c48a95efc93d00 - default default] CMD "cinder-rtstool save" returned: 0 in 0.155s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:409

[stack@osp16 ~]$ sudo podman exec -it openstack-cinder-volume-podman-0 ls -al /etc/target/saveconfig.json
-rw-r--r--. 1 root root 62346 Nov 19 07:36 /etc/target/saveconfig.json
~~~

However then trying to load after a reboot, it failes to load :

~~~
...
2020-11-19 07:01:15.329 72 DEBUG oslo_concurrency.processutils [req-6eb65b8d-a2c3-431b-bdc1-2e151ebcb9c4 - - - - -] Running cmd (subprocess): cinder-rtstool restore execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:372
2020-11-19 07:01:15.658 72 DEBUG oslo_concurrency.processutils [req-6eb65b8d-a2c3-431b-bdc1-2e151ebcb9c4 - - - - -] CMD "cinder-rtstool restore" returned: 1 in 0.329s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:409
....
2020-11-19 07:01:24.257 72 WARNING cinder.volume.targets.lio [req-6eb65b8d-a2c3-431b-bdc1-2e151ebcb9c4 - - - - -] Failed to restore iscsi LIO configuration.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
....
~~~

After a reboot the file is gone:

~~~
()[root@osp16 /]# ls -al /etc/target/
total 24
drwxr-xr-x. 4 root root 4096 Oct 21 08:20 .
drwxr-xr-x. 1 root root 4096 Nov 19 07:00 ..
drwxr-xr-x. 2 root root 4096 Dec 12  2019 backup
drwxr-xr-x. 2 root root 4096 Apr 14  2020 pr
~~~

Manual execution yields

~~~
()[root@osp16 /]# cinder-rtstool restore
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/cinder/cmd/rtstool.py", line 229, in restore_from_file
    rtsroot.restore_from_file(configuration_file)
  File "/usr/lib/python3.6/site-packages/rtslib_fb/root.py", line 479, in restore_from_file
    with open(restore_file, "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/etc/target/saveconfig.json'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/cinder-rtstool", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python3.6/site-packages/cinder/cmd/rtstool.py", line 338, in main
    restore_from_file(configuration_file)
  File "/usr/lib/python3.6/site-packages/cinder/cmd/rtstool.py", line 233, in restore_from_file
    {'file_path': configuration_file, 'exc': exc})
cinder.cmd.rtstool.RtstoolError: ('Could not restore configuration file %(file_path)s: %(exc)s', {'file_path': '/etc/target/saveconfig.json', 'exc': FileNotFoundError(2, 'No such file or directory')})
~~~

Same happens after I restart the cinder volume service without a reboot

~~~
[stack@osp16 ~]$ sudo pcs resource restart openstack-cinder-volume
openstack-cinder-volume successfully restarted

[stack@osp16 ~]$ sudo podman exec -it openstack-cinder-volume-podman-0 ls -al /etc/target
total 24
drwxr-xr-x. 4 root root 4096 Oct 21 08:20 .
drwxr-xr-x. 1 root root 4096 Nov 19 07:41 ..
drwxr-xr-x. 2 root root 4096 Dec 12  2019 backup
drwxr-xr-x. 2 root root 4096 Apr 14  2020 pr
~~~

Looking at the podman container configuration, it seems /etc/target is not mapped to the host, so it will not survive container rebuilds and updates, which would probably be another problem then ?

~~~
 Bundle: openstack-cinder-volume
  Podman: image=cluster.common.tag/openstack-cinder-volume:pcmklatest network=host options="--ipc=host --privileged=true --user=root --log-driver=k8s-file --log-opt path=/var/log/containers/stdouts/openstack-cinder-volume.log -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" replicas=1 run-command="/bin/bash /usr/local/bin/kolla_start"
  Storage Mapping:
   options=ro source-dir=/etc/hosts target-dir=/etc/hosts (cinder-volume-etc-hosts)
   options=ro source-dir=/etc/localtime target-dir=/etc/localtime (cinder-volume-etc-localtime)
   options=ro source-dir=/etc/pki/ca-trust/extracted target-dir=/etc/pki/ca-trust/extracted (cinder-volume-etc-pki-ca-trust-extracted)
   options=ro source-dir=/etc/pki/ca-trust/source/anchors target-dir=/etc/pki/ca-trust/source/anchors (cinder-volume-etc-pki-ca-trust-source-anchors)
   options=ro source-dir=/etc/pki/tls/certs/ca-bundle.crt target-dir=/etc/pki/tls/certs/ca-bundle.crt (cinder-volume-etc-pki-tls-certs-ca-bundle.crt)
   options=ro source-dir=/etc/pki/tls/certs/ca-bundle.trust.crt target-dir=/etc/pki/tls/certs/ca-bundle.trust.crt (cinder-volume-etc-pki-tls-certs-ca-bundle.trust.crt)
   options=ro source-dir=/etc/pki/tls/cert.pem target-dir=/etc/pki/tls/cert.pem (cinder-volume-etc-pki-tls-cert.pem)
   options=rw source-dir=/dev/log target-dir=/dev/log (cinder-volume-dev-log)
   options=ro source-dir=/etc/puppet target-dir=/etc/puppet (cinder-volume-etc-puppet)
   options=ro source-dir=/var/lib/config-data/puppet-generated/cinder target-dir=/var/lib/kolla/config_files/src (cinder-volume-var-lib-config-data-puppet-generated-cinder)
   options=z source-dir=/var/log/containers/cinder target-dir=/var/log/cinder (cinder-volume-var-log-containers-cinder)
   options=ro source-dir=/var/lib/kolla/config_files/cinder_volume.json target-dir=/var/lib/kolla/config_files/config.json (cinder-volume-var-lib-kolla-config_files-cinder_volume.json)
   options=ro source-dir=/etc/iscsi target-dir=/var/lib/kolla/config_files/src-iscsid (cinder-volume-etc-iscsi)
   options=ro source-dir=/etc/ceph target-dir=/var/lib/kolla/config_files/src-ceph (cinder-volume-etc-ceph)
   options=ro source-dir=/lib/modules target-dir=/lib/modules (cinder-volume-lib-modules)
   options=rw source-dir=/dev/ target-dir=/dev/ (cinder-volume-dev-)
   options=rw source-dir=/run/ target-dir=/run/ (cinder-volume-run-)
   options=rw source-dir=/sys target-dir=/sys (cinder-volume-sys)
   options=z source-dir=/var/lib/cinder target-dir=/var/lib/cinder (cinder-volume-var-lib-cinder)
   options=z source-dir=/var/lib/iscsi target-dir=/var/lib/iscsi (cinder-volume-var-lib-iscsi)
~~~

Comment 19 Robert Heinzmann 2020-11-19 10:11:34 UTC
To verify this (NOT A FIX !!) I created /etc/target on the host and mapped it to the container, by modifying the pacemaker config:

[openstack ADMIN] [root@osp16 ~]$ sudo mkdir /etc/target
[openstack ADMIN] [root@osp16 ~]$ chmod 777 /etc/target/

Then I added this to the pacemaker config

   <storage-mapping id="cinder-volume-etc-target" options="z" source-dir="/etc/target" target-dir="/etc/target"/>

Restarting pacemaker now keeps the config across pacemaker restarts

~~~
[openstack ADMIN] [stack@osp16 ~]$ sudo podman exec -it openstack-cinder-volume-podman-0 cinder-rtstool save
[openstack ADMIN] [stack@osp16 ~]$ ls -al /etc/target/
total 80
drwxrwxrwx.   2 root root  4096 Nov 19 08:14 .
drwxr-xr-x. 159 root root 12288 Nov 19 08:11 ..
-rw-r--r--.   1 root root 62341 Nov 19 08:14 saveconfig.json
[openstack ADMIN] [stack@osp16 ~]$ sudo podman exec -it iscsid targetcli  ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block ................................................................................................. [Storage Objects: 13]
  | | o- iqn.2010-10.org.openstack:volume-0a0837ed-e59d-4eb5-aaee-227df76e4fb9  [/dev/cinder-volumes/volume-0a0837ed-e59d-4eb5-aaee-227df76e4fb9 (20.0GiB) write-thru activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- iqn.2010-10.org.openstack:volume-0c5f4253-d203-4589-8cda-3c7a50d391e3  [/dev/cinder-volumes/volume-0c5f4253-d203-4589-8cda-3c7a50d391e3 (15.0GiB) write-thru activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
... works ....

Restart cinder volume and check 

[openstack ADMIN] [stack@osp16 ~]$ sudo pcs resource restart openstack-cinder-volume
openstack-cinder-volume successfully restarted

[openstack ADMIN] [stack@osp16 ~]$ sudo podman exec -it iscsid targetcli  ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block ................................................................................................. [Storage Objects: 13]
  | | o- iqn.2010-10.org.openstack:volume-0a0837ed-e59d-4eb5-aaee-227df76e4fb9  [/dev/cinder-volumes/volume-0a0837ed-e59d-4eb5-aaee-227df76e4fb9 (20.0GiB) write-thru activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- iqn.2010-10.org.openstack:volume-0c5f4253-d203-4589-8cda-3c7a50d391e3  [/dev/cinder-volumes/volume-0c5f4253-d203-4589-8cda-3c7a50d391e3 (15.0GiB) write-thru activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- iqn.2010-10.org.openstack:volume-119f7533-42c1-42ea-9a27-07c6bb4b87bb  [/dev/cinder-volumes/volume-119f7533-42c1-42ea-9a27-07c6bb4b87bb (15.0GiB) write-thru activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- iqn.2010-10.org.openstack:volume-1968b643-1275-4921-9846-0d18adf37092  [/dev/cinder-volumes/volume-1968b643-1275-4921-9846-0d18adf37092 (20.0GiB) write-thru activated]

~~~

Also after a reboot the data is there, however other parts of the system behave strange now (e.g. northbound db not starting). Not sure if this is related.

Comment 20 Alan Bishop 2020-11-19 16:23:33 UTC
A cinder colleague confirmed the LVM backend (actually, the lio target stuff) *does* need /etc/target to be persistent. This is a latent oversight from when cinder was containerized back in OSP-13.

It seems there are two problems to address, both of which can be handled in the same patch in THT.

1. The systemd dependencies for creating the loopback device need to be updated per comment #17
2. The /etc/target directory needs to be persistent

Regarding the northbound db not starting, I've also observed that happening, but am pretty sure it's unrelated (I can't imagine how it would be).

Comment 23 Robert Heinzmann 2020-12-21 09:12:28 UTC
It seems the upstream fix has been merged. Any timeframe for backporting the fix ? Possibly 16.1.4 ?

Comment 24 Alan Bishop 2020-12-21 17:49:11 UTC
Upstream backports are underway, and the hope is the train backport will land in time for inclusion in 16.1.4. The Victoria backport was approved today.

Comment 33 Tzach Shefi 2021-02-15 10:50:25 UTC
Verified on:
openstack-tripleo-heat-templates-11.3.2-1.20210104205661.el8ost.noarch

Using guide I had deployed a standalone LVM system.

Boot an instance, attached a Cinder LVM volume: 

Showing iscsid before rebooting.


(overcloud) [stack@titan93 ~]$ sudo podman exec -it iscsid targetcli  ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 1]
  | | o- iqn.2010-10.org.openstack:volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b  [/dev/cinder-volumes/volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b (1.0GiB) write-thru activated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2010-10.org.openstack:volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b ............................................. [TPGs: 1]
  |   o- tpg1 .......................................................................................... [no-gen-acls, auth per-acl]
  |     o- acls .......................................................................................................... [ACLs: 1]
  |     | o- iqn.1994-05.com.redhat:0f1f273c84 ........................................................ [1-way auth, Mapped LUNs: 1]
  |     |   o- mapped_lun0 ................. [lun0 block/iqn.2010-10.org.openstack:volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b (rw)]
  |     o- luns .......................................................................................................... [LUNs: 1]
  |     | o- lun0  [block/iqn.2010-10.org.openstack:volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b (/dev/cinder-volumes/volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b) (default_tg_pt_gp)]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 192.168.25.2:3260 ................................................................................................ [OK]
  o- loopback ......................................................................................................... [Targets: 0]


Now lets reboot the system:
(overcloud) [stack@titan93 ~]$ sudo reboot 
Connection to titan93 closed by remote host.
Connection to titan93 closed.

Re-login and recheck iscsi connection


(overcloud) [stack@titan93 ~]$ nova list
+--------------------------------------+-------+--------+------------+-------------+-----------------+
| ID                                   | Name  | Status | Task State | Power State | Networks        |
+--------------------------------------+-------+--------+------------+-------------+-----------------+
| 953d7dc4-40db-41db-99cd-d6224dffcb8e | inst1 | ACTIVE | -          | Running     | test=192.0.2.58 |
+--------------------------------------+-------+--------+------------+-------------+-----------------+
(overcloud) [stack@titan93 ~]$ cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| ID                                   | Status | Name | Size | Volume Type | Bootable | Attached to                          |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| e42cfb6d-7d2d-45d0-9802-b8e63bd9295b | in-use | -    | 1    | tripleo     | false    | 953d7dc4-40db-41db-99cd-d6224dffcb8e |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+


(overcloud) [stack@titan93 ~]$ uptime
 10:47:25 up 1 min,  1 user,  load average: 3.96, 1.84, 0.69   -> we rebooted the system


(overcloud) [stack@titan93 ~]$ sudo podman exec -it iscsid targetcli  ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 1]
  | | o- iqn.2010-10.org.openstack:volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b  [/dev/cinder-volumes/volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b (1.0GiB) write-thru activated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 1]
  | o- iqn.2010-10.org.openstack:volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b ............................................. [TPGs: 1]
  |   o- tpg1 .......................................................................................... [no-gen-acls, auth per-acl]
  |     o- acls .......................................................................................................... [ACLs: 1]
  |     | o- iqn.1994-05.com.redhat:0f1f273c84 ........................................................ [1-way auth, Mapped LUNs: 1]
  |     |   o- mapped_lun0 ................. [lun0 block/iqn.2010-10.org.openstack:volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b (rw)]
  |     o- luns .......................................................................................................... [LUNs: 1]
  |     | o- lun0  [block/iqn.2010-10.org.openstack:volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b (/dev/cinder-volumes/volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b) (default_tg_pt_gp)]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 192.168.25.2:3260 ................................................................................................ [OK]
  o- loopback ......................................................................................................... [Targets: 0]

This time connection info is kept and cinder lvm volume is attached to the instance.
Good to verify.

Comment 39 errata-xmlrpc 2021-03-17 15:35:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.1.4 director bug fix advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:0817


Note You need to log in before you can comment on or make changes to this bug.