Description of problem: When deploying the OSP 16.1 quickstart guide (https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.1/html/quick_start_guide/index) on a single node for test environments, after a node reboot, all cinder volumes are unavailable and the VM's using it do not start as all iSCSI configuration is lost. Configuration (pre reboot) ~~~ [stack@osp16 ]$ sudo podman exec -it iscsid targetcli ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 3] | | o- iqn.2010-10.org.openstack:volume-2ea6c184-0e82-4349-920d-e3e3f97fa427 [/dev/cinder-volumes/volume-2ea6c184-0e82-4349-920d-e3e3f97fa427 (20.0GiB) write-thru activated] | | | o- alua ................................................................................................... [ALUA Groups: 1] | | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] | | o- iqn.2010-10.org.openstack:volume-7484aa13-7330-491d-a8a0-49543f8321e7 [/dev/cinder-volumes/volume-7484aa13-7330-491d-a8a0-49543f8321e7 (100.0GiB) write-thru activated] | | | o- alua ................................................................................................... [ALUA Groups: 1] | | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] | | o- iqn.2010-10.org.openstack:volume-9d6f6666-fe67-4d6c-9a59-b74d303c6e54 [/dev/cinder-volumes/volume-9d6f6666-fe67-4d6c-9a59-b74d303c6e54 (20.0GiB) write-thru activated] | | o- alua ................................................................................................... [ALUA Groups: 1] | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 3] | o- iqn.2010-10.org.openstack:volume-2ea6c184-0e82-4349-920d-e3e3f97fa427 ............................................. [TPGs: 1] | | o- tpg1 .......................................................................................... [no-gen-acls, auth per-acl] | | o- acls .......................................................................................................... [ACLs: 1] | | | o- iqn.1994-05.com.redhat:cb878caf4fcd ...................................................... [1-way auth, Mapped LUNs: 1] | | | o- mapped_lun0 ................. [lun0 block/iqn.2010-10.org.openstack:volume-2ea6c184-0e82-4349-920d-e3e3f97fa427 (rw)] | | o- luns .......................................................................................................... [LUNs: 1] | | | o- lun0 [block/iqn.2010-10.org.openstack:volume-2ea6c184-0e82-4349-920d-e3e3f97fa427 (/dev/cinder-volumes/volume-2ea6c184-0e82-4349-920d-e3e3f97fa427) (default_tg_pt_gp)] | | o- portals .................................................................................................... [Portals: 1] | | o- xxx.xxx.xxx.xxx:3260 ............................................................................................. [OK] | o- iqn.2010-10.org.openstack:volume-7484aa13-7330-491d-a8a0-49543f8321e7 ............................................. [TPGs: 1] | | o- tpg1 .......................................................................................... [no-gen-acls, auth per-acl] | | o- acls .......................................................................................................... [ACLs: 1] | | | o- iqn.1994-05.com.redhat:cb878caf4fcd ...................................................... [1-way auth, Mapped LUNs: 1] | | | o- mapped_lun0 ................. [lun0 block/iqn.2010-10.org.openstack:volume-7484aa13-7330-491d-a8a0-49543f8321e7 (rw)] | | o- luns .......................................................................................................... [LUNs: 1] | | | o- lun0 [block/iqn.2010-10.org.openstack:volume-7484aa13-7330-491d-a8a0-49543f8321e7 (/dev/cinder-volumes/volume-7484aa13-7330-491d-a8a0-49543f8321e7) (default_tg_pt_gp)] | | o- portals .................................................................................................... [Portals: 1] | | o- xxx.xxx.xxx.xxx:3260 ............................................................................................. [OK] | o- iqn.2010-10.org.openstack:volume-9d6f6666-fe67-4d6c-9a59-b74d303c6e54 ............................................. [TPGs: 1] | o- tpg1 .......................................................................................... [no-gen-acls, auth per-acl] | o- acls .......................................................................................................... [ACLs: 1] | | o- iqn.1994-05.com.redhat:cb878caf4fcd ...................................................... [1-way auth, Mapped LUNs: 1] | | o- mapped_lun0 ................. [lun0 block/iqn.2010-10.org.openstack:volume-9d6f6666-fe67-4d6c-9a59-b74d303c6e54 (rw)] | o- luns .......................................................................................................... [LUNs: 1] | | o- lun0 [block/iqn.2010-10.org.openstack:volume-9d6f6666-fe67-4d6c-9a59-b74d303c6e54 (/dev/cinder-volumes/volume-9d6f6666-fe67-4d6c-9a59-b74d303c6e54) (default_tg_pt_gp)] | o- portals .................................................................................................... [Portals: 1] | o- xxx.xxx.xxx.xxx:3260 ............................................................................................. [OK] o- loopback ......................................................................................................... [Targets: 0] ~~~ Reboot the node ~~~ [stack@osp16 ]$ sudo reboot Connection to osp16 closed by remote host. Connection to osp16 closed. ~~~ After the reboot: ~~~ [stack@osp16 ]$ sudo podman exec -it iscsid targetcli ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 0] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 0] o- loopback ......................................................................................................... [Targets: 0] ~~~ Version-Release number of selected component (if applicable): OpenStack 16.1 QuickStart How reproducible: Always Steps to Reproduce: 1. Deploy QuickStart Environment (using the default LVM loop) 2. Create VM using volume 3. Reboot node 3. Try to start the VM Actual results: VM fails to start, reason is iSCSI configuration is empty. Expected results: VM can be started after a reboot. Additional info:
Is the cinder-lvm-losetup.service running after reboot? If yes, then it sounds like a recurrence of https://bugzilla.redhat.com/show_bug.cgi?id=1638922. Assigning to correct DFG i.e DFG:Storage. #sudo systemctl status cinder-lvm-losetup
Did you try restarting the service and check why it's failing? Once running it would restore loopback device after reboot.
Robert, can you please change the systemd unit cinder-lvm-losetup.service and remove any reference to lvm2-lvmetad.service from the Requires and After keys. Does it help? I suspect we are simply missing Id3b807955dadf8c99a9f90a96069f85c9d05e18d, and especially https://review.opendev.org/#/c/760899/.
I already did this as shown in https://bugzilla.redhat.com/show_bug.cgi?id=1898484#c4
(In reply to Robert Heinzmann from comment #11) > I already did this as shown in > https://bugzilla.redhat.com/show_bug.cgi?id=1898484#c4 Sorry, you are right. Then maybe something else can be used as dependency. Adding Alan.
I will test with lvm2-monitor.service (see [1]) [1] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_basic_system_settings/managing-services-with-systemd_configuring-basic-system-settings
I don't know what to make of the details in comment #5, except to say that nothing about the LVM backend will work if the losetup fails as reported in comment #4. Gating it on waiting for the lvm2-monitor service makes sense. The point is we need lvm2 ready, so that it can scan and update things when the loopback device goes active. But doesn't explain why losetup is failing. Maybe it's running waaaay too soon, and needs to include local-fs.target in the list of Requires.
I applied the following patch ~~~ [stack@osp16 ~]$ diff -u /etc/systemd/system/cinder-lvm-losetup.service /tmp/cinder-lvm-losetup.service.ORG --- /etc/systemd/system/cinder-lvm-losetup.service 2020-11-18 20:46:26.920007897 +0000 +++ /tmp/cinder-lvm-losetup.service.ORG 2020-11-18 07:49:16.161265600 +0000 @@ -2,9 +2,9 @@ Description=Cinder LVM losetup DefaultDependencies=no Conflicts=umount.target -Requires=lvm2-monitor.service systemd-udev-settle.service +Requires=lvm2-lvmetad.service systemd-udev-settle.service Before=local-fs.target umount.target -After=var.mount lvm2-monitor.service systemd-udev-settle.service +After=lvm2-lvmetad.service systemd-udev-settle.service [Service] Type=oneshot ~~~ Now after a reboot, I see the loopdevice as well as all LV's automatically started ~~~ [stack@osp16 ~]$ sudo systemctl status cinder-lvm-losetup.service ● cinder-lvm-losetup.service - Cinder LVM losetup Loaded: loaded (/etc/systemd/system/cinder-lvm-losetup.service; enabled; vendor preset: disabled) Active: active (exited) since Wed 2020-11-18 20:48:32 UTC; 1min 34s ago Main PID: 1281 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 838860) Memory: 0B CGroup: /system.slice/cinder-lvm-losetup.service Nov 18 20:48:32 osp16.elconaslab.de systemd[1]: Starting Cinder LVM losetup... Nov 18 20:48:32 osp16.elconaslab.de systemd[1]: Started Cinder LVM losetup. ~~~ LVM: ~~~ [stack@osp16 ~]$ sudo lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert cinder-volumes-pool cinder-volumes twi-aotz-- 400.00g 27.64 18.12 volume-0a0837ed-e59d-4eb5-aaee-227df76e4fb9 cinder-volumes Vwi-a-tz-- 20.00g cinder-volumes-pool 0.00 volume-0c5f4253-d203-4589-8cda-3c7a50d391e3 cinder-volumes Vwi-a-tz-- 15.00g cinder-volumes-pool 14.56 volume-119f7533-42c1-42ea-9a27-07c6bb4b87bb cinder-volumes Vwi-a-tz-- 15.00g cinder-volumes-pool 12.70 volume-1968b643-1275-4921-9846-0d18adf37092 cinder-volumes Vwi-a-tz-- 20.00g cinder-volumes-pool 2.21 volume-22aa046f-6ee1-4b8f-aa1d-07bfc3617b34 cinder-volumes Vwi-a-tz-- 15.00g cinder-volumes-pool 10.85 volume-27f02af7-5200-478c-be94-cc1943dc263e cinder-volumes Vwi-a-tz-- 10.00g cinder-volumes-pool 8.61 volume-59907561-472e-42e2-bcc6-c29694b28449 cinder-volumes Vwi-a-tz-- 20.00g cinder-volumes-pool 94.29 volume-5c0dd7fa-3e6d-423d-8b57-c85023034fe7 cinder-volumes Vwi-a-tz-- 15.00g cinder-volumes-pool 41.52 volume-7484aa13-7330-491d-a8a0-49543f8321e7 cinder-volumes Vwi-a-tz-- 100.00g cinder-volumes-pool 55.71 volume-81872a1f-cf62-4929-9ac2-e969e688bf33 cinder-volumes Vwi-a-tz-- 20.00g cinder-volumes-pool 94.78 volume-988ec97b-7fc4-4461-b270-3726d0dff7da cinder-volumes Vwi-a-tz-- 30.00g cinder-volumes-pool 12.66 volume-af48c814-5a88-43a1-b485-a856b8bd2b78 cinder-volumes Vwi-a-tz-- 15.00g cinder-volumes-pool 0.00 root vg0 -wi-ao---- 55.00g swap vg0 -wi-ao---- 8.00g tmp vg0 -wi-ao---- 20.00g var vg0 -wi-ao---- 600.00g ~~~ However iSCSI is still empty and VM's can not be started ~~~ [openstack ADMIN] [stack@osp16 ~]$ sudo podman exec -it iscsid targetcli ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 0] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 0] o- loopback ......................................................................................................... [Targets: 0] ~~~
Sorry, this is the right direction of the patch ~~~ [openstack ADMIN] [stack@osp16 ~]$ diff -u /tmp/cinder-lvm-losetup.service.ORG /etc/systemd/system/cinder-lvm-losetup.service --- /tmp/cinder-lvm-losetup.service.ORG 2020-11-18 07:49:16.161265600 +0000 +++ /etc/systemd/system/cinder-lvm-losetup.service 2020-11-18 20:46:26.920007897 +0000 @@ -2,9 +2,9 @@ Description=Cinder LVM losetup DefaultDependencies=no Conflicts=umount.target -Requires=lvm2-lvmetad.service systemd-udev-settle.service +Requires=lvm2-monitor.service systemd-udev-settle.service Before=local-fs.target umount.target -After=lvm2-lvmetad.service systemd-udev-settle.service +After=var.mount lvm2-monitor.service systemd-udev-settle.service [Service] Type=oneshot ~~~
It seems the problem is related that "sudo pcs resource restart openstack-cinder-volume" actually cleanes up /etc/target where the config is stored. Looking at the logs I found: The config is stored: ~~~ [root@osp16 /]# grep -e save /var/log/cinder/cinder-volume.log 2020-11-19 06:28:21.823 73 DEBUG oslo_concurrency.processutils [req-f2174bf8-fc03-48b0-a5c6-2f313cac7c6e e144c6f0a4704b39a8e6bec9253dc990 71bb3a3122764bba97c48a95efc93d00 - default default] Running cmd (subprocess): cinder-rtstool save execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:372 2020-11-19 06:28:21.979 73 DEBUG oslo_concurrency.processutils [req-f2174bf8-fc03-48b0-a5c6-2f313cac7c6e e144c6f0a4704b39a8e6bec9253dc990 71bb3a3122764bba97c48a95efc93d00 - default default] CMD "cinder-rtstool save" returned: 0 in 0.155s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:409 [stack@osp16 ~]$ sudo podman exec -it openstack-cinder-volume-podman-0 ls -al /etc/target/saveconfig.json -rw-r--r--. 1 root root 62346 Nov 19 07:36 /etc/target/saveconfig.json ~~~ However then trying to load after a reboot, it failes to load : ~~~ ... 2020-11-19 07:01:15.329 72 DEBUG oslo_concurrency.processutils [req-6eb65b8d-a2c3-431b-bdc1-2e151ebcb9c4 - - - - -] Running cmd (subprocess): cinder-rtstool restore execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:372 2020-11-19 07:01:15.658 72 DEBUG oslo_concurrency.processutils [req-6eb65b8d-a2c3-431b-bdc1-2e151ebcb9c4 - - - - -] CMD "cinder-rtstool restore" returned: 1 in 0.329s execute /usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py:409 .... 2020-11-19 07:01:24.257 72 WARNING cinder.volume.targets.lio [req-6eb65b8d-a2c3-431b-bdc1-2e151ebcb9c4 - - - - -] Failed to restore iscsi LIO configuration.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. .... ~~~ After a reboot the file is gone: ~~~ ()[root@osp16 /]# ls -al /etc/target/ total 24 drwxr-xr-x. 4 root root 4096 Oct 21 08:20 . drwxr-xr-x. 1 root root 4096 Nov 19 07:00 .. drwxr-xr-x. 2 root root 4096 Dec 12 2019 backup drwxr-xr-x. 2 root root 4096 Apr 14 2020 pr ~~~ Manual execution yields ~~~ ()[root@osp16 /]# cinder-rtstool restore Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/cinder/cmd/rtstool.py", line 229, in restore_from_file rtsroot.restore_from_file(configuration_file) File "/usr/lib/python3.6/site-packages/rtslib_fb/root.py", line 479, in restore_from_file with open(restore_file, "r") as f: FileNotFoundError: [Errno 2] No such file or directory: '/etc/target/saveconfig.json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/bin/cinder-rtstool", line 10, in <module> sys.exit(main()) File "/usr/lib/python3.6/site-packages/cinder/cmd/rtstool.py", line 338, in main restore_from_file(configuration_file) File "/usr/lib/python3.6/site-packages/cinder/cmd/rtstool.py", line 233, in restore_from_file {'file_path': configuration_file, 'exc': exc}) cinder.cmd.rtstool.RtstoolError: ('Could not restore configuration file %(file_path)s: %(exc)s', {'file_path': '/etc/target/saveconfig.json', 'exc': FileNotFoundError(2, 'No such file or directory')}) ~~~ Same happens after I restart the cinder volume service without a reboot ~~~ [stack@osp16 ~]$ sudo pcs resource restart openstack-cinder-volume openstack-cinder-volume successfully restarted [stack@osp16 ~]$ sudo podman exec -it openstack-cinder-volume-podman-0 ls -al /etc/target total 24 drwxr-xr-x. 4 root root 4096 Oct 21 08:20 . drwxr-xr-x. 1 root root 4096 Nov 19 07:41 .. drwxr-xr-x. 2 root root 4096 Dec 12 2019 backup drwxr-xr-x. 2 root root 4096 Apr 14 2020 pr ~~~ Looking at the podman container configuration, it seems /etc/target is not mapped to the host, so it will not survive container rebuilds and updates, which would probably be another problem then ? ~~~ Bundle: openstack-cinder-volume Podman: image=cluster.common.tag/openstack-cinder-volume:pcmklatest network=host options="--ipc=host --privileged=true --user=root --log-driver=k8s-file --log-opt path=/var/log/containers/stdouts/openstack-cinder-volume.log -e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS" replicas=1 run-command="/bin/bash /usr/local/bin/kolla_start" Storage Mapping: options=ro source-dir=/etc/hosts target-dir=/etc/hosts (cinder-volume-etc-hosts) options=ro source-dir=/etc/localtime target-dir=/etc/localtime (cinder-volume-etc-localtime) options=ro source-dir=/etc/pki/ca-trust/extracted target-dir=/etc/pki/ca-trust/extracted (cinder-volume-etc-pki-ca-trust-extracted) options=ro source-dir=/etc/pki/ca-trust/source/anchors target-dir=/etc/pki/ca-trust/source/anchors (cinder-volume-etc-pki-ca-trust-source-anchors) options=ro source-dir=/etc/pki/tls/certs/ca-bundle.crt target-dir=/etc/pki/tls/certs/ca-bundle.crt (cinder-volume-etc-pki-tls-certs-ca-bundle.crt) options=ro source-dir=/etc/pki/tls/certs/ca-bundle.trust.crt target-dir=/etc/pki/tls/certs/ca-bundle.trust.crt (cinder-volume-etc-pki-tls-certs-ca-bundle.trust.crt) options=ro source-dir=/etc/pki/tls/cert.pem target-dir=/etc/pki/tls/cert.pem (cinder-volume-etc-pki-tls-cert.pem) options=rw source-dir=/dev/log target-dir=/dev/log (cinder-volume-dev-log) options=ro source-dir=/etc/puppet target-dir=/etc/puppet (cinder-volume-etc-puppet) options=ro source-dir=/var/lib/config-data/puppet-generated/cinder target-dir=/var/lib/kolla/config_files/src (cinder-volume-var-lib-config-data-puppet-generated-cinder) options=z source-dir=/var/log/containers/cinder target-dir=/var/log/cinder (cinder-volume-var-log-containers-cinder) options=ro source-dir=/var/lib/kolla/config_files/cinder_volume.json target-dir=/var/lib/kolla/config_files/config.json (cinder-volume-var-lib-kolla-config_files-cinder_volume.json) options=ro source-dir=/etc/iscsi target-dir=/var/lib/kolla/config_files/src-iscsid (cinder-volume-etc-iscsi) options=ro source-dir=/etc/ceph target-dir=/var/lib/kolla/config_files/src-ceph (cinder-volume-etc-ceph) options=ro source-dir=/lib/modules target-dir=/lib/modules (cinder-volume-lib-modules) options=rw source-dir=/dev/ target-dir=/dev/ (cinder-volume-dev-) options=rw source-dir=/run/ target-dir=/run/ (cinder-volume-run-) options=rw source-dir=/sys target-dir=/sys (cinder-volume-sys) options=z source-dir=/var/lib/cinder target-dir=/var/lib/cinder (cinder-volume-var-lib-cinder) options=z source-dir=/var/lib/iscsi target-dir=/var/lib/iscsi (cinder-volume-var-lib-iscsi) ~~~
To verify this (NOT A FIX !!) I created /etc/target on the host and mapped it to the container, by modifying the pacemaker config: [openstack ADMIN] [root@osp16 ~]$ sudo mkdir /etc/target [openstack ADMIN] [root@osp16 ~]$ chmod 777 /etc/target/ Then I added this to the pacemaker config <storage-mapping id="cinder-volume-etc-target" options="z" source-dir="/etc/target" target-dir="/etc/target"/> Restarting pacemaker now keeps the config across pacemaker restarts ~~~ [openstack ADMIN] [stack@osp16 ~]$ sudo podman exec -it openstack-cinder-volume-podman-0 cinder-rtstool save [openstack ADMIN] [stack@osp16 ~]$ ls -al /etc/target/ total 80 drwxrwxrwx. 2 root root 4096 Nov 19 08:14 . drwxr-xr-x. 159 root root 12288 Nov 19 08:11 .. -rw-r--r--. 1 root root 62341 Nov 19 08:14 saveconfig.json [openstack ADMIN] [stack@osp16 ~]$ sudo podman exec -it iscsid targetcli ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block ................................................................................................. [Storage Objects: 13] | | o- iqn.2010-10.org.openstack:volume-0a0837ed-e59d-4eb5-aaee-227df76e4fb9 [/dev/cinder-volumes/volume-0a0837ed-e59d-4eb5-aaee-227df76e4fb9 (20.0GiB) write-thru activated] | | | o- alua ................................................................................................... [ALUA Groups: 1] | | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] | | o- iqn.2010-10.org.openstack:volume-0c5f4253-d203-4589-8cda-3c7a50d391e3 [/dev/cinder-volumes/volume-0c5f4253-d203-4589-8cda-3c7a50d391e3 (15.0GiB) write-thru activated] | | | o- alua ................................................................................................... [ALUA Groups: 1] | | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] ... works .... Restart cinder volume and check [openstack ADMIN] [stack@osp16 ~]$ sudo pcs resource restart openstack-cinder-volume openstack-cinder-volume successfully restarted [openstack ADMIN] [stack@osp16 ~]$ sudo podman exec -it iscsid targetcli ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block ................................................................................................. [Storage Objects: 13] | | o- iqn.2010-10.org.openstack:volume-0a0837ed-e59d-4eb5-aaee-227df76e4fb9 [/dev/cinder-volumes/volume-0a0837ed-e59d-4eb5-aaee-227df76e4fb9 (20.0GiB) write-thru activated] | | | o- alua ................................................................................................... [ALUA Groups: 1] | | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] | | o- iqn.2010-10.org.openstack:volume-0c5f4253-d203-4589-8cda-3c7a50d391e3 [/dev/cinder-volumes/volume-0c5f4253-d203-4589-8cda-3c7a50d391e3 (15.0GiB) write-thru activated] | | | o- alua ................................................................................................... [ALUA Groups: 1] | | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] | | o- iqn.2010-10.org.openstack:volume-119f7533-42c1-42ea-9a27-07c6bb4b87bb [/dev/cinder-volumes/volume-119f7533-42c1-42ea-9a27-07c6bb4b87bb (15.0GiB) write-thru activated] | | | o- alua ................................................................................................... [ALUA Groups: 1] | | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] | | o- iqn.2010-10.org.openstack:volume-1968b643-1275-4921-9846-0d18adf37092 [/dev/cinder-volumes/volume-1968b643-1275-4921-9846-0d18adf37092 (20.0GiB) write-thru activated] ~~~ Also after a reboot the data is there, however other parts of the system behave strange now (e.g. northbound db not starting). Not sure if this is related.
A cinder colleague confirmed the LVM backend (actually, the lio target stuff) *does* need /etc/target to be persistent. This is a latent oversight from when cinder was containerized back in OSP-13. It seems there are two problems to address, both of which can be handled in the same patch in THT. 1. The systemd dependencies for creating the loopback device need to be updated per comment #17 2. The /etc/target directory needs to be persistent Regarding the northbound db not starting, I've also observed that happening, but am pretty sure it's unrelated (I can't imagine how it would be).
It seems the upstream fix has been merged. Any timeframe for backporting the fix ? Possibly 16.1.4 ?
Upstream backports are underway, and the hope is the train backport will land in time for inclusion in 16.1.4. The Victoria backport was approved today.
Verified on: openstack-tripleo-heat-templates-11.3.2-1.20210104205661.el8ost.noarch Using guide I had deployed a standalone LVM system. Boot an instance, attached a Cinder LVM volume: Showing iscsid before rebooting. (overcloud) [stack@titan93 ~]$ sudo podman exec -it iscsid targetcli ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 1] | | o- iqn.2010-10.org.openstack:volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b [/dev/cinder-volumes/volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b (1.0GiB) write-thru activated] | | o- alua ................................................................................................... [ALUA Groups: 1] | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 1] | o- iqn.2010-10.org.openstack:volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b ............................................. [TPGs: 1] | o- tpg1 .......................................................................................... [no-gen-acls, auth per-acl] | o- acls .......................................................................................................... [ACLs: 1] | | o- iqn.1994-05.com.redhat:0f1f273c84 ........................................................ [1-way auth, Mapped LUNs: 1] | | o- mapped_lun0 ................. [lun0 block/iqn.2010-10.org.openstack:volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b (rw)] | o- luns .......................................................................................................... [LUNs: 1] | | o- lun0 [block/iqn.2010-10.org.openstack:volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b (/dev/cinder-volumes/volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b) (default_tg_pt_gp)] | o- portals .................................................................................................... [Portals: 1] | o- 192.168.25.2:3260 ................................................................................................ [OK] o- loopback ......................................................................................................... [Targets: 0] Now lets reboot the system: (overcloud) [stack@titan93 ~]$ sudo reboot Connection to titan93 closed by remote host. Connection to titan93 closed. Re-login and recheck iscsi connection (overcloud) [stack@titan93 ~]$ nova list +--------------------------------------+-------+--------+------------+-------------+-----------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+-------+--------+------------+-------------+-----------------+ | 953d7dc4-40db-41db-99cd-d6224dffcb8e | inst1 | ACTIVE | - | Running | test=192.0.2.58 | +--------------------------------------+-------+--------+------------+-------------+-----------------+ (overcloud) [stack@titan93 ~]$ cinder list +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+ | e42cfb6d-7d2d-45d0-9802-b8e63bd9295b | in-use | - | 1 | tripleo | false | 953d7dc4-40db-41db-99cd-d6224dffcb8e | +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+ (overcloud) [stack@titan93 ~]$ uptime 10:47:25 up 1 min, 1 user, load average: 3.96, 1.84, 0.69 -> we rebooted the system (overcloud) [stack@titan93 ~]$ sudo podman exec -it iscsid targetcli ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 1] | | o- iqn.2010-10.org.openstack:volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b [/dev/cinder-volumes/volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b (1.0GiB) write-thru activated] | | o- alua ................................................................................................... [ALUA Groups: 1] | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 1] | o- iqn.2010-10.org.openstack:volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b ............................................. [TPGs: 1] | o- tpg1 .......................................................................................... [no-gen-acls, auth per-acl] | o- acls .......................................................................................................... [ACLs: 1] | | o- iqn.1994-05.com.redhat:0f1f273c84 ........................................................ [1-way auth, Mapped LUNs: 1] | | o- mapped_lun0 ................. [lun0 block/iqn.2010-10.org.openstack:volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b (rw)] | o- luns .......................................................................................................... [LUNs: 1] | | o- lun0 [block/iqn.2010-10.org.openstack:volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b (/dev/cinder-volumes/volume-e42cfb6d-7d2d-45d0-9802-b8e63bd9295b) (default_tg_pt_gp)] | o- portals .................................................................................................... [Portals: 1] | o- 192.168.25.2:3260 ................................................................................................ [OK] o- loopback ......................................................................................................... [Targets: 0] This time connection info is kept and cinder lvm volume is attached to the instance. Good to verify.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.4 director bug fix advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2021:0817