From the Compute Node, Volumes are no accessible

This is been noticed while firing up the VMs
~~~
Error: Failed to launch instance "nonprodagent": Please try again later [Error: internal error: process exited while connecting to monitor: 2015-12-11T00:09:01.619135Z qemu-kvm: -drive file=/dev/disk/by-path/ip-10.123.170.22:3260-iscsi-iqn.2010-10.org.openstack:volume-2ec61633-af88-4e8c-95d0-c19edbd91074-lun-0,if=none,id=drive-virtio].
~~~


iSCSI Target is not running on the faulty node. However iSCSI services are running on the non-patched server

We are currently seeing possibility of

https://access.redhat.com/solutions/1443103
https://bugzilla.redhat.com/show_bug.cgi?id=1213430

But it creating empty  saveconfig.json

Hi team,

just to add :
we tried the following after customer upgraded & restart :
# yum update openstack-cinder
# cinder-rtstool save
# openstack-service restart cinder

But it didnt work , probably because at this stage the configuration were empty.
# cinder-rtstool get_target

Uploaded cinder show for all volumes on the storage node on collab shell .

Regards,
Kaospm R

Error in description was due to bug https://bugs.launchpad.net/cinder/1387866 that is already fixed in the next release, but in this case that was just masking another issue, that the cinder volume node was not finding on initialization the volumes to create the volume exports when using LIO.

That issue has been found, it was only creating problems from updates from cinder versions that didn't include LIO persistence and systems that were not using target.service, and resolved.  But to recover this specific node I would recommend to manually create the exports:

1- Create targets:

Look target information in db (we will retrieve volumes that were set to error by cinder on restart):

  sudo mysql -e 'select provider_location, provider_auth from cinder.volumes where deleted=0 and status="error" and provider_location is not NULL;'

That will output something like:

 +--------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------+
 | provider_location                                                                                                                                                  | provider_auth                                  |
 +--------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------+
 | 192.168.121.240:3260,iqn.2010-10.org.openstack:volume-11040865-e428-45ca-a7ff-be99461b19c1 iqn.2010-10.org.openstack:volume-11040865-e428-45ca-a7ff-be99461b19c1 0 | CHAP zvACAevwzD5mTDsrrFiC ZhCqwZ4CnxPHRWJY9P3g |
 | 192.168.121.240:3260,iqn.2010-10.org.openstack:volume-59f17e88-4851-40d3-bd5d-5eb0f7987b4a iqn.2010-10.org.openstack:volume-59f17e88-4851-40d3-bd5d-5eb0f7987b4a 0 | CHAP 6gHs6rPmFfUrspRyEqbB wW5sg948Ro4JHNqH7KPc |
 +--------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------+
Where we have:                                    ^ Volume                                    ^ IQN                                                                          ^ User               ^ Password

With that information we construct the commands to create the targets following this pattern:
 cinder-rtstool create /dev/$VOLUME_GROUP/$Volume $IQN $User $Password

For this example:
 cinder-rtstool create /dev/cinder-volumes/volume-11040865-e428-45ca-a7ff-be99461b19c1 iqn.2010-10.org.openstack:volume-11040865-e428-45ca-a7ff-be99461b19c1 zvACAevwzD5mTDsrrFiC ZhCqwZ4CnxPHRWJY9P3g
 cinder-rtstool create /dev/cinder-volumes/volume-59f17e88-4851-40d3-bd5d-5eb0f7987b4a iqn.2010-10.org.openstack:volume-59f17e88-4851-40d3-bd5d-5eb0f7987b4a 6gHs6rPmFfUrspRyEqbB wW5sg948Ro4JHNqH7KPc

2- Change status in cinder, as volumes will be in error status
 cinder reset-state --state in-use 11040865-e428-45ca-a7ff-be99461b19c1
 cinder reset-state --state in-use 59f17e88-4851-40d3-bd5d-5eb0f7987b4a

3- Save configuration (in case there's a reboot before there are any changes in the targets)
 targetcli saveconfig

With that, volumes will be properly exported.

As to how to prevent this situation when upgrading any other node from a pre LIO persistence version, the procedure should include executing "targetcli saveconfig" after the cinder volume service has been stopped.

Well, if Nova instance is running on a different host then you need to add its initiator name into the ACL for this target.

1. get the initiator name of the relevant compute host:

less /etc/iscsi/initiatorname.iscsi

2. Retrieve chap credentials from the connection info for that volume from Nova's block_device_mapping table:

select connection_info from block_device_mapping where volume_id = <volume-id>;

Now you should add a new initiator to the target:

cinder-rtsool add-initiator <target-iqn> <chap-username> <chap-password] <initiator-name>

chap-username/chap-password - connection info contains that information. see (2)
initiator-name - should be found in /etc/iscsi/initiatorname.iscsi, see (1)

Tested using:
openstack-cinder-2014.1.5-8.el7ost.noarch
python-cinder-2014.1.5-8.el7ost.noarch
python-cinderclient-1.0.9-2.el7ost.noarch

Verification flow:

[root@lironrhel7rhos5 ~(keystone_admin)]# nova boot --image a9f77b57-faec-435c-8f6f-663894c49edc --flavor 1 --nic net-id=f1c95b3b-304a-460c-92d8-88c622310ed7 vm1

[root@lironrhel7rhos5 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+---------------------+
| ID                                   | Name | Status | Task State | Power State | Networks            |
+--------------------------------------+------+--------+------------+-------------+---------------------+
| a187a8d2-a130-4aa7-a1f9-0ed05d204bab | vm1  | BUILD  | spawning   | NOSTATE     | public=172.24.4.227 |
+--------------------------------------+------+--------+------------+-------------+---------------------+
                                                                                                        
[root@lironrhel7rhos5 ~(keystone_admin)]# cinder create 1 --display-name vol1

[root@lironrhel7rhos5 ~(keystone_admin)]# cinder create 1 --display-name vol2

[root@lironrhel7rhos5 ~(keystone_admin)]# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+
| 2359a240-4bd0-4b46-a059-e06f464036de | available |     vol2     |  1   |     None    |  false   |             |
| 507abe1d-aec7-4306-861e-55a5b30baf88 | available |     vol1     |  1   |     None    |  false   |             |
+--------------------------------------+-----------+--------------+------+-------------+----------+-------------+

[root@lironrhel7rhos5 ~(keystone_admin)]# nova volume-attach a187a8d2-a130-4aa7-a1f9-0ed05d204bab 507abe1d-aec7-4306-861e-55a5b30baf88
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdb                             |
| id       | 507abe1d-aec7-4306-861e-55a5b30baf88 |
| serverId | a187a8d2-a130-4aa7-a1f9-0ed05d204bab |
| volumeId | 507abe1d-aec7-4306-861e-55a5b30baf88 |
+----------+--------------------------------------+

[root@lironrhel7rhos5 ~(keystone_admin)]# nova volume-attach a187a8d2-a130-4aa7-a1f9-0ed05d204bab 2359a240-4bd0-4b46-a059-e06f464036de
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdc                             |
| id       | 2359a240-4bd0-4b46-a059-e06f464036de |
| serverId | a187a8d2-a130-4aa7-a1f9-0ed05d204bab |
| volumeId | 2359a240-4bd0-4b46-a059-e06f464036de |
+----------+--------------------------------------+

[root@lironrhel7rhos5 ~(keystone_admin)]# cinder list
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  | Status | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
| 2359a240-4bd0-4b46-a059-e06f464036de | in-use |     vol2     |  1   |     None    |  false   | a187a8d2-a130-4aa7-a1f9-0ed05d204bab |
| 507abe1d-aec7-4306-861e-55a5b30baf88 | in-use |     vol1     |  1   |     None    |  false   | a187a8d2-a130-4aa7-a1f9-0ed05d204bab |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+

[root@lironrhel7rhos5 ~(keystone_admin)]# cinder-rtstool save

[root@lironrhel7rhos5 ~(keystone_admin)]# openstack-service restart cinder

[root@lironrhel7rhos5 ~(keystone_admin)]# nova list
+--------------------------------------+------+--------+------------+-------------+---------------------+
| ID                                   | Name | Status | Task State | Power State | Networks            |
+--------------------------------------+------+--------+------------+-------------+---------------------+
| a187a8d2-a130-4aa7-a1f9-0ed05d204bab | vm1  | ACTIVE | -          | Running     | public=172.24.4.227 |
+--------------------------------------+------+--------+------------+-------------+---------------------+

[root@lironrhel7rhos5 ~(keystone_admin)]# cinder list
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  | Status | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+
| 2359a240-4bd0-4b46-a059-e06f464036de | in-use |     vol2     |  1   |     None    |  false   | a187a8d2-a130-4aa7-a1f9-0ed05d204bab |
| 507abe1d-aec7-4306-861e-55a5b30baf88 | in-use |     vol1     |  1   |     None    |  false   | a187a8d2-a130-4aa7-a1f9-0ed05d204bab |
+--------------------------------------+--------+--------------+------+-------------+----------+--------------------------------------+

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1192