Bug 1923975

It turned out we are hitting an issue with iscsiadm command in RHEL8.2.
Since current os-brick executes volume connection (iscsiadm -m session and isciadm -m node ... --login) in thread
we are likely to hit this bug.

Since in controller nodes we don't keep iscsi connections but create ones when creating a volume from an image
we have more possibility to hit this, but we can hit the same in compute node when attaching a first volume to
that node.

Pablo, 

Is there a way I could verify this?
I see iSCSI and multipath mentioned, 
suspect this also depends on specific driver implementation  
"it executes "iscsiadm -m session" first to obtain all existing sessions"

Do you happen to know if I should use 3par iSCSI or netapp iSCSI for this case?

Reproduce steps would be what, just successfully attach a volume to an instance?

Tzach, it was reported on different backends iteration using iscsi like
 - NetApp Storage backend
 -  iscsi storage( Dell-Emc XtremIO)

Requirement is iscsi-initiator-utils installed at the nova-compute container and cinder-volume as
    https://bugzilla.redhat.com/show_bug.cgi?id=1924768 (for RHEL8.4)
    and
    https://bugzilla.redhat.com/show_bug.cgi?id=1940666 (for RHEL8.2)

You can use the netapp and the 3par if you times allow you.

Basically the test need  iscsi storage and multipath enabled, so when attaching a volume cinder run iscsi commands and multipath commands to initialize connection and multipath device on multiple iscsi devices.

Run a attach volume command to specific instance and observe 
 When cinder connects the volume it first runs
   Run manually  `iscsiadm -m session -d ` is run through cinder. 
  to get a list of all sessions and then run   
   iscsiadm -m node -T <target> -p <portal> --login
  if the target/portal is not yet logged in.

If multipath is enabled, it execute these process in threads and run iscsi commands concurrently
for multiple iscsi devices under an multipath device it is attaching.

Old os-brick code  doesn't properly catch some possible exceptions during connectiing to iSCSI portals, like failures in "iscsiadm -m session". about the failure in "iscsiadm -m session", and if the command fails the volume attachment never completes.

If volume is attached succesfully, this should be enough for passing the test.

Thanks for double checking on this.
Regards.

Verified on
python3-os-brick-2.10.7-2.20210528134947.el8ost.2.noarch

On a multipath deployment using netapp iSCSI as Cinder backend, 
I ran the below scripts in parallel terminals:

[root@controller-2 ~]# cat iscsiwatch.sh                                                                                                                                                                                                     
# cat check_session                                                                                                                                                                                                                          
while true; do                                                                                                                                                                                                                               
date ; iscsiadm -m session                                                                                                                                                                                                                   
echo "==================================================================================="                                                                                                                                                   
done 

[root@controller-2 ~]# cat lin.sh 
# cat login loop                                                                                                                                                                                                                             
while true; do                                                                                                                                                                                                                               
date ; iscsiadm -m node -T iqn.1992-08.com.netapp:sn.83806661cc2f11eba182d039ea28c8f6:vs.19 -p 10.XXXXXXXX:3260 --login     (xxx ->internal IP)                                                                                                                
echo "==================================================================================="                                                                                                                                                   
done

[root@controller-2 ~]# cat lout.sh                                                                                                                                                                                                
# cat logout loop
while true; do
date ; iscsiadm -m node -T iqn.1992-08.com.netapp:sn.83806661cc2f11eba182d039ea28c8f6:vs.19 -p 10.XXXXXXXXX:3260 --logout
echo "==================================================================================="
done


To try to simulate a "iscsiadm -m session" failure, I'd even downgraded to:
iscsi-initiator-utils-6.2.0.878-5
Yet still I never managed to hit it. 

Several cycles of volume attach/detach to an instance, completed without any issues.
This was done while the loops were executed in parallel on the compute node. 

I then tested several cycles of volume create from a rhel.qcow2 image,
while running loops on the controller were c-vol was running, 
Again no issues were hit, all volumes were available. 

Looks good to verify.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform (RHOSP) 16.2 enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:3483