Created attachment 1837546 [details]
tcmu-runner.log

Description of problem:[RBD] ISCSI - Login failed and unable to discover luns after setting mutual chap auth in gwcli whereas CHAP auth works fine. We see issue only with mutual chap as we do not see login successful.


Version-Release number of selected component (if applicable):
[ceph: root@magna031 /]# ceph version
ceph version 16.2.0-141.el8cp (95ec5218d662f83202bc5841c93e9b220a303080) pacific (stable)
[ceph: root@magna031 /]# 


How reproducible:


Steps to Reproduce:
1.Deploy ceph cluster with 4 ISCSI gw collocated with OSD nodes
2. Login to gateway node and configure target, create client and add disks and set mutal chap auth using gwcli 
3. SSh to Linux initiator node and Provide a CHAP user name and password by updating the /etc/iscsi/iscsid.conf file accordingly as below
4. Discover the Gateways with " iscsiadm -m discovery -t st -p 10.8.128.6"
5. Perform Login to target using " iscsiadm -m node -l" and observe the behaviour

Actual results: Fails to login to target after setting the mutual chap



Expected results: Login should be successful 


Additional info: magna031- Linux initiator


gateway node details:
depressa001
depressa002
depressa003
magna006

Bootstrap node magna031

Output:
[root@magna031 ubuntu]# iscsiadm -m node -l
Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:ceph-igw, portal: 10.1.172.201,3260]
Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:ceph-igw, portal: 10.8.128.6,3260]
Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:ceph-igw, portal: 10.1.172.202,3260]
Logging in to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:ceph-igw, portal: 10.1.172.203,3260]
iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:ceph-igw, portal: 10.1.172.201,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:ceph-igw, portal: 10.8.128.6,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:ceph-igw, portal: 10.1.172.202,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not login to [iface: default, target: iqn.2003-01.com.redhat.iscsi-gw:ceph-igw, portal: 10.1.172.203,3260].
iscsiadm: initiator reported error (19 - encountered non-retryable iSCSI login failure)
iscsiadm: Could not log into all portals
GWCLI output:
/iscsi-target...at:rh8-client> ls
o- iqn.1994-05.com.redhat:rh8-client  [Auth: CHAP_MUTUAL, Disks: 5(250G)]
  o- lun 0  [iscsi/iscsi1(50G), Owner: depressa001]
  o- lun 1  [iscsi/iscsi2(50G), Owner: depressa002]
  o- lun 2  [iscsi/iscsi3(50G), Owner: depressa003]
  o- lun 3 ... [iscsi/iscsi4(50G), Owner: magna006]
  o- lun 4  [iscsi/iscsi5(50G), Owner: depressa001]
/iscsi-target...at:rh8-client> ls
o- iqn.1994-05.com.redhat:rh8-client ........................................................... [Auth: CHAP_MUTUAL, Disks: 5(250G)]
  o- lun 0 ................................................................................. [iscsi/iscsi1(50G), Owner: depressa001]
  o- lun 1 ................................................................................. [iscsi/iscsi2(50G), Owner: depressa002]
  o- lun 2 ................................................................................. [iscsi/iscsi3(50G), Owner: depressa003]
  o- lun 3 .................................................................................... [iscsi/iscsi4(50G), Owner: magna006]
  o- lun 4 ................................................................................. [iscsi/iscsi5(50G), Owner: depressa001]
/iscsi-target...at:rh8-client> cd
disk GET status for iscsi/iscsi1
Chap settings under /etc/iscsi.conf
# *************
# CHAP Settings
# *************

# To enable CHAP authentication set node.session.auth.authmethod
# to CHAP. The default is None.
node.session.auth.authmethod = CHAP_MUTUAL

# To configure which CHAP algorithms to enable set
# node.session.auth.chap_algs to a comma seperated list.
# The algorithms should be listen with most prefered first.
# Valid values are MD5, SHA1, SHA256, and SHA3-256.
# The default is MD5.
#node.session.auth.chap_algs = SHA3-256,SHA256,SHA1,MD5

# To set a CHAP username and password for initiator
# authentication by the target(s), uncomment the following lines:
node.session.auth.username = iscsiuser1
node.session.auth.password = temp12345678

# To set a CHAP username and password for target(s)
# authentication by the initiator, uncomment the following lines:
node.session.auth.username_in = iscsiuser1
node.session.auth.password_in = temp12345678


NOTE: I tried with option CHAP/CHAP_MUTUAL both in conf to check if login works


GWCLI output:


/> ls
o- / ......................................................................................................................... [...]
  o- cluster ......................................................................................................... [Clusters: 1]
  | o- ceph ............................................................................................................ [HEALTH_OK]
  |   o- pools ......................................................................................................... [Pools: 13]
  |   | o- .rgw.root ............................................................. [(x3), Commit: 0.00Y/2160547584K (0%), Used: 48K]
  |   | o- cephfs.backup.data ............................................ [(x3), Commit: 0.00Y/2160547584K (0%), Used: 1555529424K]
  |   | o- cephfs.backup.meta ............................................ [(x3), Commit: 0.00Y/2160547584K (0%), Used: 8318211209b]
  |   | o- default.rgw.buckets.data ............................................. [(x3), Commit: 0.00Y/2160547584K (0%), Used: 120K]
  |   | o- default.rgw.buckets.index ....................................... [(x3), Commit: 0.00Y/2160547584K (0%), Used: 49056605b]
  |   | o- default.rgw.control ................................................. [(x3), Commit: 0.00Y/2160547584K (0%), Used: 0.00Y]
  |   | o- default.rgw.log ...................................................... [(x3), Commit: 0.00Y/2160547584K (0%), Used: 408K]
  |   | o- default.rgw.meta ................................................. [(x3), Commit: 0.00Y/2160547584K (0%), Used: 3579213b]
  |   | o- device_health_metrics .......................................... [(x3), Commit: 0.00Y/2160547584K (0%), Used: 106562952b]
  |   | o- ec-pool ......................................................... [(9+2), Commit: 0.00Y/4713922048K (0%), Used: 1441836K]
  |   | o- iscsi ............................................................ [(x3), Commit: 250G/2160547584K (12%), Used: 3111876b]
  |   | o- rbd .......................................................... [(x3), Commit: 0.00Y/2160547584K (0%), Used: 68095321209b]
  |   | o- rep-pool .......................................................... [(x3), Commit: 0.00Y/2160547584K (0%), Used: 130872K]
  |   o- topology ............................................................................................... [OSDs: 27,MONs: 3]
  o- disks ........................................................................................................ [250G, Disks: 5]
  | o- iscsi ........................................................................................................ [iscsi (250G)]
  |   o- iscsi1 ....................................................................................... [iscsi/iscsi1 (Online, 50G)]
  |   o- iscsi2 ....................................................................................... [iscsi/iscsi2 (Online, 50G)]
  |   o- iscsi3 ....................................................................................... [iscsi/iscsi3 (Online, 50G)]
  |   o- iscsi4 ....................................................................................... [iscsi/iscsi4 (Online, 50G)]
  |   o- iscsi5 ....................................................................................... [iscsi/iscsi5 (Online, 50G)]
  o- iscsi-targets ............................................................................... [DiscoveryAuth: None, Targets: 1]
    o- iqn.2003-01.com.redhat.iscsi-gw:ceph-igw .......................................................... [Auth: None, Gateways: 4]
      o- disks .......................................................................................................... [Disks: 5]
      | o- iscsi/iscsi1 ............................................................................... [Owner: depressa001, Lun: 0]
      | o- iscsi/iscsi2 ............................................................................... [Owner: depressa002, Lun: 1]
      | o- iscsi/iscsi3 ............................................................................... [Owner: depressa003, Lun: 2]
      | o- iscsi/iscsi4 .................................................................................. [Owner: magna006, Lun: 3]
      | o- iscsi/iscsi5 ............................................................................... [Owner: depressa001, Lun: 4]
      o- gateways ............................................................................................ [Up: 4/4, Portals: 4]
      | o- depressa001 ......................................................................................... [10.1.172.201 (UP)]
      | o- depressa002 ......................................................................................... [10.1.172.202 (UP)]
      | o- depressa003 ......................................................................................... [10.1.172.203 (UP)]
      | o- magna006 .............................................................................................. [10.8.128.6 (UP)]
      o- host-groups .................................................................................................. [Groups : 0]
      o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 1]
        o- iqn.1994-05.com.redhat:rh8-client ................................................... [Auth: CHAP_MUTUAL, Disks: 5(250G)]
          o- lun 0 ......................................................................... [iscsi/iscsi1(50G), Owner: depressa001]
          o- lun 1 ......................................................................... [iscsi/iscsi2(50G), Owner: depressa002]
          o- lun 2 ......................................................................... [iscsi/iscsi3(50G), Owner: depressa003]
          o- lun 3 ............................................................................ [iscsi/iscsi4(50G), Owner: magna006]
          o- lun 4 ......................................................................... [iscsi/iscsi5(50G), Owner: depressa001]
/> 


*******************************************