Bug 1532864 - hosted-engine --deploy does not work with iSCSI target
Summary: hosted-engine --deploy does not work with iSCSI target
Keywords:
Status: CLOSED DUPLICATE of bug 1353713
Alias: None
Product: ovirt-hosted-engine-setup
Classification: oVirt
Component: Plugins.Block
Version: 2.2.1
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
: ---
Assignee: Simone Tiraboschi
QA Contact: meital avital
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-09 22:37 UTC by Ab
Modified: 2018-06-08 17:51 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-01-11 22:57:16 UTC
oVirt Team: Integration
Embargoed:


Attachments (Terms of Use)
vdsm.log.5 log file (2.00 MB, text/plain)
2018-01-10 09:10 UTC, Ab
no flags Details
requested vdsm log file (2.00 MB, text/plain)
2018-01-10 09:11 UTC, Ab
no flags Details

Description Ab 2018-01-09 22:37:55 UTC
Description of problem:
======================
I am trying to deploy self-hosted engine using iSCSI target but it does not work


Version-Release number of selected component:
=============================================
Self-Hosted Engine version: ovirt-hosted-engine-setup-2.2.3-1.el7.centos.noarch
OS version: CentOS 7.4 (KVM VM with nested virtualization)
OS is member of FreeIPA 4.4 domain using ipa-client

How reproducible:
================
Steps to Reproduce:
1. Deploy self-hosted engine to using iSCSI produced the following authentication error:
[root@rhevh1 ~]# hosted-engine --deploy
[.. OUTPUT OMITTED..]
          --== STORAGE CONFIGURATION ==--
         
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]: iscsi
          Please specify the iSCSI portal IP address: 192.168.160.132
          Please specify the iSCSI portal port [3260]: 
          Please specify the iSCSI portal user: ab
          Please specify the iSCSI portal password: 
[ ERROR ] Command ISCSIConnection.discoverSendTargets with args {'host': '192.168.160.132', 'password': 'pass123', 'user': 'ab', 'port': '3260'} failed:
         (code=475, message=Failed discovery of iSCSI targets: u"portal=192.168.160.132:3260, err=(24, [], ['iscsiadm: Login failed to authenticate with target ', 'iscsiadm: discovery login to 192.168.160.132 rejected: initiator failed authorization', 'iscsiadm: Could not perform SendTargets discovery: iSCSI login failed due to authorization failure'])")
          Please specify the iSCSI portal IP address: 

2. I tried to discover iSCSI target manually:
[root@rhevh1 ~]# iscsiadm -m discovery -t st -p 192.168.160.132
192.168.160.132:3260,1 iqn.2018-01.lab.ab:rheviscsi
192.168.160.132:3260,1 iqn.2018-01.lab.ab:iscsitarget

3. Again, trying step #1 resulted the same error:
[root@rhevh1 ~]# hosted-engine --deploy
[.. OUTPUT OMITTED..]
          --== STORAGE CONFIGURATION ==--
         
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]: iscsi
          Please specify the iSCSI portal IP address: 192.168.160.132
          Please specify the iSCSI portal port [3260]: 
          Please specify the iSCSI portal user: ab
          Please specify the iSCSI portal password: 
[ ERROR ] Command ISCSIConnection.discoverSendTargets with args {'host': '192.168.160.132', 'password': 'pass123', 'user': 'ab', 'port': '3260'} failed:
         (code=475, message=Failed discovery of iSCSI targets: u"portal=192.168.160.132:3260, err=(24, [], ['iscsiadm: Login failed to authenticate with target ', 'iscsiadm: discovery login to 192.168.160.132 rejected: initiator failed authorization', 'iscsiadm: Could not perform SendTargets discovery: iSCSI login failed due to authorization failure'])")
          Please specify the iSCSI portal IP address: 

4. I tried to attach the iSCSI disks manually and it was working fine:
[root@rhevh1 ~]# iscsiadm -m discovery -t st -p 192.168.160.132
192.168.160.132:3260,1 iqn.2018-01.lab.ab:rheviscsi
192.168.160.132:3260,1 iqn.2018-01.lab.ab:iscsitarget

[root@rhevh1 ~]# iscsiadm -m node -T iqn.2018-01.lab.ab:rheviscsi -p 192.168.160.132 -l
Logging in to [iface: default, target: iqn.2018-01.lab.ab:rheviscsi, portal: 192.168.160.132,3260] (multiple)
Login to [iface: default, target: iqn.2018-01.lab.ab:rheviscsi, portal: 192.168.160.132,3260] successful.

[root@rhevh1 ~]# lsblk --scsi
NAME HCTL       TYPE VENDOR   MODEL             REV TRAN
sda  2:0:0:0    disk LIO-ORG  self_hosted_lv   4.0  iscsi
sdb  2:0:0:1    disk LIO-ORG  vms_data         4.0  iscsi
sr0  0:0:0:0    rom  QEMU     QEMU DVD-ROM     1.5. ata


5. I tried step #1 again, but it resulted the same error message:
[root@rhevh1 ~]# hosted-engine --deploy
[.. OUTPUT OMITTED..]
          --== STORAGE CONFIGURATION ==--
         
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]: iscsi
          Please specify the iSCSI portal IP address: 192.168.160.132 
          Please specify the iSCSI portal port [3260]: 
          Please specify the iSCSI portal user: ab
          Please specify the iSCSI portal password: 
[ ERROR ] Command ISCSIConnection.discoverSendTargets with args {'host': '192.168.160.132', 'password': 'pass123', 'user': 'ab', 'port': '3260'} failed:
         (code=475, message=Failed discovery of iSCSI targets: u"portal=192.168.160.132:3260, err=(24, [], ['iscsiadm: Login failed to authenticate with target ', 'iscsiadm: discovery login to 192.168.160.132 rejected: initiator failed authorization', 'iscsiadm: Could not perform SendTargets discovery: iSCSI login failed due to authorization failure'])")
          Please specify the iSCSI portal IP address: 


6. I withdrew OS from the FreeIPA domain, then tried again, and I got the same error. 


7. iSCSI targetcli output:
[root@iscsi ~]# targetcli ls
o- / ......................................................................................................................... [...]
  o- backstores .............................................................................................................. [...]
  | o- block .................................................................................................. [Storage Objects: 4]
  | | o- export ......................................................... [/dev/nfs_server_vg/export (50.0GiB) write-thru activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- isos ............................................................. [/dev/nfs_server_vg/isos (50.0GiB) write-thru activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- self_hosted_lv ....................................... [/dev/self_hosted_vg/self_hosted_lv (100.0GiB) write-thru activated]
  | | | o- alua ................................................................................................... [ALUA Groups: 1]
  | | |   o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | | o- vms_data ...................................................... [/dev/vms_data_vg/vms_data (100.0GiB) write-thru activated]
  | |   o- alua ................................................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................................................ [Targets: 2]
  | o- iqn.2018-01.lab.ab:iscsitarget .................................................................................... [TPGs: 1]
  | | o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  | |   o- acls .......................................................................................................... [ACLs: 1]
  | |   | o- iqn.2018-01.lab.ab:nfsserver ......................................................................... [Mapped LUNs: 2]
  | |   |   o- mapped_lun0 .................................................................................. [lun0 block/isos (rw)]
  | |   |   o- mapped_lun1 ................................................................................ [lun1 block/export (rw)]
  | |   o- luns .......................................................................................................... [LUNs: 2]
  | |   | o- lun0 ........................................................ [block/isos (/dev/nfs_server_vg/isos) (default_tg_pt_gp)]
  | |   | o- lun1 .................................................... [block/export (/dev/nfs_server_vg/export) (default_tg_pt_gp)]
  | |   o- portals .................................................................................................... [Portals: 1]
  | |     o- 0.0.0.0:3260 ..................................................................................................... [OK]
  | o- iqn.2018-01.lab.ab:rheviscsi ...................................................................................... [TPGs: 1]
  |   o- tpg1 ............................................................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................................................... [ACLs: 1]
  |     | o- iqn.2018-01.lab.ab:rhevh ............................................................................. [Mapped LUNs: 2]
  |     |   o- mapped_lun0 ........................................................................ [lun0 block/self_hosted_lv (rw)]
  |     |   o- mapped_lun1 .............................................................................. [lun1 block/vms_data (rw)]
  |     o- luns .......................................................................................................... [LUNs: 2]
  |     | o- lun0 ................................... [block/self_hosted_lv (/dev/self_hosted_vg/self_hosted_lv) (default_tg_pt_gp)]
  |     | o- lun1 .................................................. [block/vms_data (/dev/vms_data_vg/vms_data) (default_tg_pt_gp)]
  |     o- portals .................................................................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ..................................................................................................... [OK]
  o- loopback ......................................................................................................... [Targets: 0]


8. iSCSI target /var/log/messages log files:
Jan  9 23:19:36 iscsi kernel: CHAP user or password not set for Initiator ACL
Jan  9 23:19:36 iscsi kernel: Security negotiation failed.
Jan  9 23:19:36 iscsi kernel: iSCSI Login negotiation failed.
Jan  9 23:27:19 iscsi kernel: CHAP user or password not set for Initiator ACL
Jan  9 23:27:19 iscsi kernel: Security negotiation failed.
Jan  9 23:27:19 iscsi kernel: iSCSI Login negotiation failed.


9. Checking target ACL:
[root@iscsi ~]# targetcli /iscsi/iqn.2018-01.lab.ab:rheviscsi/tpg1/acls/iqn.2018-01.lab.ab:rhevh get auth userid
userid=ab 
[root@iscsi ~]# targetcli /iscsi/iqn.2018-01.lab.ab:rheviscsi/tpg1/acls/iqn.2018-01.lab.ab:rhevh get auth password
password=pass123

10. Checking initiator: 
[root@rhevh1 ~]# cat /etc/iscsi/initiatorname.iscsi 
InitiatorName=iqn.2018-01.lab.ab:rhevh

[root@rhevh1 ~]# grep 'node.session.auth.authmethod =' /etc/iscsi/iscsid.conf 
node.session.auth.authmethod = CHAP

[root@rhevh1 ~]# grep 'node.session.auth.username =' /etc/iscsi/iscsid.conf 
node.session.auth.username = ab

[root@rhevh1 ~]# grep 'node.session.auth.password =' /etc/iscsi/iscsid.conf 
node.session.auth.password = pass123

Comment 1 Simone Tiraboschi 2018-01-10 08:13:52 UTC
Could you please attach vdsm logs?

Comment 2 Ab 2018-01-10 08:48:51 UTC
Unfortunately, I had to go with NFS share way to install the self-hosted engine.
and it worked fine for NFS share.

Lucky thing that vdsm logs records still there.

Some records from /var/log/vdsm/vds.log.5:

2018-01-09 22:05:12,838+0100 INFO  (jsonrpc/2) [vdsm.api] START discoverSendTargets(con={'ipv6_enabled': False, 'connection': u'192.168.160.132', 'password': '********', 'port': u'3260', 'user': u'ab'}, options=None) from=::1,40584, task_id=fcbf29d3-a72e-43b2-a22b-70f3ec922b81 (api:46)
2018-01-09 22:05:13,167+0100 ERROR (jsonrpc/2) [storage.HSM] Discovery failed (hsm:2956)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2954, in discoverSendTargets
    targets = iscsi.discoverSendTargets(iface, portal, cred)
  File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsi.py", line 274, in discoverSendTargets
    str(portal))
  File "/usr/lib/python2.7/site-packages/vdsm/storage/iscsiadm.py", line 256, in discoverydb_discover
    raise IscsiAuthenticationError(rc, out, err)
IscsiAuthenticationError: (24, [], ['iscsiadm: Login failed to authenticate with target ', 'iscsiadm: discovery login to 192.168.160.132 rejected: initiator failed authorization', 'iscsiadm: Could not perform SendTargets discovery: iSCSI login failed due to authorization failure'])
2018-01-09 22:05:13,169+0100 INFO  (jsonrpc/2) [vdsm.api] FINISH discoverSendTargets error=Failed discovery of iSCSI targets: u"portal=192.168.160.132:3260, err=(24, [], ['iscsiadm: Login failed to authenticate with target ', 'iscsiadm: discovery login to 192.168.160.132 rejected: initiator failed authorization', 'iscsiadm: Could not perform SendTargets discovery: iSCSI login failed due to authorization failure'])" from=::1,40584, task_id=fcbf29d3-a72e-43b2-a22b-70f3ec922b81 (api:50)
2018-01-09 22:05:13,169+0100 ERROR (jsonrpc/2) [storage.TaskManager.Task] (Task='fcbf29d3-a72e-43b2-a22b-70f3ec922b81') Unexpected error (task:875)
[..OUTPUT OMITTED..]
2018-01-09 23:19:36,350+0100 INFO  (jsonrpc/6) [vdsm.api] START discoverSendTargets(con={'ipv6_enabled': False, 'connection': u'192.168.160.132', 'password': '********', 'port': u'3260', 'user': u'ab'}, options=None) from=::1,39164, task_id=7c58a714-1557-4a1f-8749-2a31ead93ff9 (api:46)
2018-01-09 23:19:36,640+0100 ERROR (jsonrpc/6) [storage.HSM] Discovery failed (hsm:2956)


I am going to attach vdsm.log.5 for your review and I will keep the current implementation for now. :)


Best regards,
Ab

Comment 3 Ab 2018-01-10 09:10:27 UTC
Created attachment 1379395 [details]
vdsm.log.5 log file

requested vdsm.log file during the problem time.

Comment 4 Ab 2018-01-10 09:11:22 UTC
Created attachment 1379396 [details]
requested vdsm log file

Comment 5 Ab 2018-01-10 09:14:08 UTC
As it is a big file, Here is a link for the whole file:
https://drive.google.com/file/d/1T_Pwcok8iB3-r4kDbLUq-Y6bCJtOjoVt/view?usp=sharing

Best regards,
Ab

Comment 6 Simone Tiraboschi 2018-01-10 09:23:21 UTC
Thanks,
I'm supposing that this is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1474209

Current iSCSI support in hosted-engine-setup doesn't discriminate between the CHAP password user for the discovery and the CHAP password user to login to the portal so, if you set a CHAP password, you should now use the same value for discovery and login.

Was that your case?

Comment 7 Ab 2018-01-10 10:43:14 UTC
Sorry, but I am a bit confused. 

Do you mean that 'hosted-engine --deploy' was able to discover the iSCSI target but it was not able to login?

If so, given that:
1) I set ACL 'userid' and 'password' in iSCSI target side.
2) I set 'node.session.auth.authmethod', 'node.session.auth.username' and 'node.session.auth.password' values in iSCSI initiator sid

What should be the additional configuration that allow 'hosted-engine --deploy' to discover and login to the iSCSI target?


[root@iscsi ~]# targetcli /iscsi/iqn.2018-01.lab.ab:rheviscsi/tpg1/acls/iqn.2018-01.lab.ab:rhevh get auth userid
userid=ab 
[root@iscsi ~]# targetcli /iscsi/iqn.2018-01.lab.ab:rheviscsi/tpg1/acls/iqn.2018-01.lab.ab:rhevh get auth password
password=pass123

10. Checking initiator: 
[root@rhevh1 ~]# cat /etc/iscsi/initiatorname.iscsi 
InitiatorName=iqn.2018-01.lab.ab:rhevh

[root@rhevh1 ~]# grep 'node.session.auth.authmethod =' /etc/iscsi/iscsid.conf 
node.session.auth.authmethod = CHAP

[root@rhevh1 ~]# grep 'node.session.auth.username =' /etc/iscsi/iscsid.conf 
node.session.auth.username = ab

[root@rhevh1 ~]# grep 'node.session.auth.password =' /etc/iscsi/iscsid.conf 
node.session.auth.password = pass123

Best regards,
Ab

Comment 8 Yaniv Kaul 2018-01-10 11:03:31 UTC
(In reply to Ab from comment #7)
> Sorry, but I am a bit confused. 
> 
> Do you mean that 'hosted-engine --deploy' was able to discover the iSCSI
> target but it was not able to login?
> 
> If so, given that:
> 1) I set ACL 'userid' and 'password' in iSCSI target side.
> 2) I set 'node.session.auth.authmethod', 'node.session.auth.username' and
> 'node.session.auth.password' values in iSCSI initiator sid
> 
> What should be the additional configuration that allow 'hosted-engine
> --deploy' to discover and login to the iSCSI target?
> 
> 
> [root@iscsi ~]# targetcli
> /iscsi/iqn.2018-01.lab.ab:rheviscsi/tpg1/acls/iqn.2018-01.lab.ab:rhevh get
> auth userid
> userid=ab 
> [root@iscsi ~]# targetcli
> /iscsi/iqn.2018-01.lab.ab:rheviscsi/tpg1/acls/iqn.2018-01.lab.ab:rhevh get
> auth password
> password=pass123
> 
> 10. Checking initiator: 
> [root@rhevh1 ~]# cat /etc/iscsi/initiatorname.iscsi 
> InitiatorName=iqn.2018-01.lab.ab:rhevh
> 
> [root@rhevh1 ~]# grep 'node.session.auth.authmethod ='
> /etc/iscsi/iscsid.conf 
> node.session.auth.authmethod = CHAP
> 
> [root@rhevh1 ~]# grep 'node.session.auth.username =' /etc/iscsi/iscsid.conf 
> node.session.auth.username = ab
> 
> [root@rhevh1 ~]# grep 'node.session.auth.password =' /etc/iscsi/iscsid.conf 
> node.session.auth.password = pass123

Was the service restarted?

> 
> Best regards,
> Ab

Comment 9 Ab 2018-01-10 11:05:46 UTC
Yup.

Even, I tried to reboot the reboot the two machines to make sure.

Best regards,
Ab

Comment 10 Simone Tiraboschi 2018-01-11 13:15:48 UTC
(In reply to Ab from comment #7)
> Sorry, but I am a bit confused. 
> 
> Do you mean that 'hosted-engine --deploy' was able to discover the iSCSI
> target but it was not able to login?

No, exactly the opposite:
your issue is 'iscsiadm: discovery login to 192.168.160.132 rejected: initiator failed authorization'.

So it failed the discovery due to an authentication issue (but it says authorization in the log message to it's worth to double check also that).
We are using (it's a bug as for BZ#1474209) the same CHAP password for iSCSI discovery and for the login to the ISCSI portal.

> If so, given that:
> 1) I set ACL 'userid' and 'password' in iSCSI target side.

Until BZ#1474209 get fixed, you also have to enable the discovery authentication on targetd side and set the same userid and password you set for the login.
Something like:
  /iscsi> set discovery_auth enable=1 userid=mytargetuid password=mytargetsecret
Please take care that it will affect all of your initiators.

> 2) I set 'node.session.auth.authmethod', 'node.session.auth.username' and
> 'node.session.auth.password' values in iSCSI initiator sid

Not relevant at all, we are just using the password you supply.

Comment 11 Simone Tiraboschi 2018-01-11 13:24:54 UTC
(In reply to Ab from comment #7)
> 10. Checking initiator: 
> [root@rhevh1 ~]# cat /etc/iscsi/initiatorname.iscsi 
> InitiatorName=iqn.2018-01.lab.ab:rhevh
> 
> [root@rhevh1 ~]# grep 'node.session.auth.authmethod ='
> /etc/iscsi/iscsid.conf 
> node.session.auth.authmethod = CHAP
> 
> [root@rhevh1 ~]# grep 'node.session.auth.username =' /etc/iscsi/iscsid.conf 
> node.session.auth.username = ab
> 
> [root@rhevh1 ~]# grep 'node.session.auth.password =' /etc/iscsi/iscsid.conf 
> node.session.auth.password = pass123

Just to be sure that the issue is really there,
could you please set also 

discovery.sendtargets.auth.authmethod = CHAP
discovery.sendtargets.auth.username = ab
discovery.sendtargets.auth.password = pass123

And retry the discovery from CLI with
 iscsiadm -m discovery -t st -p 192.168.160.132

Comment 12 Ab 2018-01-11 16:00:57 UTC
(In reply to Simone Tiraboschi from comment #11)
> Just to be sure that the issue is really there,
> could you please set also 
> 
> discovery.sendtargets.auth.authmethod = CHAP
> discovery.sendtargets.auth.username = ab
> discovery.sendtargets.auth.password = pass123
> 
> And retry the discovery from CLI with
>  iscsiadm -m discovery -t st -p 192.168.160.132

I am replying to your last update first. 

You are right!
When I enabled iSCSI 'discovery.sendtargets.auth' parameters beside current configured 'node.session.auth' parameters, it produced the same error message I got during 'hosted-engine --deploy' iSCSI part:

[root@rhevh1 ~]# grep 'node.session.auth.authmethod =' /etc/iscsi/iscsid.conf 
node.session.auth.authmethod = CHAP
[root@rhevh1 ~]# grep 'node.session.auth.username =' /etc/iscsi/iscsid.conf 
node.session.auth.username = ab
[root@rhevh1 ~]# grep 'node.session.auth.password =' /etc/iscsi/iscsid.conf 
node.session.auth.password = pass123


[root@rhevh1 ~]# grep 'discovery.sendtargets.auth.username =' /etc/iscsi/iscsid.conf 
discovery.sendtargets.auth.username = ab
[root@rhevh1 ~]# grep 'discovery.sendtargets.auth.password =' /etc/iscsi/iscsid.conf 
discovery.sendtargets.auth.password = pass123
[root@rhevh1 ~]# grep 'discovery.sendtargets.auth.authmethod =' /etc/iscsi/iscsid.conf 
discovery.sendtargets.auth.authmethod = CHAP

[root@rhevh1 ~]# systemctl restart iscsi

[root@rhevh1 ~]# iscsiadm -m discovery -t st -p 192.168.160.132
iscsiadm: Login failed to authenticate with target 
iscsiadm: discovery login to 192.168.160.132 rejected: initiator failed authorization
iscsiadm: Could not perform SendTargets discovery: iSCSI login failed due to authorization failure


Now, I going to try your update in comment #10

Best regards,
Ab

Comment 13 Ab 2018-01-11 16:50:13 UTC
(In reply to Simone Tiraboschi from comment #10)
> (In reply to Ab from comment #7)
> > Sorry, but I am a bit confused. 
> > 
> > Do you mean that 'hosted-engine --deploy' was able to discover the iSCSI
> > target but it was not able to login?
> 
> No, exactly the opposite:
> your issue is 'iscsiadm: discovery login to 192.168.160.132 rejected:
> initiator failed authorization'.
> 
> So it failed the discovery due to an authentication issue (but it says
> authorization in the log message to it's worth to double check also that).
> We are using (it's a bug as for BZ#1474209) the same CHAP password for iSCSI
> discovery and for the login to the ISCSI portal.
> 
> > If so, given that:
> > 1) I set ACL 'userid' and 'password' in iSCSI target side.
> 
> Until BZ#1474209 get fixed, you also have to enable the discovery
> authentication on targetd side and set the same userid and password you set
> for the login.
> Something like:
>   /iscsi> set discovery_auth enable=1 userid=mytargetuid
> password=mytargetsecret
> Please take care that it will affect all of your initiators.
> 
> > 2) I set 'node.session.auth.authmethod', 'node.session.auth.username' and
> > 'node.session.auth.password' values in iSCSI initiator sid
> 
> Not relevant at all, we are just using the password you supply.


1) I disabled 'discovery.sendtargets.auth' partameters again:
[root@rhevh1 ~]# grep 'discovery.sendtargets.auth.authmethod =' /etc/iscsi/iscsid.conf 
#discovery.sendtargets.auth.authmethod = CHAP

[root@rhevh1 ~]# grep 'discovery.sendtargets.auth.username =' /etc/iscsi/iscsid.conf 
#discovery.sendtargets.auth.username = ab

[root@rhevh1 ~]# grep 'discovery.sendtargets.auth.password =' /etc/iscsi/iscsid.conf 
#discovery.sendtargets.auth.password = Redhat13

2) Enabling the "discovery authentication" on targetd side:
/iscsi> set discovery_auth enable=1 userid=ab password=pass123
Parameter enable is now 'True'.
Parameter password is now 'pass123'.
Parameter userid is now 'ab'.


3) Restarting 'iSCSI-target' and 'iSCSI-initiator' daemons:
[root@iscsi ~]# systemctl restart target
[root@rhevh1 ~]# systemctl restart iscsi

4) Deploying the self-hosted engine:
[root@rhevh1 ~]# hosted-engine --deploy
[.. OUTPUT IS OMITTED ..]
          --== STORAGE CONFIGURATION ==--
         
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]: iscsi
          Please specify the iSCSI portal IP address: 192.168.160.132
          Please specify the iSCSI portal port [3260]: 
          Please specify the iSCSI portal user: ab
          Please specify the iSCSI portal password: 
          The following targets have been found:
          	[1]	iqn.2018-01.lab.ab:iscsitarget
          		TPGT: 1, portals:
          			192.168.160.132:3260
         
          Please select a target (1) [1]: 
[ INFO  ] Connecting to the storage server
[ INFO  ] Connecting to the storage server
[ ERROR ] Failed to execute stage 'Environment customization': Unable to retrieve the list of LUN(s) please check the SELinux log and settings on your iscsi target
[ INFO  ] Stage: Clean up
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20180111171822.conf'
[ INFO  ] Stage: Pre-termination
[ INFO  ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20180111171755-79px58.log



5) SELinux is disabled on my iSCSI target:
[root@iscsi ~]# sestatus 
SELinux status:                 disabled


6) Checking my iSCSI target 'iscsi' TPGs (I created a new one as the previous one was deleted, so I have one iSCSI target iqn with TWO TPGs, first for NFS server and the second is for 'rhevh1'):
/iscsi> ls
o- iscsi ............................................................................................. [1-way disc auth, Targets: 1]
  o- iqn.2018-01.lab.ab:iscsitarget ...................................................................................... [TPGs: 2]
    o- tpg1 ................................................................................................. [no-gen-acls, no-auth]
    | o- acls ............................................................................................................ [ACLs: 1]
    | | o- iqn.2018-01.lab.ab:nfsserver ........................................................................... [Mapped LUNs: 2]
    | |   o- mapped_lun0 .................................................................................... [lun0 block/isos (rw)]
    | |   o- mapped_lun1 .................................................................................. [lun1 block/export (rw)]
    | o- luns ............................................................................................................ [LUNs: 2]
    | | o- lun0 .......................................................... [block/isos (/dev/nfs_server_vg/isos) (default_tg_pt_gp)]
    | | o- lun1 ...................................................... [block/export (/dev/nfs_server_vg/export) (default_tg_pt_gp)]
    | o- portals ...................................................................................................... [Portals: 1]
    |   o- 0.0.0.0:3260 ....................................................................................................... [OK]
    o- tpg2 ................................................................................................. [no-gen-acls, no-auth]
      o- acls ............................................................................................................ [ACLs: 1]
      | o- iqn.2018-01.lab.ab:rhevh1 .............................................................................. [Mapped LUNs: 2]
      |   o- mapped_lun0 .......................................................................... [lun0 block/self_hosted_lv (rw)]
      |   o- mapped_lun1 ................................................................................ [lun1 block/vms_data (rw)]
      o- luns ............................................................................................................ [LUNs: 2]
      | o- lun0 ..................................... [block/self_hosted_lv (/dev/self_hosted_vg/self_hosted_lv) (default_tg_pt_gp)]
      | o- lun1 .................................................... [block/vms_data (/dev/vms_data_vg/vms_data) (default_tg_pt_gp)]
      o- portals ...................................................................................................... [Portals: 0]


'hosted-engine --deploy' chose TPG1 instead of TPG2 for unknown reason!


8) I  had to delete iSCSI target iqn and created a new one with only one TPG as the following:
/iscsi> ls
o- iscsi ............................................................................................. [1-way disc auth, Targets: 1]
  o- iqn.2003-01.org.linux-iscsi.iscsi.x8664:sn.c7eed344c2db ............................................................. [TPGs: 1]
    o- tpg1 ................................................................................................. [no-gen-acls, no-auth]
      o- acls ............................................................................................................ [ACLs: 1]
      | o- iqn.2018-01.lab.ab:rhevh1 .............................................................................. [Mapped LUNs: 2]
      |   o- mapped_lun0 .......................................................................... [lun0 block/self_hosted_lv (rw)]
      |   o- mapped_lun1 ................................................................................ [lun1 block/vms_data (rw)]
      o- luns ............................................................................................................ [LUNs: 2]
      | o- lun0 ..................................... [block/self_hosted_lv (/dev/self_hosted_vg/self_hosted_lv) (default_tg_pt_gp)]
      | o- lun1 .................................................... [block/vms_data (/dev/vms_data_vg/vms_data) (default_tg_pt_gp)]
      o- portals ...................................................................................................... [Portals: 1]
        o- 0.0.0.0:3260 ....................................................................................................... [OK]


9) Restarting 'iSCSI-target' and 'iSCSI-initiator' daemons:
[root@iscsi ~]# systemctl restart target
[root@rhevh1 ~]# systemctl restart iscsi


10) Deploying the self-hosted engine again:
[root@rhevh1 ~]# hosted-engine --deploy
          --== STORAGE CONFIGURATION ==--
         
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]: iscsi
          Please specify the iSCSI portal IP address: 192.168.160.132
          Please specify the iSCSI portal port [3260]: 
          Please specify the iSCSI portal user: ab
          Please specify the iSCSI portal password: 
          The following targets have been found:
          	[1]	iqn.2003-01.org.linux-iscsi.iscsi.x8664:sn.c7eed344c2db
          		TPGT: 1, portals:
          			192.168.160.132:3260
         
          Please select a target (1) [1]: 
[ INFO  ] Connecting to the storage server
          The following luns have been found on the requested target:
          	[1]	360014050f8704dd7f7547c0b17ca3783	99GiB	LIO-ORG	self_hosted_lv
          		status: free, paths: 1 active
         
          	[2]	36001405d5f44c68531348baabf59ec3c	99GiB	LIO-ORG	vms_data
          		status: free, paths: 1 active
         
          Please select the destination LUN (1, 2) [1]: 1
[ INFO  ] Connecting to the storage server



11) Self-hosted engine was deployed successfully!


Questions:
==========
1) What is the estimated time or oVirt engine future version to fix BZ#1474209? as your workaround means that I need a separate iSCSI target for the self-hosted engine because other iSCSI initiators will not be able to use iSCSI target set with your workaround. 
2) What is your fix way for BZ#1474209?
2) Why 'hosted-engine' --deploy where seeking first TPG? 



Best regards,
Ab

Comment 14 Simone Tiraboschi 2018-01-11 22:57:16 UTC
(In reply to Ab from comment #13)

> .............................................................................
> ......... [TPGs: 2]
>     o- tpg1
> .............................................................................
> .................... [no-gen-acls, no-auth]
>     | o- acls
> .............................................................................
> ............................... [ACLs: 1]
>     | | o- iqn.2018-01.lab.ab:nfsserver
> ...........................................................................
> [Mapped LUNs: 2]
>     | |   o- mapped_lun0
> .............................................................................
> ....... [lun0 block/isos (rw)]
>     | |   o- mapped_lun1
> .............................................................................
> ..... [lun1 block/export (rw)]
>     | o- luns
> .............................................................................
> ............................... [LUNs: 2]
>     | | o- lun0 ..........................................................
> [block/isos (/dev/nfs_server_vg/isos) (default_tg_pt_gp)]
>     | | o- lun1 ......................................................
> [block/export (/dev/nfs_server_vg/export) (default_tg_pt_gp)]
>     | o- portals
> .............................................................................
> ......................... [Portals: 1]
>     |   o- 0.0.0.0:3260
> .............................................................................
> .......................... [OK]
>     o- tpg2
> .............................................................................
> .................... [no-gen-acls, no-auth]
>       o- acls
> .............................................................................
> ............................... [ACLs: 1]
>       | o- iqn.2018-01.lab.ab:rhevh1
> .............................................................................
> . [Mapped LUNs: 2]
>       |   o- mapped_lun0
> ..........................................................................
> [lun0 block/self_hosted_lv (rw)]
>       |   o- mapped_lun1
> .............................................................................
> ... [lun1 block/vms_data (rw)]
>       o- luns
> .............................................................................
> ............................... [LUNs: 2]
>       | o- lun0 ..................................... [block/self_hosted_lv
> (/dev/self_hosted_vg/self_hosted_lv) (default_tg_pt_gp)]
>       | o- lun1 ....................................................
> [block/vms_data (/dev/vms_data_vg/vms_data) (default_tg_pt_gp)]
>       o- portals
> .............................................................................
> ......................... [Portals: 0]
> 
> 
> 'hosted-engine --deploy' chose TPG1 instead of TPG2 for unknown reason!

Pretty easy:
you had 1 portal under tpg1 and 0 portals under tpg2, that's why you could not select tpg2



> Questions:
> ==========
> 1) What is the estimated time or oVirt engine future version to fix
> BZ#1474209? as your workaround means that I need a separate iSCSI target for
> the self-hosted engine because other iSCSI initiators will not be able to
> use iSCSI target set with your workaround.

I hope 4.2.2. something more than one month.

> 2) What is your fix way for BZ#1474209?

Let the different user/password for discovery and portal login

> 2) Why 'hosted-engine' --deploy where seeking first TPG? 

Just because you had no portals under tpg2.
Please try again adding at least one portal there.
With 4.2 we support iSCSI multipath connecting all the portals of the same TPG.

*** This bug has been marked as a duplicate of bug 1353713 ***

Comment 15 Ab 2018-01-12 07:13:49 UTC
(In reply to Simone Tiraboschi from comment #14)
> (In reply to Ab from comment #13)
> 
> > .............................................................................
> > ......... [TPGs: 2]
> >     o- tpg1
> > .............................................................................
> > .................... [no-gen-acls, no-auth]
> >     | o- acls
> > .............................................................................
> > ............................... [ACLs: 1]
> >     | | o- iqn.2018-01.lab.ab:nfsserver
> > ...........................................................................
> > [Mapped LUNs: 2]
> >     | |   o- mapped_lun0
> > .............................................................................
> > ....... [lun0 block/isos (rw)]
> >     | |   o- mapped_lun1
> > .............................................................................
> > ..... [lun1 block/export (rw)]
> >     | o- luns
> > .............................................................................
> > ............................... [LUNs: 2]
> >     | | o- lun0 ..........................................................
> > [block/isos (/dev/nfs_server_vg/isos) (default_tg_pt_gp)]
> >     | | o- lun1 ......................................................
> > [block/export (/dev/nfs_server_vg/export) (default_tg_pt_gp)]
> >     | o- portals
> > .............................................................................
> > ......................... [Portals: 1]
> >     |   o- 0.0.0.0:3260
> > .............................................................................
> > .......................... [OK]
> >     o- tpg2
> > .............................................................................
> > .................... [no-gen-acls, no-auth]
> >       o- acls
> > .............................................................................
> > ............................... [ACLs: 1]
> >       | o- iqn.2018-01.lab.ab:rhevh1
> > .............................................................................
> > . [Mapped LUNs: 2]
> >       |   o- mapped_lun0
> > ..........................................................................
> > [lun0 block/self_hosted_lv (rw)]
> >       |   o- mapped_lun1
> > .............................................................................
> > ... [lun1 block/vms_data (rw)]
> >       o- luns
> > .............................................................................
> > ............................... [LUNs: 2]
> >       | o- lun0 ..................................... [block/self_hosted_lv
> > (/dev/self_hosted_vg/self_hosted_lv) (default_tg_pt_gp)]
> >       | o- lun1 ....................................................
> > [block/vms_data (/dev/vms_data_vg/vms_data) (default_tg_pt_gp)]
> >       o- portals
> > .............................................................................
> > ......................... [Portals: 0]
> > 
> > 
> > 'hosted-engine --deploy' chose TPG1 instead of TPG2 for unknown reason!
> 
> Pretty easy:
> you had 1 portal under tpg1 and 0 portals under tpg2, that's why you could
> not select tpg2

Thank you for your reply. 

This machine has only one NIC for now.
iSCSI target refused to create additional portal using the same NIC, so this is why I had only one portal. I think to need to add such situation in your bug fixing plan?
However, until you fix this issue, I will be needing to have dedicated iSCSI target for the self-hosted engine deployment which I do not prefer now. 

I will just wating for your reply regarding having only one portal for multiple LUNS over one NIC and you can close this bug.

Thank you for your efforts,
Ab

Comment 16 Simone Tiraboschi 2018-01-12 11:44:29 UTC
(In reply to Ab from comment #15)
> Thank you for your reply. 
> 
> This machine has only one NIC for now.
> iSCSI target refused to create additional portal using the same NIC, so this
> is why I had only one portal. I think to need to add such situation in your
> bug fixing plan?

Maybe you can simply try adding an additional IP address on your interface, not that sure.

> However, until you fix this issue, I will be needing to have dedicated iSCSI
> target for the self-hosted engine deployment which I do not prefer now. 
> 
> I will just wating for your reply regarding having only one portal for
> multiple LUNS over one NIC and you can close this bug.

Please take care that the discovery happens only once at deploy stage and not at runtime: once we know the target we don't need to discover again.
So you could simply temporary change your configuration, deploy and eventually then revert your iscsi configuration if needed.


Note You need to log in before you can comment on or make changes to this bug.