Bug 1009299

Summary: Fail to discover iscsi target | iscsiadm: can not connect to iSCSI daemon (111)!
Product: Red Hat Enterprise Linux 6 Reporter: Meni Yakove <myakove>
Component: iscsi-initiator-utilsAssignee: Chris Leech <cleech>
Status: CLOSED ERRATA QA Contact: Bruno Goncalves <bgoncalv>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 6.5CC: agrover, c.hendrickson09, cleech, coughlan, cww, ederevea, eedri, iheim, jherrman, myakove, salmy, yanwang
Target Milestone: rcKeywords: AutomationBlocker, Regression, Reopened, TestBlocker
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: iscsi-initiator-utils-6.2.0.873-11.el6 Doc Type: Bug Fix
Doc Text:
Due to changes in the addressing of inter-process communication (IPC) sockets, updating the isciadm administration utility caused it to be incompatible with the already-running iscsid processes. As a consequence, iscsiadm could no longer communicate with the running iscsid processes after the update. This update adds a retroactive compatibility IPC mode, which automatically triggers when a new IPC socket fails to connect. As a result, iscsiadm can now correctly be used to control pre-existing iscsid processes after an update.
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-14 08:10:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Bug Depends On:    
Bug Blocks: 994246    

Description Meni Yakove 2013-09-18 07:19:52 UTC
Description of problem:
iscsiadm fail to discover iscsi target with error iscsiadm: can not connect to iSCSI daemon (111)!


Version-Release number of selected component (if applicable):
iscsi-initiator-utils-6.2.0.873-8.el6.x86_64

How reproducible:
100%

Steps to Reproduce:
1.iscsiadm -m discoverydb -t st -p IP:PORT --discover



Additional info:
Starting iscsid: 
iscsiadm: can not connect to iSCSI daemon (111)!
Starting iscsid: 
iscsiadm: can not connect to iSCSI daemon (111)!
iscsiadm: Cannot perform discovery. Initiatorname required.
iscsiadm: Could not perform SendTargets discovery: could not connect to iscsid

Comment 4 Chris Leech 2013-09-18 19:04:36 UTC
(In reply to Meni Yakove from comment #0)

Is it possible that you have an iscsid process running from an older package, with a newer iscsiadm after updating the package?  Could you try restarting iscsid?

Comment 5 Meni Yakove 2013-09-18 21:04:00 UTC
(In reply to Chris Leech from comment #4)
> (In reply to Meni Yakove from comment #0)
> 
> Is it possible that you have an iscsid process running from an older
> package, with a newer iscsiadm after updating the package?  Could you try
> restarting iscsid?

It's seems that is the issue, I updated to iscsi-initiator-utils-6.2.0.873-8.el6.x86_64 and after reboot I can run discovery.

When trying to restart iscsid service I got:
Not stopping iscsid: iscsi sessions still active           [WARNING]

so the only way is to kill all iscsid processes or reboot the host.

Comment 6 Corbin Hendrickson 2014-01-03 17:55:08 UTC
Not sure this is entirely not a bug. I also experience this issue. After upgrading from iscsi-initiator-utils-6.2.0.873-2.el6.x86_64.rpm to iscsi-initiator-utils-6.2.0.873-10.el6.x86_64.rpm you can see in a strace output we can no longer connect to the iscsid socket via a Connection refused when I run iscsiadm -m session -P 3. 

20599 write(1, "Target: iqn.2010-01.com.myhostname"..., 53) = 53
20599 write(1, "\tCurrent Portal: 172.16.0.115:32"..., 37) = 37
20599 write(1, "\tPersistent Portal: 172.16.0.115"..., 40) = 40
20599 write(1, "\t\t**********\n", 13)  = 13
20599 write(1, "\t\tInterface:\n", 13)  = 13
20599 write(1, "\t\t**********\n", 13)  = 13
20599 write(1, "\t\tIface Name: default\n", 22) = 22
20599 write(1, "\t\tIface Transport: tcp\n", 23) = 23
20599 write(1, "\t\tIface Initiatorname: iqn.2010-"..., 55) = 55
20599 write(1, "\t\tIface IPaddress: 172.16.4.41\n", 31) = 31
20599 write(1, "\t\tIface HWaddress: <empty>\n", 27) = 27
20599 write(1, "\t\tIface Netdev: <empty>\n", 24) = 24
20599 write(1, "\t\tSID: 9\n", 9)       = 9
20599 socket(PF_FILE, SOCK_STREAM, 0)   = 3
20599 connect(3, {sa_family=AF_FILE, path=@"ISCSIADM_ABSTRACT_NAMESPACE"}, 30) = -1 ECONNREFUSED (Connection refused)
20599 open("/etc/iscsi/iscsid.conf", O_RDONLY) = 4
20599 fstat(4, {st_mode=S_IFREG|0600, st_size=11608, ...}) = 0
20599 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f848fca4000
20599 read(4, "#\n# Open-iSCSI default configura"..., 4096) = 4096
20599 close(4)                          = 0


Here's the strace output of iscsiadm -m -P 3 after I upgrade from iscsi-initiator-utils-6.2.0.872-41.el6.x86_64.rpm to iscsi-initiator-utils-6.2.0.873-2.el6.x86_64.rpm where its able to connect just fine

23083 write(1, "Target: iqn.2010-01.com.bluehost"..., 53) = 53
23083 write(1, "\tCurrent Portal: 172.16.0.115:32"..., 37) = 37
23083 write(1, "\tPersistent Portal: 172.16.0.115"..., 40) = 40
23083 write(1, "\t\t**********\n", 13)  = 13
23083 write(1, "\t\tInterface:\n", 13)  = 13
23083 write(1, "\t\t**********\n", 13)  = 13
23083 write(1, "\t\tIface Name: default\n", 22) = 22
23083 write(1, "\t\tIface Transport: tcp\n", 23) = 23
23083 write(1, "\t\tIface Initiatorname: iqn.2010-"..., 55) = 55
23083 write(1, "\t\tIface IPaddress: 172.16.4.41\n", 31) = 31
23083 write(1, "\t\tIface HWaddress: <empty>\n", 27) = 27
23083 write(1, "\t\tIface Netdev: <empty>\n", 24) = 24
23083 write(1, "\t\tSID: 10\n", 10)     = 10
23083 socket(PF_FILE, SOCK_STREAM, 0)   = 3
23083 connect(3, {sa_family=AF_FILE, path=@"ISCSIADM_ABSTRACT_NAMESPACE"}, 110) = 0
23083 write(3, "\r\0\0\0\0\0\0\0\n\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 12856) = 12856
23083 recvfrom(3, "\r\0\0\0\0\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2464, MSG_WAITALL, NULL, NULL) = 2464
23083 close(3)                          = 0
23083 write(1, "\t\tiSCSI Connection State: LOGGED"..., 36) = 36
23083 stat("/sys/subsystem/iscsi_session/devices/session10", 0x7fff6dab6b80) = -1 ENOENT (No such file or directory)
23083 stat("/sys/bus/iscsi_session/devices/session10", 0x7fff6dab6b80) = -1 ENOENT (No such file or directory)
23083 stat("/sys/class/iscsi_session/session10", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
23083 lstat("/sys/class/iscsi_session/session10/state", {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0
23083 open("/sys/class/iscsi_session/session10/state", O_RDONLY) = 3
23083 read(3, "LOGGED_IN\n", 256)       = 10
23083 close(3)                          = 0

Comment 7 Corbin Hendrickson 2014-01-03 20:13:55 UTC
Additional information, after removing the following patch: iscsi-initiator-utils-aligned-all-socket-comm-between-iscsiadm-iscsid-and-.patch 

I was able to update from iscsi-initiator-utils-6.2.0.873-2.el6.x86_64.rpm to my own rebuilt rpm without the iscsi-initiator-utils-aligned-all-socket-comm-between-iscsiadm-iscsid-and-.patch patch. 

24799 write(1, "Target: iqn.2010-01.com.myhostname"..., 53) = 53
24799 write(1, "\tCurrent Portal: 172.16.0.115:32"..., 37) = 37
24799 write(1, "\tPersistent Portal: 172.16.0.115"..., 40) = 40
24799 write(1, "\t\t**********\n", 13)  = 13
24799 write(1, "\t\tInterface:\n", 13)  = 13
24799 write(1, "\t\t**********\n", 13)  = 13
24799 write(1, "\t\tIface Name: default\n", 22) = 22
24799 write(1, "\t\tIface Transport: tcp\n", 23) = 23
24799 write(1, "\t\tIface Initiatorname: iqn.2010-"..., 55) = 55
24799 write(1, "\t\tIface IPaddress: 172.16.4.41\n", 31) = 31
24799 write(1, "\t\tIface HWaddress: <empty>\n", 27) = 27
24799 write(1, "\t\tIface Netdev: <empty>\n", 24) = 24
24799 write(1, "\t\tSID: 13\n", 10)     = 10
24799 socket(PF_FILE, SOCK_STREAM, 0)   = 3
24799 connect(3, {sa_family=AF_FILE, path=@"ISCSIADM_ABSTRACT_NAMESPACE"}, 110) = 0
24799 write(3, "\r\0\0\0\0\0\0\0\r\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 13624) = 13624
24799 recvfrom(3, "\r\0\0\0\0\0\0\0\0\0\0\0\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 2464, MSG_WAITALL, NULL, NULL) = 2464
24799 close(3)                          = 0

Comment 12 Bruno Goncalves 2014-07-09 09:20:16 UTC
Tested iscsi-initiator-utils update from RHEL-6.4 and also 6.5, and they was successful.

RHEL-6.4:

1. rpm -q iscsi-initiator-utils
iscsi-initiator-utils-6.2.0.873-2.el6.x86_64

2. Establish iSCSI session
iscsiadm -m node -l 
Logging in to [iface: default, target: iqn.1992-08.com.netapp:sn.151753773, portal: 10.16.41.222,3260] (multiple)
Logging in to [iface: default, target: iqn.1992-08.com.netapp:sn.151753773, portal: 10.16.43.127,3260] (multiple)
Login to [iface: default, target: iqn.1992-08.com.netapp:sn.151753773, portal: 10.16.41.222,3260] successful.
Login to [iface: default, target: iqn.1992-08.com.netapp:sn.151753773, portal: 10.16.43.127,3260] successful.

3. Update initiator package
rpm -q iscsi-initiator-utils
iscsi-initiator-utils-6.2.0.873-11.el6.x86_64

4. iscsiadm still works
iscsiadm -m node -u
Logging out of session [sid: 1, target: iqn.1992-08.com.netapp:sn.151753773, portal: 10.16.41.222,3260]
Logging out of session [sid: 2, target: iqn.1992-08.com.netapp:sn.151753773, portal: 10.16.43.127,3260]
Logout of [sid: 1, target: iqn.1992-08.com.netapp:sn.151753773, portal: 10.16.41.222,3260] successful.
Logout of [sid: 2, target: iqn.1992-08.com.netapp:sn.151753773, portal: 10.16.43.127,3260] successful.

----------

Same on 6.5

1. rpm -q iscsi-initiator-utils
iscsi-initiator-utils-6.2.0.873-10.el6.x86_64

2. Establish iSCSI session
iscsiadm -m node -l 
Logging in to [iface: default, target: iqn.1992-08.com.netapp:sn.151753773, portal: 10.16.41.222,3260] (multiple)
Logging in to [iface: default, target: iqn.1992-08.com.netapp:sn.151753773, portal: 10.16.43.127,3260] (multiple)
Login to [iface: default, target: iqn.1992-08.com.netapp:sn.151753773, portal: 10.16.41.222,3260] successful.
Login to [iface: default, target: iqn.1992-08.com.netapp:sn.151753773, portal: 10.16.43.127,3260] successful.

3. Update initiator package
rpm -q iscsi-initiator-utils
iscsi-initiator-utils-6.2.0.873-11.el6.x86_64

4. iscsiadm still works
iscsiadm -m node -u
Logging out of session [sid: 1, target: iqn.1992-08.com.netapp:sn.151753773, portal: 10.16.41.222,3260]
Logging out of session [sid: 2, target: iqn.1992-08.com.netapp:sn.151753773, portal: 10.16.43.127,3260]
Logout of [sid: 1, target: iqn.1992-08.com.netapp:sn.151753773, portal: 10.16.41.222,3260] successful.
Logout of [sid: 2, target: iqn.1992-08.com.netapp:sn.151753773, portal: 10.16.43.127,3260] successful.

Comment 14 errata-xmlrpc 2014-10-14 08:10:16 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1580.html