Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 656284

Summary:

[rhel6] [scsi-utils] iSCSI authentication error: Could not stat /var/lib/iscsi/nodes//,3260,-1 /default to delete node err 2\n\niscsiadm: Could not add/update

Product:

Red Hat Enterprise Linux 6

Reporter:

Haim <hateya>

Component:

iscsi-initiator-utils

Assignee:

Andy Grover <agrover>

Status:

CLOSED WORKSFORME

QA Contact:

Red Hat Kernel QE team <kernel-qe>

Severity:

high

Docs Contact:

Priority:

medium

Version:

6.1

CC:

abaron, bdonahue, coughlan, cpelland, danken, hateya, iheim, mchristi, mgoldboi, nicolas, plyons, realrichardsharpe, yeylon, ykaul, zliu

Target Milestone:

Target Release:

---

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Environment:

Last Closed:

2011-09-18 13:25:31 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

682015

Attachments:

Description	Flags
iscsi command list	none
full vdsm log	none

Description Haim 2010-11-23 12:23:05 UTC

Created attachment 462309 [details]
iscsi command list

Description of problem:

context: 

every now and then I get the following error in vdsm (component in rhev product, which responsible to handle and configure storage, iscsi in this case). 

 <err> = 'iscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1
/default to delete node err 2\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.35.65.25,3260,1 scale_sd2]\n'

this problem prevents from host connecting to iscsi target. 
problem is scenario is not conclusive, and i don't know what triggers it. 

commands are (also see attachment):

'/usr/bin/sudo -n /sbin/iscsiadm -m discoverydb -t sendtargets -p 10.35.6
4.25 -o new'

'/usr/bin/sudo -n /sbin/iscsiadm -m discoverydb -t sendtargets -p 10.35.6
4.25 -o update -n discovery.sendtargets.auth.authmethod -v CHAP' (cwd None)

'/usr/bin/sudo -n /sbin/iscsiadm -m discoverydb -t sendtargets -p 10.35.6
4.25 -o update -n discovery.sendtargets.auth.username -v (null)' (cwd None)

'/sbin/iscsiadm -m discoverydb -t sendtargets -p 10.35.64.25 -o update -n
 discovery.sendtargets.auth.password -v ******' (cwd None)

'/usr/bin/sudo -n /sbin/iscsiadm -m discoverydb -t sendtargets -p 10.35.6
4.25 --discover' (cwd None)

<err> = 'iscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1
/default to delete node err 2\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.35.64.25,3260,1 scale_sd2]\n'; <rc> = 0

Comment 1 Haim 2010-11-23 12:34:56 UTC

please note that file /var/lib/iscsi/nodes/scale_sd2/10.35.64.25,3260,1/default is empty. 
please also note that i'm using multi path round robin configuration.

Comment 4 Mike Christie 2010-11-23 22:46:02 UTC

Can some of these commands run in parallel? If so I think it might be iscsiadm instances working on the db at the same time. I am working on patches to fix the locking for this.

Comment 5 Ayal Baron 2010-11-24 07:34:31 UTC

(In reply to comment #4)
> Can some of these commands run in parallel? If so I think it might be iscsiadm
> instances working on the db at the same time. I am working on patches to fix
> the locking for this.

Actually I would think that these commands should be collapsed into 1 command and a single db update, e.g.:

iscsiadm -m discoverydb -t sendtargets -p 10.35.64.25 -o new -n discovery.sendtargets.auth.authmethod -v CHAP -n discovery.sendtargets.auth.username -v "" -n discovery.sendtargets.auth.password -v ""

iscsiadm -m discoverydb -t sendtargets -p 10.35.64.25 --discover


but what does running them in parallel have to do with anything?
What is the issue to begin with?
and why is the command returning with rc = 0 ?

Comment 6 Ayal Baron 2010-11-24 10:41:45 UTC

Please ignore previous comment, I just realized I misread your comment.
No, all these commands are run sequentially from the same thread.

Comment 8 Mike Christie 2010-12-07 01:43:16 UTC

Just to confirm the log in https://bugzilla.redhat.com/attachment.cgi?id=462309 is after the "Could not stat" error, right?

When you are running these commands, are you logged into the scale_sd2 target?

Comment 9 Mike Christie 2010-12-07 01:46:02 UTC

Can you give me access to the target setup you are using for this. While trying to recreate it here, I am probably missing something that is causing the problem.

Comment 10 Mike Christie 2010-12-07 02:17:40 UTC

One more question.

In the log you attached, is this a similar problem?

(execCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m discoverydb -t sendtargets -p 10.35.64.25 --discover' (cwd None)
[hw=,ip=,net_if=,iscsi_if=default] 10.35.64.25,3260,1 scale_sd2]\n'; <rc> = 0


Does this output indicate that a target with the name scale_sd2 was returned from discovery?


(execCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m node -T scale_sd2 -o update -n node.session.auth.authmethod -v CHAP' (cwd None)
 no records found!\n'; <rc> = 255
3260' (cwd None)
 no records found!\n'; <rc> = 255
(addiSCSINode) Unexpected error


But then here when you tried to update the CHAP setting you got a error?

Comment 11 Ayal Baron 2010-12-07 07:59:44 UTC

(In reply to comment #10)
> One more question.
> 
> In the log you attached, is this a similar problem?
> 
> (execCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m discoverydb -t sendtargets -p
> 10.35.64.25 --discover' (cwd None)
> [hw=,ip=,net_if=,iscsi_if=default] 10.35.64.25,3260,1 scale_sd2]\n'; <rc> = 0
> 
> 
> Does this output indicate that a target with the name scale_sd2 was returned
> from discovery?

looks like a grep error or something.
Haim, do you still have the raw log?
Can you reproduce?
What about giving Mike access to the setup?

> 
> 
> (execCmd) '/usr/bin/sudo -n /sbin/iscsiadm -m node -T scale_sd2 -o update -n
> node.session.auth.authmethod -v CHAP' (cwd None)
>  no records found!\n'; <rc> = 255
> 3260' (cwd None)
>  no records found!\n'; <rc> = 255
> (addiSCSINode) Unexpected error
> 
> 
> But then here when you tried to update the CHAP setting you got a error?

Comment 12 Haim 2010-12-07 08:34:43 UTC

Created attachment 465187 [details]
full vdsm log

attached full vdsm log that contains the problem.

Comment 13 Mike Christie 2011-01-18 06:12:15 UTC

Hey,

Just wanted to update this. I can replicate it now. I think I should be able to fix it by the end of the week.

Comment 14 Mike Christie 2011-01-20 20:58:58 UTC

(In reply to comment #13)
> Just wanted to update this. I can replicate it now. I think I should be able to
> fix it by the end of the week.

I am not sure what happened. I had a script that recreated the problem about a like 20 times, but now I cannot replicate it. Been running the script for a couple million runs and have not hit it again.

If you guys have a setup that can easily reproduce this could you give me ssh access to the box?

It does not seem to be auth related, I could reproduce it just by running the discovery command.

Comment 22 Richard Sharpe 2015-11-16 16:14:35 UTC

We are hitting this and the problem seems to be corruption of the config files. Don't yet know why they are corrupt, but we see two type of corruption:

---------------------------
/var/lib/iscsi/nodes/iqn.2010-06.com.nutanix:volumegroup-d86fb4c6-0a26-4a99-a4c4-38966b1955c8/10.5.20.138,3260,1^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@# BEGIN RECORD 6.2.0-873.13.el6
node.name = iqn.2010-06.com.nutanix:volumegroup-5bd06b21-fb06-4b8e-b973-e81b86409e39
------------------------------------------

and one where the second line is missing entirely, which seems to cause the problems mentioned above, since various idbm routines bail out.