Bug 692565

Summary: VDSM - Storage: VDSM doesn't connect to iSCSI storage because of iscsiadm errors
Product: Red Hat Enterprise Linux 6 Reporter: Daniel Paikov <dpaikov>
Component: vdsmAssignee: Eduardo Warszawski <ewarszaw>
Status: CLOSED WORKSFORME QA Contact: yeylon <yeylon>
Severity: high Docs Contact:
Priority: high    
Version: 6.1CC: abaron, bazulay, hateya, iheim, srevivo, ykaul
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Linux   
Whiteboard: Storage
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-04-11 14:31:42 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
vdsm.log none

Description Daniel Paikov 2011-03-31 15:11:01 UTC
I rebooted the SPM host during an SPM action (create snapshot). Since the host returned from boot, it could no longer connect to iSCSI storage. Each attempt to activate the host made it become Non Operational. Attaching relevant vdsm.log.

Comment 1 Daniel Paikov 2011-03-31 15:11:53 UTC
Created attachment 489122 [details]
vdsm.log

Comment 3 Yaniv Kaul 2011-03-31 18:18:35 UTC
Please retest with -56 or later.

Comment 4 Daniel Paikov 2011-04-03 08:32:40 UTC
Tested with 57, same failure.

Comment 5 Yaniv Kaul 2011-04-03 08:40:59 UTC
Some of the issues from the log:
Thread-44::DEBUG::2011-03-31 10:02:52,137::iscsi::370::Storage.Misc.excCmd::(addiSCSIPortal) '/usr/bin/sudo -n /sbin/iscsiadm -m discoverydb -t sendtargets -p 10.16.45.61:3260 --discover' (cwd None)
Thread-44::DEBUG::2011-03-31 10:02:52,349::iscsi::370::Storage.Misc.excCmd::(addiSCSIPortal) SUCCESS: <err> = 'iscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-449b74a06-d07d255a7d44d5a5-yeylon-data-01]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-f6cb74a06-bf1d255a9074d7f3-yeylon-data-05]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-678b74a06-194d255a90a4d872-daffi-export]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-203b74a06-fa7000000cd4c3dd-north-i-1]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-214b74a06-e5d000000d04c3dd-north-i-2]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-3cab74a06-f9d000000d34c3dd-north-i-3]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-8ffb74a06-12cd255a8d84d7e0-daffi-01]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-9b7b74a06-1bcd255a90d4d90f-jlibosva-01]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-959b74a06-aced255a8484d738-paikov-large-3]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-95bb74a06-2fad255a9104d918-jlibosva-02]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-27fb74a06-ca6d255a9134d919-jlibosva-export]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-6bab74a06-865d255a9164d919-jlibosva-03]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-fabb74a06-d4dd255a9234d91a-paikov-3]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-ba8b74a06-a55000001654c8ea-rescure]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-b3db74a06-d21d255a9264d91b-atal1]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-b06b74a06-6a30000016b4c8f5-rescue2]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-057b74a06-6f10000016e4c8f6-test-01]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-069b74a06-fbe000001714c8f6-test-02]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-777b74a06-59a000001744c8f6-test-03]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-78fb74a06-e26000001774c8f6-test-04]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-fbab74a06-a700000017a4cc89-rhevh]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-22bb74a06-8c7d255a8a84d74d-paikov-export]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-35eb74a06-5b1000e7a024cf84-rhevh2]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-e6ab74a06-a58000e7a074cf92-test-iscsi-offload]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-0b1b74a06-755000e7a0a4cf93-test-iscsi-offload-2]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-e2bb74a06-f18000e7a0f4cfd3-mpio-test]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-32fb74a06-06f000e7a124d07d-multiiqn]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-277b74a06-7b2000e7a154d081-test-ipv6]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-871b74a06-af2d255a8c64d7cd-paikov-large-4]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-a7bb74a06-237d255a8ce4d7ce-paikov-large-6]\niscsiadm: Could not stat /var/lib/iscsi/nodes//,3260,-1/default to delete node: No such file or directory\n\niscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 10.16.45.61,3260,1 iqn.2001-05.com.equallogic:0-8a0906-2bbb74a06-2c5d255a8d14d7cf-paikov-large-7]\n'; <rc> = 0
Thread-44::DEBUG::2011-03-31 10:02:52,350::iscsi::468::Storage.Misc.excCmd::(addiSCSINode) '/usr/bin/sudo -n /sbin/iscsiadm -m node -T iqn.2001-05.com.equallogic:0-8a0906-2bbb74a06-2c5d255a8d14d7cf-paikov-large-7 -l -p 10.16.45.61:3260' (cwd None)
Thread-44::DEBUG::2011-03-31 10:02:52,380::iscsi::468::Storage.Misc.excCmd::(addiSCSINode) FAILED: <err> = 'iscsiadm: No records found\n'; <rc> = 21
Thread-44::DEBUG::2011-03-31 10:02:52,381::misc::1011::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi._getiSCSISessionList)
Thread-44::DEBUG::2011-03-31 10:02:52,382::misc::1013::SamplingMethod::(__call__) Got in to sampling method
Thread-44::DEBUG::2011-03-31 10:02:52,382::iscsi::533::Storage.Misc.excCmd::(_getiSCSISessionList) '/usr/bin/sudo -n /sbin/iscsiadm -m session' (cwd None)
Thread-44::DEBUG::2011-03-31 10:02:52,407::iscsi::533::Storage.Misc.excCmd::(_getiSCSISessionList) FAILED: <err> = 'iscsiadm: No active sessions.\n'; <rc> = 21
Thread-44::ERROR::2011-03-31 10:02:52,408::iscsi::476::Storage.iScsi::(addiSCSINode) Could not get iscsi session list
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/iscsi.py", line 473, in addiSCSINode
    if checkSession(ip, port, iqn, tpgt, username, password):
  File "/usr/share/vdsm/storage/iscsi.py", line 602, in checkSession
    return checkSessionList([[host, port, iqn, tpgt, user, password]])[0]
  File "/usr/share/vdsm/storage/iscsi.py", line 576, in checkSessionList
    l = _getiSCSISessionList()
  File "/usr/share/vdsm/storage/misc.py", line 1028, in helper
    return sm(*args, **kwargs)
  File "/usr/share/vdsm/storage/misc.py", line 1015, in __call__
    self.__lastResult = self.__func(*args, **kwargs)
  File "/usr/share/vdsm/storage/iscsi.py", line 535, in _getiSCSISessionList
    raise se.GetiSCSISessionListError
GetiSCSISessionListError: Get iSCSI session list error: ()
Thread-44::DEBUG::2011-03-31 10:02:52,409::iscsi::507::Storage.Misc.excCmd::(remiSCSINode) '/usr/bin/sudo -n /sbin/iscsiadm -m node -o delete -T iqn.2001-05.com.equallogic:0-8a0906-2bbb74a06-2c5d255a8d14d7cf-paikov-large-7 -p 10.16.45.61:3260' (cwd None)
Thread-44::DEBUG::2011-03-31 10:02:52,440::iscsi::507::Storage.Misc.excCmd::(remiSCSINode) FAILED: <err> = 'iscsiadm: No records found\n'; <rc> = 21
Thread-44::ERROR::2011-03-31 10:02:52,441::iscsi::481::Storage.iScsi::(addiSCSINode) Could not remove iscsi node
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/iscsi.py", line 479, in addiSCSINode
    remiSCSINode(ip, port, iqn, tpgt, username, password, logout=False)
  File "/usr/share/vdsm/storage/iscsi.py", line 509, in remiSCSINode
    raise se.RemoveiSCSINodeError(portal)
RemoveiSCSINodeError: Remove iSCSI node error: ('10.16.45.61:3260',)
Thread-44::ERROR::2011-03-31 10:02:52,442::storage_connection::223::Storage.ServerConnection::(__connectiSCSIServer) Error during storage connection: Failed to login to iSCSI node: ('10.16.45.61:3260',)
Traceback (most recent call last):
  File "/usr/share/vdsm/storage/storage_connection.py", line 217, in __connectiSCSIServer
    con['user'], con['password'])
  File "/usr/share/vdsm/storage/iscsi.py", line 483, in addiSCSINode
    raise e

Comment 6 Daniel Paikov 2011-04-03 08:44:58 UTC
The iscsi-initiator-utils version is 6.2.0.872-17.

Comment 7 Daniel Paikov 2011-04-11 14:31:42 UTC
Stopped reproducing. Closing for now.