Bug 1529226 - [otopi deployment] Deploy SHE failed with iscsi via cockpit.
Summary: [otopi deployment] Deploy SHE failed with iscsi via cockpit.
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: cockpit-ovirt
Classification: oVirt
Component: Hosted Engine
Version: ---
Hardware: Unspecified
OS: Unspecified
high
urgent
Target Milestone: ovirt-4.2.2
: 0.11.12
Assignee: Simone Tiraboschi
QA Contact: Yihui Zhao
URL:
Whiteboard:
Depends On: 1539598 1541029 1548958
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-12-27 07:59 UTC by Yihui Zhao
Modified: 2018-03-29 10:58 UTC (History)
14 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-03-29 10:58:37 UTC
oVirt Team: Integration
Embargoed:
rule-engine: ovirt-4.2+
rule-engine: blocker+
cshao: testing_ack+


Attachments (Terms of Use)
cockpit_iscsi_he_failed.png (38.17 KB, image/png)
2017-12-27 07:59 UTC, Yihui Zhao
no flags Details
cockpit_iscsi_failed.png (166.18 KB, image/png)
2018-01-23 05:31 UTC, Yihui Zhao
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1539598 0 unspecified CLOSED Retrieval of iSCSI targets failed while deploying HE via cockpit based otopi deployment. 2021-02-22 00:41:40 UTC
oVirt gerrit 87141 0 None None None 2018-02-05 14:12:21 UTC

Internal Links: 1539598

Description Yihui Zhao 2017-12-27 07:59:34 UTC
Created attachment 1372706 [details]
cockpit_iscsi_he_failed.png

Description of problem:
Deploy SHE failed with iscsi via cockpit. The error is "Failed to execute stage Environment customization': local variable 'f_targets' referenced before assignment".

Version-Release number of selected component (if applicable):
cockpit-ovirt-dashboard-0.11.3-0.1.el7ev.noarch
ovirt-hosted-engine-ha-2.2.2-1.el7ev.noarch
ovirt-hosted-engine-setup-2.2.2-1.el7ev.noarch
rhvh-4.2.0.6-0.20171218.0+1
rhvm-appliance-4.2-20171219.0.el7.noarch

How reproducible:
100%

Steps to Reproduce:
1. Install the latest RHVH 4.2
2. Deploy the SHE with iscsi target lun via cockpit


Actual results:
After step2, occured the error "Failed to execute stage Environment customization': local variable 'f_targets' referenced before assignment".


Expected results:
After step2, there is no error and deploy SHE successfully.
Additional info:
Deploy the SHE with the same iscsi target lun via CLI successfully.
[root@bootp-73-75-201 tmp]# hosted-engine --vm-status


--== Host 1 status ==--

conf_on_shared_storage             : True
Status up-to-date                  : True
Hostname                           : bootp-73-75-201.lab.eng.pek2.redhat.com
Host ID                            : 1
Engine status                      : {"health": "good", "vm": "up", "detail": "Up"}
Score                              : 3400
stopped                            : False
Local maintenance                  : False
crc32                              : 6b03dc01
local_conf_timestamp               : 71852
Host timestamp                     : 71847
Extra metadata (valid at timestamp):
	metadata_parse_version=1
	metadata_feature_version=1
	timestamp=71847 (Wed Dec 27 14:55:48 2017)
	host-id=1
	score=3400
	vm_conf_refresh_time=71852 (Wed Dec 27 14:55:54 2017)
	conf_on_shared_storage=True
	maintenance=False
	state=EngineUp
	stopped=False

######################################################
# Deploy process
######################################################
'''''''''
[ INFO  ] Stage: Environment customization
         
          --== STORAGE CONFIGURATION ==--
         
          Please specify the storage you would like to use (glusterfs, iscsi, fc, nfs3, nfs4)[nfs3]: iscsi
          Please specify the iSCSI portal IP address: 10.66.8.158
          Please specify the iSCSI portal port [3260]: 
          Please specify the iSCSI portal user: yzhao
          Please specify the iSCSI portal password: 
          The following targets have been found:
          	[1]	iqn.iscsi-2017.02.com
          		TPGT: 1, portals:
          			10.66.8.158:3260
         
          	[2]	iqn.iscsi-2017.com
          		TPGT: 1, portals:
          			10.66.8.158:3260
         
          	[3]	iqn.iscsi-2017.03.com
          		TPGT: 1, portals:
          			10.66.8.158:3260
         
          	[4]	iqn.iscsi-2017.01.com
          		TPGT: 1, portals:
          			10.66.8.158:3260
         
          Please select a target (1, 2, 3, 4) [1]: 3
[ INFO  ] Connecting to the storage server
          The following luns have been found on the requested target:
          	[1]	360000000000000000e00000000040001	60GiB	IET	VIRTUAL-DISK
          		status: free, paths: 1 active
         
          Please select the destination LUN (1) [1]: 
[ INFO  ] Connecting to the storage server
         
          --== HOST NETWORK CONFIGURATION ==--
''''''''

Comment 1 Ryan Barry 2017-12-27 08:04:46 UTC
What if you try with the same answerfile on the couch?

Comment 3 Ryan Barry 2017-12-27 08:21:07 UTC
"local variable referenced before assignment" reads like Python.

Cockpit generates an answer file. If that answer file is used with the CLI, does it work?

Comment 8 Phillip Bailey 2018-01-04 20:16:07 UTC
There's a patch [1] that was merged on December 28th, to address this issue. Could you please test with a build that includes this patch and let me know if you still experience this issue?

[1] https://gerrit.ovirt.org/#/c/85779/

Comment 10 kukucz 2018-01-05 23:09:49 UTC
(In reply to Phillip Bailey from comment #8)
> There's a patch [1] that was merged on December 28th, to address this issue.
> Could you please test with a build that includes this patch and let me know
> if you still experience this issue?
> 
> [1] https://gerrit.ovirt.org/#/c/85779/

Hi. 
Probably this patch not work.
I found this bug description by google.
I used latest ovirt-node-ng-installer-ovirt-4.2-2018010506.iso, when I try deploy Hosted Engine on iscsi drive I still get in GUI:

Failed to execute stage 'Environment customization': local variable 'f_targets' referenced before assignment
Hosted Engine deployment failed

Part of log file:

2018-01-05 22:51:50,172+0100 DEBUG otopi.plugins.gr_he_setup.storage.blockd blockd._iscsi_discovery:359 [u'192.168.0.100:3260,1 iqn.1991-05.com.microsoft:iscsi-iscsi-target', u'192.168.1.100:3260,1 iqn.1991-05.com.microsoft:iscsi-iscsi
-target', u'192.168.20.100:3260,1 iqn.1991-05.com.microsoft:iscsi-iscsi-target']
                                                                                                                                                        
2018-01-05 22:51:50,172+0100 DEBUG otopi.plugins.gr_he_setup.storage.blockd blockd._iscsi_discovery:382 found: {u'iqn.1991-05.com.microsoft:iscsi-iscsi-target': {u'1': [{'ip': u'192.168.0.100', 'port': u'3260'}, {'ip': u'192.168.1.100'
, 'port': u'3260'}, {'ip': u'192.168.20.100', 'port': u'3260'}]}} 
                                                                                                                                                                         
2018-01-05 22:51:50,173+0100 DEBUG otopi.context context._executeMethod:143 method exception                                                                                                                                               
Traceback (most recent call last):                                                                                                                                                                                                         
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod                                                                                                                                                    
    method['method']()                                                                                                                                                                                                                     
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-setup/storage/blockd.py", line 681, in _customization                                                                                                                
    values=valid_targets_dict,                                                                                                                                                                                                             
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-setup/storage/blockd.py", line 243, in _customize_target                                                                                                             
    f_targets[int(s_target)-1]['target'],                                                                                                                                                                                                  
UnboundLocalError: local variable 'f_targets' referenced before assignment
                                                                                                                                                                 
2018-01-05 22:51:50,174+0100 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Environment customization': local variable 'f_targets' referenced before assignment

Comment 11 kukucz 2018-01-05 23:39:25 UTC
The same for oVirt node deployed from ovirt-node-ng-installer-ovirt-4.2-2018010506.iso by hosted-engine --deploy

Hosted engine is not deployed on iscsi drive.

Part of script output:

[ INFO  ] Stage: Setup validation                                                                                                                                                                                                          
                                                                                                                                                                                                                                           
          --== CONFIGURATION PREVIEW ==--                                                                                                                                                                                                  
                                                                                                                                                                                                                                           
          Bridge interface                   : ens32                                                                                                                                                                                       
          Engine FQDN                        : engine.plopclab                                                                                                                                                                             
          Bridge name                        : ovirtmgmt                                                                                                                                                                                   
          Host address                       : node1.plopclab                                                                                                                                                                              
          SSH daemon port                    : 22                                                                                                                                                                                          
          Firewall manager                   : iptables                                                                                                                                                                                    
          Gateway address                    : 192.168.20.2                                                                                                                                                                                
          Storage Domain type                : iscsi                                                                                                                                                                                       
          LUN ID                             : 360003ff44dc75adcaabfd1d09868c746                                                                                                                                                           
          Image size GB                      : 50                                                                                                                                                                                          
          iSCSI Portal IP Address            : 192.168.0.100,192.168.1.100,192.168.20.100                                                                                                                                                  
          iSCSI Target Name                  : iqn.1991-05.com.microsoft:iscsi-iscsi-target                                                                                                                                                
          iSCSI Portal port                  : 3260,3260,3260                                                                                                                                                                              
          Host ID                            : 1                                                                                                                                                                                           
          iSCSI Target Portal Group Tag      : 1                                                                                                                                                                                           
          iSCSI Portal user                  :                                                                                                                                                                                             
          Memory size MB                     : 4096                                                                                                                                                                                        
          Console type                       : vnc                                                                                                                                                                                         
          Number of CPUs                     : 2                                                                                                                                                                                           
          MAC address                        : 00:16:3e:77:f9:41                                                                                                                                                                           
          OVF archive (for disk boot)        : /usr/share/ovirt-engine-appliance/ovirt-engine-appliance-4.2-20171219.1.el7.centos.ova                                                                                                      
          Appliance version                  : 4.2-20171219.1.el7.centos                                                                                                                                                                   
          Restart engine VM after engine-setup: True                                                                                                                                                                                       
          Engine VM timezone                 : Europe/Warsaw                                                                                                                                                                               
          CPU Type                           : model_Broadwell                                                                                                                                                                             
                                                                                                                                                                                                                                           
          Please confirm installation settings (Yes, No)[Yes]:                                                                                                                                                                             
[ INFO  ] Stage: Transaction setup                                                                                                                                                                                                         
[ INFO  ] Stage: Misc configuration                                                                                                                                                                                                        
[ INFO  ] Stage: Package installation                                                                                                                                                                                                      
[ INFO  ] Stage: Misc configuration                                                                                                                                                                                                        
[ INFO  ] Configuring libvirt                                                                                                                                                                                                              
[ INFO  ] Configuring VDSM                                                                                                                                                                                                                 
[ INFO  ] Starting vdsmd                                                                                                                                                                                                                   
[ INFO  ] Configuring the management bridge                                                                                                                                                                                                
[ INFO  ] Creating Volume Group                                                                                                                                                                                                            
[ ERROR ] Failed to execute stage 'Misc configuration': Connection to storage server failed                                                                                                                                                
[ INFO  ] Yum Performing yum transaction rollback                                                                                                                                                                                          
[ INFO  ] Stage: Clean up                                                                                                                                                                                                                  
[ INFO  ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20180105232224.conf'                                                                                                                                  
[ INFO  ] Stage: Pre-termination                                                                                                                                                                                                           
[ INFO  ] Stage: Termination                                                                                                                                                                                                               
[ ERROR ] Hosted Engine deployment failed: this system is not reliable, please check the issue,fix and redeploy                                                                                                                            
          Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20180105231838-i98rlg.log


Part of log file:

2018-01-05 23:22:24,011+0100 DEBUG otopi.plugins.gr_he_setup.storage.storage storage._storageServerConnection:504 [{u'status': 0, u'id': u'f18a2681-eff4-4556-a001-678f36d88cdf'}, {u'status': 0, u'id': u'f18a2681-eff4-4556-a001-678f36d8
8cdf'}, {u'status': 465, u'id': u'f18a2681-eff4-4556-a001-678f36d88cdf'}] 
                                                                                                                                                                 
2018-01-05 23:22:24,012+0100 DEBUG otopi.context context._executeMethod:143 method exception                                                                                                                                               
Traceback (most recent call last):                                                                                                                                                                                                         
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod                                                                                                                                                    
    method['method']()                                                                                                                                                                                                                     
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-setup/storage/storage.py", line 1049, in _misc                                                                                                                       
    self._storageServerConnection()                                                                                                                                                                                                        
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/gr-he-setup/storage/storage.py", line 512, in _storageServerConnection                                                                                                     
    _('Connection to storage server failed')                                                                                                                                                                                               
RuntimeError: Connection to storage server failed   
                                                                                                                                                                                       
2018-01-05 23:22:24,015+0100 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Misc configuration': Connection to storage server failed
                                                                              
2018-01-05 23:22:24,016+0100 DEBUG otopi.transaction transaction.abort:119 aborting 'Yum Transaction'

Comment 12 Martin Sivák 2018-01-10 11:35:47 UTC
Can you please test with 4.2.1 build where the fix is very probably included?

Comment 14 kukucz 2018-01-11 16:50:45 UTC
 (In reply to Martin Sivák from comment #12)
> Can you please test with 4.2.1 build where the fix is very probably included?

Where build 4.2.1 is available to download? I don't see it under  oVirt Node 4.2 or oVirt Node Master links on https://ovirt.org/node/ page.

Comment 16 Yihui Zhao 2018-01-23 05:31:54 UTC
Created attachment 1384678 [details]
cockpit_iscsi_failed.png

Comment 17 Yihui Zhao 2018-01-23 05:32:35 UTC
Also deployed failed with 4.2.1 build.

Test version:
rhvh-4.2.1.1-0.20180115.0+1
cockpit-ws-157-1.el7.x86_64
cockpit-dashboard-157-1.el7.x86_64
cockpit-bridge-157-1.el7.x86_64
cockpit-157-1.el7.x86_64
cockpit-storaged-157-1.el7.noarch
cockpit-system-157-1.el7.noarch
cockpit-ovirt-dashboard-0.11.4-0.1.el7ev.noarch
ovirt-hosted-engine-setup-2.2.5-1.el7ev.noarch
ovirt-hosted-engine-ha-2.2.3-1.el7ev.noarch
rhvm-appliance-4.2-20171219.0.el7.noarch

Test steps:
1. Deploy HE with iscsi storage via cockpit .

Test results:
"""
The requested device is not listed by VDSM
 Failed to execute stage 'Environment customization': Cannot access LUN
 Hosted Engine deployment failed
"""
https://bugzilla.redhat.com/attachment.cgi?id=1384678

Additional info:
Works well with CLI.

Comment 18 Sandro Bonazzola 2018-01-31 07:17:08 UTC
Please test again with latest compose, it includes updates for cockpit-ovirt-dashboard, hosted engine setup and ha and appliance

Comment 19 Yihui Zhao 2018-01-31 07:53:51 UTC
(In reply to Sandro Bonazzola from comment #18)
> Please test again with latest compose, it includes updates for
> cockpit-ovirt-dashboard, hosted engine setup and ha and appliance

Comment 20 Yihui Zhao 2018-01-31 08:01:01 UTC
Due to the bug 1539598: Retrieval of iSCSI targets failed while deploying HE via cockpit based otopi deployment. 

Deploy HE with iscsi storage failed.

Test version:
cockpit-ws-157-1.el7.x86_64
cockpit-bridge-157-1.el7.x86_64
cockpit-storaged-157-1.el7.noarch
cockpit-dashboard-157-1.el7.x86_64
cockpit-157-1.el7.x86_64
cockpit-ovirt-dashboard-0.11.6-0.1.el7ev.noarch
cockpit-system-157-1.el7.noarch
ovirt-hosted-engine-setup-2.2.8-2.el7ev.noarch
ovirt-hosted-engine-ha-2.2.4-1.el7ev.noarch
rhvm-appliance-4.2-20180125.0.el7.noarch
rhvh-4.2.1.2-0.20180126.0+1

Comment 21 Sandro Bonazzola 2018-02-01 21:00:52 UTC
Moved to 4.2.2 since dependency bug #1539598 moved to 4.2.2.
Simone please move back to 4.2.1 if this can be reproduced consistently.

Comment 22 Simone Tiraboschi 2018-02-05 14:12:22 UTC
Moving to MODIFIED as for https://gerrit.ovirt.org/#/c/87141/

Comment 23 Yihui Zhao 2018-03-02 05:09:33 UTC
Tested cockpit-ovirt-0.11.14-0.1 on RHEL7.4 , deploy with iscsi successfully.

Test version:
Red Hat Enterprise Linux Server 7.4 Beta (Maipo)
ovirt-hosted-engine-ha-2.2.6-1.el7ev.noarch
cockpit-ovirt-dashboard-0.11.14-0.1.el7ev.noarch
ovirt-hosted-engine-setup-2.2.11-1.el7ev.noarch
cockpit-storaged-158-1.el7.noarch
cockpit-bridge-157-1.el7.x86_64
cockpit-157-1.el7.x86_64
cockpit-dashboard-158-1.el7.x86_64
cockpit-ws-157-1.el7.x86_64
cockpit-system-157-1.el7.noarch
rhvm-appliance-4.2-20180202.0.el7.noarch

So, change this bug's status to verified.

Comment 24 Sandro Bonazzola 2018-03-29 10:58:37 UTC
This bugzilla is included in oVirt 4.2.2 release, published on March 28th 2018.

Since the problem described in this bug report should be
resolved in oVirt 4.2.2 release, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.