Bug 1190636 - [hosted-engine] [iSCSI support] connectStoragePools fails with "SSLError: The read operation timed out" while adding a new host to the setup
Summary: [hosted-engine] [iSCSI support] connectStoragePools fails with "SSLError: The...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-hosted-engine-setup
Version: 3.5.0
Hardware: x86_64
OS: Unspecified
high
urgent
Target Milestone: ---
: 3.5.1
Assignee: Simone Tiraboschi
QA Contact: movciari
URL:
Whiteboard: integration
Depends On: 1169290 1173638 1190207
Blocks: 1193058 1197441
TreeView+ depends on / blocked
 
Reported: 2015-02-09 10:22 UTC by rhev-integ
Modified: 2015-04-28 18:46 UTC (History)
21 users (show)

Fixed In Version: ovirt-hosted-engine-setup-1.2.2-3.el7
Doc Type: Bug Fix
Doc Text:
Previously, hosted engine used the vdsClient utility to communicate with VDSM, which meant that SSL would timeout on sync commands that would take more than 60 seconds to complete. Now, the vdscli library is used for storage operations, due to configurable timeout, and longer sync commands no longer fail due to SSL timeout.
Clone Of: 1169290
Environment:
Last Closed: 2015-04-28 18:46:37 UTC
oVirt Team: ---
Target Upstream Version:
ylavi: Triaged+


Attachments (Terms of Use)
ovirt-hosted-engine-setup-20150312115314-ezn6al.log (271.35 KB, text/plain)
2015-03-12 11:00 UTC, Jiri Belka
no flags Details
vdsm.log (712.28 KB, text/plain)
2015-03-12 13:24 UTC, Jiri Belka
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Bugzilla 1190207 unspecified CLOSED vdsClient/vdscli SSLError timeout error 2021-01-20 06:05:38 UTC
Red Hat Product Errata RHBA-2015:0910 normal SHIPPED_LIVE ovirt-hosted-engine-setup bug fix update 2015-04-28 22:49:43 UTC
Red Hat Product Errata RHSA-2015:0888 normal SHIPPED_LIVE Moderate: Red Hat Enterprise Virtualization Manager 3.5.1 update 2015-04-28 22:40:04 UTC
oVirt gerrit 38751 master MERGED vdscli: allow a connection with a configurable timeout Never
oVirt gerrit 38961 ovirt-hosted-engine-setup-1.2 MERGED packaging: setup: increasing vdscli SSL timeout Never

Internal Links: 1190207

Comment 1 Sandro Bonazzola 2015-02-20 11:07:55 UTC
Automated message: can you please update doctext or set it as not required?

Comment 3 Jiri Belka 2015-03-12 11:00:54 UTC
Created attachment 1000930 [details]
ovirt-hosted-engine-setup-20150312115314-ezn6al.log

2015-03-12 11:57:26 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 142, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 965, in _misc
    self._activateStorageDomain()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 650, in _activateStorageDomain
    spUUID
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1294, in single_request
    response = h.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1045, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 409, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib64/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib64/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/usr/lib64/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)
SSLError: The read operation timed out
2015-03-12 11:57:26 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Misc configuration': The read operation timed out
2015-03-12 11:57:26 DEBUG otopi.transaction transaction.abort:131 aborting 'File transaction for '/etc/ovirt-hosted-engine/firewalld/hosted-console.xml''
2015-03-12 11:57:26 DEBUG otopi.transaction transaction.abort:131 aborting 'File transaction for '/etc/ovirt-hosted-engine/iptables.example''
2015-03-12 11:57:26 DEBUG otopi.transaction transaction.abort:131 aborting 'File transaction for '/etc/ovirt-hosted-engine-ha/broker.conf''
2015-03-12 11:57:26 DEBUG otopi.context context.dumpEnvironment:490 ENVIRONMENT DUMP - BEGIN
2015-03-12 11:57:26 DEBUG otopi.context context.dumpEnvironment:500 ENV BASE/error=bool:'True'
2015-03-12 11:57:26 DEBUG otopi.context context.dumpEnvironment:500 ENV BASE/exceptionInfo=list:'[(<class 'ssl.SSLError'>, SSLError('The read operation timed out',), <traceback object at 0x563b7e8>)]'
2015-03-12 11:57:26 DEBUG otopi.context context.dumpEnvironment:504 ENVIRONMENT DUMP - END
2015-03-12 11:57:26 INFO otopi.context context.runSequence:417 Stage: Clean up
2015-03-12 11:57:26 DEBUG otopi.context context.runSequence:421 STAGE cleanup
2015-03-12 11:57:26 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.core.remote_answerfile.Plugin._cleanup
[root@dell-r210ii-03 ovirt-hosted-engine-setup]# rpm -qa ovirt-hosted-engine-setup\*
ovirt-hosted-engine-setup-1.2.2-1.el7.noarch

Comment 4 Jiri Belka 2015-03-12 11:01:54 UTC
our env problem or related to this BZ?

Comment 5 Simone Tiraboschi 2015-03-12 12:30:21 UTC
No, the issue is still there and it's still the same:

2015-03-12 11:56:25 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._activateStorageDomain:645 activateStorageDomain
2015-03-12 11:57:26 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 142, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 965, in _misc
    self._activateStorageDomain()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 650, in _activateStorageDomain
    spUUID
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1294, in single_request
    response = h.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1045, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 409, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib64/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib64/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/usr/lib64/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)
SSLError: The read operation timed out

It sent activateStorageDomain command over vdscli at 11:56:25 and it failed at 2015-03-12 11:57:26, so 61 seconds after, for the 60 seconds SSL timeout as it was failing with vdsClient.
So moving from vdsClient to vdscli is not enough by itself so solve this.
We need to fix it within vdscli.

Comment 6 Simone Tiraboschi 2015-03-12 12:45:29 UTC
Could you please attach also VDSM logs?

Comment 7 Jiri Belka 2015-03-12 13:24:28 UTC
Created attachment 1000984 [details]
vdsm.log

Comment 10 errata-xmlrpc 2015-04-28 18:46:37 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0888.html


Note You need to log in before you can comment on or make changes to this bug.