Bug 1190636

Summary:

[hosted-engine] [iSCSI support] connectStoragePools fails with "SSLError: The read operation timed out" while adding a new host to the setup

Product:

Red Hat Enterprise Virtualization Manager

Reporter:

rhev-integ

Component:

ovirt-hosted-engine-setup

Assignee:

Simone Tiraboschi <stirabos>

Status:

CLOSED ERRATA

QA Contact:

movciari

Severity:

urgent

Docs Contact:

Priority:

high

Version:

3.5.0

CC:

aburden, amureini, bazulay, dfediuck, didi, ecohen, fsimonce, gklein, istein, jbelka, lpeer, lsurette, lveyde, nsoffer, oourfali, sbonazzo, scohen, sherold, stirabos, yeylon, ylavi

Target Milestone:

---

Keywords:

Reopened, ZStream

Target Release:

3.5.1

Flags:

ylavi: Triaged+

Hardware:

x86_64

OS:

Unspecified

Whiteboard:

integration

Fixed In Version:

ovirt-hosted-engine-setup-1.2.2-3.el7

Doc Type:

Bug Fix

Doc Text:

Previously, hosted engine used the vdsClient utility to communicate with VDSM, which meant that SSL would timeout on sync commands that would take more than 60 seconds to complete. Now, the vdscli library is used for storage operations, due to configurable timeout, and longer sync commands no longer fail due to SSL timeout.

Story Points:

---

Clone Of:

1169290

Environment:

Last Closed:

2015-04-28 18:46:37 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

1169290, 1173638, 1190207

Bug Blocks:

1193058, 1197441

Attachments:

Description	Flags
ovirt-hosted-engine-setup-20150312115314-ezn6al.log	none
vdsm.log	none

Comment 1 Sandro Bonazzola 2015-02-20 11:07:55 UTC

Automated message: can you please update doctext or set it as not required?

Comment 3 Jiri Belka 2015-03-12 11:00:54 UTC

Created attachment 1000930 [details]
ovirt-hosted-engine-setup-20150312115314-ezn6al.log

2015-03-12 11:57:26 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 142, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 965, in _misc
    self._activateStorageDomain()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 650, in _activateStorageDomain
    spUUID
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1294, in single_request
    response = h.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1045, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 409, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib64/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib64/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/usr/lib64/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)
SSLError: The read operation timed out
2015-03-12 11:57:26 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Misc configuration': The read operation timed out
2015-03-12 11:57:26 DEBUG otopi.transaction transaction.abort:131 aborting 'File transaction for '/etc/ovirt-hosted-engine/firewalld/hosted-console.xml''
2015-03-12 11:57:26 DEBUG otopi.transaction transaction.abort:131 aborting 'File transaction for '/etc/ovirt-hosted-engine/iptables.example''
2015-03-12 11:57:26 DEBUG otopi.transaction transaction.abort:131 aborting 'File transaction for '/etc/ovirt-hosted-engine-ha/broker.conf''
2015-03-12 11:57:26 DEBUG otopi.context context.dumpEnvironment:490 ENVIRONMENT DUMP - BEGIN
2015-03-12 11:57:26 DEBUG otopi.context context.dumpEnvironment:500 ENV BASE/error=bool:'True'
2015-03-12 11:57:26 DEBUG otopi.context context.dumpEnvironment:500 ENV BASE/exceptionInfo=list:'[(<class 'ssl.SSLError'>, SSLError('The read operation timed out',), <traceback object at 0x563b7e8>)]'
2015-03-12 11:57:26 DEBUG otopi.context context.dumpEnvironment:504 ENVIRONMENT DUMP - END
2015-03-12 11:57:26 INFO otopi.context context.runSequence:417 Stage: Clean up
2015-03-12 11:57:26 DEBUG otopi.context context.runSequence:421 STAGE cleanup
2015-03-12 11:57:26 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.core.remote_answerfile.Plugin._cleanup
[root@dell-r210ii-03 ovirt-hosted-engine-setup]# rpm -qa ovirt-hosted-engine-setup\*
ovirt-hosted-engine-setup-1.2.2-1.el7.noarch

Comment 4 Jiri Belka 2015-03-12 11:01:54 UTC

our env problem or related to this BZ?

Comment 5 Simone Tiraboschi 2015-03-12 12:30:21 UTC

No, the issue is still there and it's still the same:

2015-03-12 11:56:25 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._activateStorageDomain:645 activateStorageDomain
2015-03-12 11:57:26 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 142, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 965, in _misc
    self._activateStorageDomain()
  File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 650, in _activateStorageDomain
    spUUID
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1578, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1264, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python2.7/xmlrpclib.py", line 1294, in single_request
    response = h.getresponse(buffering=True)
  File "/usr/lib64/python2.7/httplib.py", line 1045, in getresponse
    response.begin()
  File "/usr/lib64/python2.7/httplib.py", line 409, in begin
    version, status, reason = self._read_status()
  File "/usr/lib64/python2.7/httplib.py", line 365, in _read_status
    line = self.fp.readline(_MAXLINE + 1)
  File "/usr/lib64/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
  File "/usr/lib64/python2.7/ssl.py", line 241, in recv
    return self.read(buflen)
  File "/usr/lib64/python2.7/ssl.py", line 160, in read
    return self._sslobj.read(len)
SSLError: The read operation timed out

It sent activateStorageDomain command over vdscli at 11:56:25 and it failed at 2015-03-12 11:57:26, so 61 seconds after, for the 60 seconds SSL timeout as it was failing with vdsClient.
So moving from vdsClient to vdscli is not enough by itself so solve this.
We need to fix it within vdscli.

Comment 6 Simone Tiraboschi 2015-03-12 12:45:29 UTC

Could you please attach also VDSM logs?

Comment 7 Jiri Belka 2015-03-12 13:24:28 UTC

Created attachment 1000984 [details]
vdsm.log

Comment 10 errata-xmlrpc 2015-04-28 18:46:37 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2015-0888.html